The solution is that it’s a social contract. I agree to tolerate your weirdness and quirks. You agree to do the same to myself and others.

By being intolerant (without a good reason), they break the social contract. Therefore they are no longer protected by it either.