Well, they do have the right to be heard. As lone nuts. They don't have the right to form hate groups, or be activists in advancing their cause, and that's been illegal over here in the US for decades now.
And yes, it's hypocritical to say "I can't tolerate intolerance." in a perfect world, where we're all just spitballing ideas. But this is the real world. Where intolerance has societal consequences and lasting affects. After all, the real world isn't "fair and balanced".