The First Amendment of the United States proclaims that citizens have the freedom to assemble. This freedom is for every United States citizen. It does not specify the purpose that people have to have in order to assemble; instead, it says that all citizens have this freedom.
Over the years, there have been many groups that have assembled for the purpose of showing their support for ideas that have not been popular. Such ideas have included the KKK. There are people who oppose gatherings for certain groups. I believe that every group should be able to assemble.
The First Amendment did not say that only certain groups could or could not assemble. It would be a direct discrimination against ideas if some people could not assemble because their ideas are not "normal" for society. Also, it is not the assembly of those people that is the problem, it would be the actions that they took.
However, just because there is a risk for those people to take action, it is not right to limit their freedoms just because there is a potential risk. That would completely contradict the aspect of freedom.