Note: my answer is about the site and how it works/should work; it is NOT about any user or group of users.
In my opinion, wrong answers and non-mainstream answers are completely different, and should be dealt with differently. The definition of non-mainstream physics given in the FAQ
What defines mainstream physics?
Mainstream physics is physics which has been accepted by a significant portion of the physics community. In the case of modern physics, if a theory has not been published in a reputable journal, it is not considered mainstream.
is rather objective: the papers backing your answer either do exist or do not exist. It's not subjective, and has nothing to do with squaring with personal views.
I, personally, think that bad answers should be downvoted and non-mainstream answers should be flagged.
The scope of this site (as discused in the tour, the help center and several meta posts) is only mainstream physics, not physics in general. Therefore, any non-mainstream answer is off-topic, and should be removed. On the other hand, a wrong answer is perfectly on-topic.
Non-mainstream answers harm this site. Imagine that someone answers that "the Earth is flat", and that answer is not deleted. It will attract some downvotes and will be grayed out, I know, but it will be still hosted by our site, and with our logo as a kind of seal of approval. This site is a reference of good content, and has a potential audience of thousands of people. We are saying to that audience that the whole community (except the few downvoters, who disagree) regard the "flat Earth theory" as acceptable for our site. If non-mainstream physics proliferate, the reputation of the site will suffer, and we won't be a useful resource anymore. Non-mainstream physics is an attack against Physics.SE, and the community should defend itself, and that's the purpose of flags. On the other hand, a wrong answer, marked as such, can have an educational value (showing common pitfalls, teaching something to both OP and answerers...), and be benefical for the site.
Downvoting has a cost (yeah, I know, only one little shiny internet point, but that's the point of gamification...) which discourages downvoting. In the case of wrong answers it makes sense: you have to ponder if the answer is bad enough for a downvote, or if it needs only a comment. But pointing non-mainstream answers always helps the community, and shouldn't be punished. Using flags seems to be the best option (better that meta or chat, that are too public for sensible matters).
People who get downvotes tend to regard them as personal attacks. Non-mainstream answers usually attract lots of downvotes, and they might think that that is the result of some sort of non-existent collusion. If their posts are dealt with by the site instead of some of the users, that (hopefully) would ease their mind.