Fanatics make people nervous. Cults are full of fanatics. Unease translates very easily(maybe even automatically) into hostility. Mormonism has still not managed to escape the cult stage. Who knows if it ever will; the cult/religion line is drawn not by members of the faith in question but by the observers.