It seems pretty obvious that certain users are voting down questions etc. based on personal opinion and lack of perspective - i.e. dislike of certain other posters.
How can we stop this, given the size of the community?
The Gist: Downvotes are cast by more regular users of this site than upvotes and should therefore be taken more seriously. Many downvotes indicate a problem with the posts; not with the people downvoting. Read further for the long story...
You have asked for explanation on downvotes before in Why so many down / close votes. Of course, this question is geared toward downvotes on all posts and not specifically your posts, but I'm very interested what kind of evidence you have for the situation you claim to be present. With your level of reputation you can see detailed vote counts, but you can't see who voted what on which question. Due to the lack of data supporting your claim, I voted to close this question as 'unclear what you're asking'.
Some interesting data can be found using this query, but it's not enough to support your claim.
The help center tells us when to vote down:
Use your downvotes whenever you encounter an egregiously sloppy, no-effort-expended post, or an answer that is clearly and perhaps dangerously incorrect.
You have a limited number of votes per day, and answer down-votes cost you a tiny bit of reputation on top of that; use them wisely.
If a certain type of question attracts many downvotes, that's a reason for the community to discuss about that kind of question. Not about the amount of downvotes. Down voting is a privilege for 125+ users, so everyone who can downvote should know this site a bit at least. Hence, down votes should be taken seriously, especially when some questions attract many. Unfortunately, this meta is not very active.
Lastly, calling people trolls isn't very constructive. I am an active downvoter having 166 downvotes and 129 upvotes. Still, I wouldn't call myself a troll and I can give a detailed explanation for every single downvote I cast. I won't, because I don't have the time. My main reason is fairly well explained (and unfortunately, I have gotten little response) in What can be done to get less rubbish answers?. Allow me also to link to Friends, we are not philosophers by Joseph Weissman.
Sometimes I think a question doesn't fit on the site, but it's a new user and a potentially good question. Then I only downvote if the question has a positive score. I find it important that users aren't encouraged to write crap.
Opposite to this healthy self-critical attitude, there exist actual trolls who vote down questions and answers of specific users, typically high-rep users. This is called serial downvoting and is detected automatically by the system.
I think the system currently in place, if used as expected, deals with this problem.
We need to encourage those who see a good answer / question to up-vote it. The peer voting system we currently have in place should result in a net vote; representative of the quality of the question / answer.
There is perhaps an argument for changing the score ratio between a good vote and a bad vote. eg. +1.5 for a up-vote, -1 for a down-vote. Therefore, ensuring that good votes are given more credibility; that said - it wouldn't be particularly balanced in the long term.
Equally, it's worth noting that often, especially in the field of philosophy, one individual may disagree with the premise / structure of a question and another may not. This occurs less so in many other fields.
As a result, to down-vote a question because you have the personal opinion that it is badly structured / the premise is incorrect is, in many ways; a legitimate down-vote.
Clearly, down-voting an individual due to a dislike of another poster is not acceptable, this is difficult to protect against. The only solution I could see would be to hide the identity of those with a question until a later point in time - not brilliantly usable.
All in all, I do think that the system is not the problem but the way in which it is being used. How do we change behavior? Probably by changing the reward schedule / structure.
I'd actually be in favor of an ever-changing reward structure based on some kind of algorithm which takes into account the current mean behavior of everyone in the community. Not easy to implement (or explain!)...