I have made three down-votes, total, on this SE. For all three I gave a comment why, but I quickly realized that was not the most effective way of responding to posts I did not like for the following reasons:
I lost one reputation point for each of those three down-votes. So the site rules punish me for down-voting. Losing a reputation point should be a wake-up call for anyone down-voting.
The down-voted post lost only two reputation points. An up-vote would have given the poster five points for a question and ten points for an answer. One up-vote on an answer is the same as five down-votes on that answer. Again the site rules do not encourage down-voting.
The down-vote doesn't lead to any further action. A flag, on the other hand, sends the post to a review queue or to a moderator for action leading to a possible deletion. A down-vote is just a down-vote.
If the user disagrees with some assumption, a down-vote without an explanation in a comment doesn't clarify that assumption. It is ineffective except to annoy the original poster. I have no interest in annoying people.
So, I don't see why anyone down-votes on an SE site given the current rules, but sometimes people don't realize the options they have to deal with posts they do not like.
To answer the question:
But, in all honesty, is that really necessary on a philosophy question site? Even if the question is insane, philosophers may be able to clarify the mistake. Not just down-vote without explanation.
Given the above I don't see the necessity of down-voting on any SE site, but I do see the necessity of commenting, flagging and voting to close or delete posts.