I have been trying to find a good fit for a question about meta-analyses. The validity of general scientific research methods seems to be either a question suited for "academia" though apparently that SE was rejected as being a fit or philosophy. Philosophy seems to be a catch-all, and that seems reasonable as philosophy is a general process of asking whether or not the way we think about things is reasonable (From my answer on "Who is a Philosopher").

My take is that whether or not a research method, such as meta-analyses, are reasonable, seems to be a philosophy of science question.

  • 1
    The idea that philosophy is a catchall for topics is, prima facie, not true and in practice has been rejected many times by the community on this site. I can think of a handful of recent examples where some other SE closed a question and said "take it to philosophy" but that question ended up being rejected here because it was obviously not a question about philosophy. There are people who actively use this site and would disagree with your stated characterization of what philosophy is, both for the scope of the site and in general.
    – Not_Here
    Nov 27 '17 at 15:55
  • 1
    There have been multiple meta discussions about what the scope of "philosophy" for this site is, and while there has not (and probably could not) be a 100% consensus, most of the regular users agreed that it is more strict than something like you proposed (e.g. "could a cat and a mouse be friends?" is off topic even if it seems to hint at something 'deeper'). All of that being said, I agree with the academia user who said that the question seems open ended and soliciting discussions or opinions, it isn't objectively answerable.
    – Not_Here
    Nov 27 '17 at 15:58
  • 1
    If the question was posed to ask more about the philosophy of science, about the nature of different techniques in data analysis (thats what I'm assuming you're asking about, your question was vague to me), then I can see being reasonably well received but as it is I have to again say I agree with the comments on Academia that suggest that it just isn't an SE type of question with an objective answer. The only thing philosophy related you could hope for would be some sort of references to philosophers of science writing about the issue but what you'll end up with are just opinions.
    – Not_Here
    Nov 27 '17 at 16:05
  • It is a catch all, but not even all questions fit that. It cannot be too opinion based of course. There has to be some kind of reasonable argument that could be made, and I would definitely say that "is this research method valid" is a reasonable philosophy of science question. Nov 27 '17 at 16:05
  • "Philosophy" is not a catch all term, that is the fundamental misunderstanding of the majority of people who come to this site and end up getting into arguments with moderators and other regular users and then leave after a few weeks. It is a well defined subject that has existed in cultures around the world for thousands of years, and this site leans more towards formal, Western philosophy. It is not a catch all.
    – Not_Here
    Nov 27 '17 at 16:07
  • 2
    Surely there are philosophical questions about research methods, that would be under methodology of science. The problem with your question as currently phrased, "are meta-analyses in general, useful and reliable?", is that "in general" there is no answer, it depends on context. More to the point, this context may involve philosophy or it may be purely technical, Wikipedia has a long list of the latter problems. To ask the question here you need to identify what philosophical problems come up there rather than give a generic catch-all.
    – Conifold
    Nov 27 '17 at 21:07
  • 1
    There's an active literature on meta-analysis in philosophy of science / philosophy of medicine. These include general critiques of meta-analysis (and randomized controlled trials), analyses of the epistemic strengths and limits of these methods, and discussions of how they're used in actual scientific practice. So there are philosophical answers to the general question about meta-analysis. There are also discussions of the original specific question, namely, using data from different study designs in a single meta-analysis.
    – Dan Hicks
    Dec 3 '17 at 16:40

In all fairness, the reasons why meta-analyses are problematic are statistical in nature: As long as the data is not collected with the very same scope and method, it cannot be precluded that different moderating effects, especially in choosing the participants and asking the questions, are at work. Simply hoping for the best and that they kind of even out is not exactly scientific, see e.g. publication bias.

My own statistics lecturer pointed out that after-the-fact analyses are something that differs from scientific inquiry through statistical methods. Testing is completely depending on the hypothesis you test and includes every single phase.

I have seen meta-analyses that completely ignored important factors like different takes on what the population is, different metabolic peculiarities because of ethnicity and culture (in medicine papers) etc, etc. The problem is not "saying something about different papers", the problem is "pretending you have one single set of data while you, in fact, have dozens of different ones where you cannot even determine the differences and commonalities".

For literature on how to circumvent all these objections (which surely is not philosophical), see: Schmidt, F. L., & Hunter, J. E. (2014). Methods of meta-analysis: Correcting error and bias in research findings. Sage publications.

That being said: Although there is a huge discussion going on in the philosophy of science about what it takes to provide sufficient scientific validity and reliability (esp. regarding causal connections), the mathematical or statistical method itself is seldom the point of the particularly philosophical contribution to the discussion. Philosophers do, as far as I am aware of, refer to the standards established in the sciences and statistics.

Overall, the question in Health.SE seems to be a good fit and is considered a good fit as far as I can see. Maybe a carefully phrased question about the statistical obstacles in meta-analyses could be a good fit for Mathematics.SE as well, but it would have to be very technical.

As @Keelan points out in his comment, CrossValidated.SE would probably be the perfect place, as both the purely technical as well as problems in application of statistical methods are in the scope of that community.

  • 2
    I agree. In principle, there could be some philosophical issues related to interpretation of meta-analyses, but the questioners would have to identify what those are to ask here.
    – Conifold
    Nov 27 '17 at 21:11
  • 1
    There is also Cross Validated about statistics, which may be more practical than Mathematics.
    – user2953
    Nov 27 '17 at 23:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .