It’s not exactly what you were asking about, but I have become really concerned about the widespread abuse of statistics in the social sciences. There are a few excellent papers pointing out serious errors in widespread practice, but the vast majority of researchers blithely continue to use incorrect methods. The real problem is that there is no incentive for people to get this stuff right – if nearly everyone is making the same error, it doesn’t stop results being published.
An area where this particularly concerns me is meta-analysis; there are specific issues there which mean that the widespread reliance on meta-analyses as the highest standard of evidence is shakier than is widely realised.
The key quote is in the conclusion: “Our modest sample of recent meta-analyses shows that standardization is a popular approach to combine mean effects from controlled trials and crossovers, but the use of pre-intervention SDs to standardize is uncommon(…) If standardization was performed with anything other than pre-intervention SDs, or if it is unclear which SDs were used, readers should not cite the meta-analysis or act on its outcome, unless they can make a reasonable correction to the meta-analyzed magnitude. Many published meta-analyses will need to be discounted and updated.”
If you ping me once you’ve had a look at that, I will dig out older papers which discuss related issues but don’t necessarily highlight how they affect meta analysis in particular.
It’s not exactly what you were asking about, but I have become really concerned about the widespread abuse of statistics in the social sciences. There are a few excellent papers pointing out serious errors in widespread practice, but the vast majority of researchers blithely continue to use incorrect methods. The real problem is that there is no incentive for people to get this stuff right – if nearly everyone is making the same error, it doesn’t stop results being published.
An area where this particularly concerns me is meta-analysis; there are specific issues there which mean that the widespread reliance on meta-analyses as the highest standard of evidence is shakier than is widely realised.
Whoa! Thanks for flagging. I wasn’t familiar with this at all! I’d love to learn more! Do you have favorite papers you’d recommend?
I would start with this recent paper as it situates the issues in the context of meta analysis… https://onlinelibrary.wiley.com/doi/10.1002/sim.10114.
The key quote is in the conclusion: “Our modest sample of recent meta-analyses shows that standardization is a popular approach to combine mean effects from controlled trials and crossovers, but the use of pre-intervention SDs to standardize is uncommon(…) If standardization was performed with anything other than pre-intervention SDs, or if it is unclear which SDs were used, readers should not cite the meta-analysis or act on its outcome, unless they can make a reasonable correction to the meta-analyzed magnitude. Many published meta-analyses will need to be discounted and updated.”
If you ping me once you’ve had a look at that, I will dig out older papers which discuss related issues but don’t necessarily highlight how they affect meta analysis in particular.