Why So Many Psychological Studies Fail To Hold Up

One of the key facets of any scientific study is that no matter what, its results are questioned and tested again. A number of popular psychological studies, like the idea that smiling makes you happier or that willpower is a limited resource, haven’t held up to that scrutiny. They’re not totally bogus, but they’re not definitive either. Here’s what’s really going on.

Illustration by Angelica Alzona. Images by Bridget Coila, Sara and University of Liverpool.

Researchers Can Manipulate Results to Meet Different Theories

Back in 1998, researchers published a paper titled “Ego Depletion: Is the Active Self a Limited Resource?” in the Journal of Personality and Social Psychology that changed how we thought about willpower completely. The idea was simple: Put some cookies on a plate next to a bowl of radishes, tell participants to hang out in the room, then test them with an impossible puzzle afterwards. The people who ate the radishes gave up on the puzzle before the people who ate cookies, which suggested to researchers that using willpower (not eating the cookies) taps into some kind of willpower reserve. The more self-control you use in the moment, the less you have to use in the future. Since this landmark study in 1998, ego-depletion, which includes self-control or willpower, has been a huge subject of research.

We’ve mentioned this idea of limited willpower a couple of times, because at its core, it feels like it’s true. In fact, many similar studies followed over the years that seemed to reinforce this idea.

In 2010, a meta-analyses of 83 studies in ego depletion research published in Psychological Bulletin found overwhelmingly that ego depletion is a real thing. Then, in a weird twist, another set of researchers analysed much of the exact same data in an article in Frontiers in Psychology and found no evidence for ego depletion. Those same researchers did a follow up meta-analyses in the Journal of Experimental Psychology that added in unpublished research and again found no evidence of ego depletion. There is a bias that comes with analysing data, and that’s extremely evident here. Researchers can manipulate data to meet either end.

So, what does all this mean to you as a normal person? It means scientists aren’t totally certain how willpower or self-control work. Self-control can lapse, but we don’t know exactly why. We might have a reserve of willpower that we’re tapping every time we make a choice. Or we might not. In the end, ego depletion is still a theory that needs testing.

It’s likely best for most of us to go back to chalking up bad decisions to general fatigue. Instead of worrying about some imaginary reserve of willpower, just try to avoid those tempting situations altogether.

Your Background Plays a Role That’s Hard to Test For

Have you heard that simply thinking about money makes you selfish? According to a 2013 paper published in the Journal of Experimental Psychology, that’s certainly the case. The study had participants fill out two questionnaires, one with a plain coloured background and another with a faint background image of a $100 bill. They found participants who saw the $100 bill had less sympathy for victims and socially disadvantaged groups, and even subtly altered their political views about the free market. Considering the US is in a big election year, this idea of why some people are more sympathetic than others is certainly on a lot of minds right now. This study, and those that followed it by the same researchers brought about headlines like “How Money Makes People Act Less Human” and “Just Looking at Cash Makes People Selfish and Less Social“.

Yet, when 36 labs attempted to replicate the tests, they didn’t get the same results. Their study, published in Social Psychology, was a part of the “Many Labs” replication project, a project meant to replicate psychology experiments on an international scale. Out of the 36 labs, only one reported the same significant effects as the original study.

A 2015 study, also published in Journal of Experimental Psychology, attempted to replicate the same 2013 test on a larger scale with a bigger sample size, but failed to get the same results.

In reply to this, one author of the original study pointed to several other studies over the years that have produced similar results, namely that just thinking about money makes people selfish and even antisocial. Which is, again, all to say that we still have evidence on both sides.

So what’s really going on here? Should you avoid looking at $100 bills? Do you suddenly become an arsehole every time you open your wallet? Does the very act of seeing a $100 bill turn you into a free market capitalist? Well, it depends.

There’s an argument here that the biggest influencing factor is how important money is to you to begin with. If it’s important to you, this priming might make you more selfish, if it’s not, there probably won’t be any change in your responses. Or maybe that’s not a factor at all. We don’t actually know, that’s just a theory. For the time being, you’re probably best off ignoring how much money you have and just fixing your habits.

Social Sciences Are Inherently Hard to Test

You’ve certainly heard this one before: Smile and you’ll feel better. There’s even research to prove it. A study published in 1988 in the Journal of Personality and Social Psychology found that people thought Far Side comics were funnier when researchers forced them to smile. This is the kind of stuff that fills the heads of authors of self-help books. Similar studies followed with similar results, including one published in Cognition and Emotion that tested the effect of a furrowed brow and another from the Creative Research Journal that suggests that a pencil between the teeth might make people more creative. Unfortunately, these studies aren’t as definitive as you’d think.

Earlier this year, 17 different labs attempted to replicate the results of the 1988 experiment and in an article published last month in Perspectives on Psychological Science, found that their results were inconsistent with the original study. In the end, nine of the 17 labs found similar results to the original study, but the size of this difference was smaller than the original study. The other eight labs found either no evidence of higher ratings or results in the opposite direction altogether.

That’s not the whole story though. There were some differences in how the study was conducted in the replication, the biggest being that participants were filmed. It’s arguable the participants altered their behaviour since they knew a camera was on them. To complicate things a bit more, it’s also worth noting that while it might seem like a good practice to use the same Far Side cartoons from the original 1988 study, participants in 2016 certainly have a different sense of humour than people 28 years ago. That alone could have skewed the results.

Replicating studies is hard, especially with social sciences. Even if you’re doing it by the book and replicating a study exactly, the results might be different. This doesn’t mean the original results are false, but it does mean it’s worth looking deeper.

For a more recent example, we recently pointed out that another priming study had its results called into question. In this case, it was the idea that faking certain body language, “power poses”, reduces stress and improves performance. In the case of power poses, one of the co-authors, Amy Cuddy, came out and said the results were statistically tweaked to get the results the researchers wanted. She even adds that the study wasn’t that great in the first place, saying too many participants understood the hypothesis they were testing, which should automatically negate the results.

Whether it’s power posing or smiling, it’s really up in the air what, if any, effects these priming techniques might have on behaviour. They might fall under the “if it works for you then keep doing it” mantra, but it’s just as important as a reader to remain sceptical. None of these studies, no matter how fanciful a headline you read, applies to 100 per cent of people. There are no magic bullets, and there is no such thing as one-size-fits-all advice, so always be wary of any study that suggests there’s a single fix to common problem.

Recently, the Reproducibility Project launched with backing from the Association for Psychological Science. This project’s goal is to reproduce psychological studies, analyse results and come up with more reliable methods of testing. Right now, the failure rate for reproduction is pretty high, with somewhere around 64 per cent of studies failing. That sounds bleak, but it means more tests are needed and that it’s worth challenging our long-standing beliefs just as often as new research.

You may have noticed Lifehacker Australia looks different. We’re keen to hear your thoughts on the redesign. Share your feedback here!