For a number or reasons, from the ease of sharing information online to the increasing availability of studies through open-source means, this type of headline is becoming more apparent. Increasingly, this headline is meant to grab attention and provide incomplete information to spin a narrative than information share. Chocolate, acai, kale, are super-foods that will lead to weight loss and beautiful skin and so many other amazing things. Most of the time, this is either incorrect or is at best a misrepresentation of the results of a study. So, how can we protect ourselves from these types of shenanigans?
Research Basics and Design
First, a quick disclaimer. My Ph.D. training is in marketing, so my writing will inherently be biased toward social science research. This shouldn’t make too much of an issue with the broad strokes of what we are trying to accomplish.
What is the purpose?
That is to say, when I look at research, the first thing I notice is the purpose of the study. What was the intention of the author(s) before the start of the study and what warranted its investigation. If the purpose is somewhat nebulous, then the study is already suspect in my mind. This is also critical in “hard science” research (such as the STEM fields) since a finding that is outside of the purpose of the study can’t be given too much credence, e.g. studying X on Y and it turns out that there is another random effect that was captured in the data that wasn’t within the purpose, so we don’t pay too much attention to the latter.
Do you know what others have done?
Secondly, I want to see how this study ties to other research as research inherently builds on one another. The study should clearly indicate what has been studied prior, what are the main contributions, and what this particular research would add based on the unique characteristics of the research. If this is incomplete, the researcher(s) may not know the area to the depth required to make the necessary predictions. You need to read other’s works to make sure what you are studying has merit.
Extreme and ridiculous example
If you come running into a room and yell, “If I drop this book, it would fall to the ground! I am calling this ground-pull effect!” It would be difficult to see the merit of the argument as being novel. The ground-pull effect put in to a unique context could be useful. If you are showing how gravity works on a unique planet or in a specific space-based phenomenon, then there easily could be something interesting to note there. Other fields and disciplines can benefit disparate theories and hypotheses Gravity may be studied in several disciplines and utilizing theories from other areas can lead to discoveries in all areas. A rising tide raises all ships.
In theory or hypothetically speaking?
Once you confirm what others have researched on a topic, that means that it is reasonable to say, “based on what these other people have said, I hypothesize that this thing will occur”. A hypothesis is a prediction, the theory is what drives this prediction based on prior research. If you really want to mess with someone who is a researcher, you can say “theoretically, blah blah blah”. Most of the time, the right word is “hypothetically” since these words have different meanings. In the gravity example, gravity is the theory that leads to the hypothesis “dropping this book will make it fall to the ground”.
Did you do it right?
Once there is a hypothesis to test, the third piece is the methodology. Methodology refers to how the study(ies) are set up and if they are done properly. There are entire books on proper methodology, but I will use a common example to illustrate how methodology can lead us astray. If I am wanting to predict that dropping a book will make it fall to the ground, it wouldn’t be valuable to do this in a room without gravity, underwater, etc. This would be adding variables outside of just the force of gravity, so the test itself is already suspect. While this may sound extreme, there have been many, many instances where bad methodology has lead to incorrect predictions of phenomenon.
Did you collect the right data?
Next is the best part of research in my opinion, the data. My research and professional backgrounds all are data-driven, so I am the most critical here. Having a good methodology with bad data and statistical methods is a fatal flaw to me. A fairly famous example of this, which I wont link because it has already received so much scrutiny, is stopping data collection early when there are multiple instances of data collection. Stated differently, we are looking at studying three different time points, but the data were only collected for two of those three. The third point could be the main crux of the hypothesis, so stopping early would deny the full investigation of the phenomenon. There are plenty of other examples of poor statistical choices with data, but I want to make sure I don’t go too strong on one particular area.
Do you know your limits?
The final component is the the limitations. All studies are flawed, but some are useful. If an article doesn’t highlight tangible, realistic limitations, then I tend to wonder why this would be the case. There is nothing wrong with limitations since everything is flawed in one way or another, but to not acknowledge this or pretend that there aren’t any is a big drawback in my mind. Some limitations, such as testing gravity in water, may be too much to overcome. A limitation such as “we did the drop test on a mattress so it wasn’t so loud” may be acceptable as long as it is noted that “future research should investigate this phenomenon without the mattress present”.
Have a skeptic’s eye
As you probably have guessed, I am a bit of a skeptic about pretty much everything, particularly research. Even with this skepticism, I always recommend to go in to things with an open mind, but make sure to question everything. If the research is sound, then it should stand up to scrutiny. Plus, the researcher is probably excited that someone read their stuff, so that could potentially make their day.
Robert Allen King
Assistant Professor of Marketing & Wilder Professor of Business
