Remember that game where you guess how many jelly beans are in a jar? Well, if I guess once, I'll probably be wrong. But if I'm allowed to guess 500 times, I might get it right once. But that doesn't mean I'm a good guesser of jelly beans numbers.
The same concept applies to research studies. The more times you test a question, the more likely you are to get a postive result. That doesn't mean that the answer to the question is "yes". It just means you guessed so many times that your were bound to get a correct answer.
The New York Times Missed this point in their article when they wrote the headline:
"Obesity Rate for Young Children Plummets 43% in a Decade"
Sounds like a huge drop. I wish it was. But the researchers tested many age groups to see if there was a decline over 10 years. They checked the age ranges 2-5, 6-11, 12-19, 20-39, 40-59, and >60. When you "guess" if there is a decrease in obesity in all of these age groups, you are likely to find one "yes".
The researchers admit this and even caution interpreting their results this way:
"When multiple statistical tests are undertaken, by chance some tests will be statistically significant (eg, 5% of the time using α of .05). "
And they conclude:
"Overall, there have been no significant changes in obesity prevalence in youth or adults between 2003-2004 and 2011-2012. "
Yet some how the New York Times missed this, and focused their headline on a minor result. While we might have made small gains against childhood obesity, we haven't really begun to change the trajectory of the epidemic.
