Cherry Picking

by Craig Shrives

What Is Cherry Picking in Statistics?

"Cherry picking" is a underhanded technique employed by statisticians. Cherry picking occurs when a person makes a claim about a subject and then selects only the tests that support the claim or ignores any translation of the data that could refute the claim. In other words, it's about selecting your data to prove your claim.

Cherry picking can be described as deliberate Confirmation Bias.
What is cherry picking in statistics?
The cherry-picking technique is particularly prevalent in the world of nutritional supplements, where commercially driven scientists have become adept at finding evidence to support the marketeers who peddle potions and supplements designed, or – more correctly – marketed, to cure all ills.
"A common cold if given proper treatment will be cured in seven days but left untreated it will hang around for a week."
(Common saying)

Seemingly Reputable Scientists Will Engage in Cherry Picking

cherry picking survey Marketeers have been using cherry-picked tests for years to sell us stuff, ranging from the notion that Vitamin C prevents the common cold (in spite of the numerous medical studies proving that it doesn't) to the notion that root vegetables and lemon rind prevent HIV. I'm always up for a bit of blackcurrant-flavoured "Lemsip" on the off chance that Vitamin C does actually help a cold, but clearly not all the marketeers' assertions are so harmless. The root-vegetables-and-lemon-rind "cure" and other similar quack "remedies" have been responsible for hundreds of thousands of deaths from AIDS in South Africa over the last 10 years. When I say "quack", don't imagine some backstreet snake-oil merchant. It seems some of the big drug companies have been cherry-picking too. GlaxoSmithKline (one of the world's largest pharmaceutical companies) was fined a whopping $1.9 billion in July 2012 for misselling drugs, and numerous pharmaceutical companies have been caught bending or outright falsifying statistical data in an attempt to prise your cash from your grasp.

Cherry Picking Can Be Very Underhanded

statistics cherry picking Some of the tricks employed in spinning statistics are blatantly underhand. An obvious one is only reporting test results that support your claim. That's pretty bad, but companies have also been caught duplicating positive data in different scientific papers to make it look as if several studies have reached the same conclusion. Another ploy they use is to stop a study as soon as they've gained sufficient positive data. In other words, they're not letting the Law of Large Numbers kick in. They know that by continuing the trial, the likelihood of their positive result being down to chance would increase.

This stuff is all pretty dishonest, but the worst manipulation of all is suppressing negative data on drugs shown to be harmful – even to the point of suing doctors attempting to highlight a problem.

So, how prevalent is cherry picking? Well, back in 2003, it was found that trials run by drug companies were four times more likely to give a favourable outcome than a study carried out independently.

Globally, pharmaceutical industries are worth £150 billion a year. And some of that is because they've been looking at the data from their trials through their "profit first" goggles.

Critical Thinking Test

Are you good at spotting the biases, fallacies, and other cognitive effects? Can you spot when statistics have been manipulated? Can you read body language? Well, let's see!
gold cup
Gold

gold cup
Silver

gold cup
Bronze

0
  • This test has questions.
  • A correct answer is worth 5 points.
  • You can get up to 5 bonus points for a speedy answer.
  • Some questions demand more than one answer. You must get every part right.
  • Beware! Wrong answers score 0 points.
  • 🏆 If you beat one of the top 3 scores, you will be invited to apply for the Hall of Fame.
Scoring System

Guru (+)
Hero (+)
Captain (+)
Sergeant (+)
Recruit (+)
Help Us To Improve

  • Do you disagree with something on this page?
  • Did you spot a typo?
  • Do you know a bias or fallacy that we've missed?
Please tell us using this form

See Also