How We’re Fooled By Statistics

How We’re Fooled By Statistics

Is punishment or reward more effective as feedback? Do new medical treatments really work? What about streaks in sport? Without considering regression to the mean, we are prone to making significant errors.

Check out
Filmed at Perimeter Institute:

Is punishment or reward more effective for helping people learn. A lot of people would say different incentives motivate different people, or in different circumstances, but in psychology there is a sizable body of evidence that in order to learn skills, positive feedback is more effective. This fining has been verified not just with humans, but also with other species.

It was strange then that after Daniel Kahneman discussed this research with Israeli fighter pilot instructors that he was met with resistance. They found the opposite was true: when they reprimanded a cadet for performing poorly, he invariably improved, but if they praised a cadet for an excellent performance, the next attempt was not as good. In order to solve this apparent contradiction we first need to understand regression to the mean.

Teacher study:
Rugby player study:


Like it? Share with your friends!


What's Your Reaction?

hate hate
confused confused
fail fail
fun fun
geeky geeky
love love
lol lol
omg omg
win win


Your email address will not be published. Required fields are marked *

  1. I can hardly believe that too much positive feedback makes people perform better. Could you show the supporting evidence for that claim, together with an analysis of the methods?
    I have never seen anyone getting really professional at something if they were rewarded too much for first steps (or rewarded way beyond the positive feeling of having solved a task). So how much positive feedback is necessary and when is it getting too much?

  2. Casinos know a lot about regression to the mean. Regression to the mean is why Las Vegas is so big and why most blokes come home broke. If they ever offer you a free suite, it's probably time to go home.

  3. I'm a recent subscriber. Great content. But, if I can be a little picky, the A10 Warthog model that you use in this video is not a "fighter", it's purely a ground-support plane, and excellent at that. Also, the Israelis never bought the A10, as far as I know. Nevertheless, great show.

  4. I mean, it actually makes sense from a human perspective too: If a teacher keeps encouraging people when they do well in the subject, they enjoy it more an are more likely to want to do the work. If you only discourage them, then they won't enjoy the subject and thus do as little of it as they can…

  5. Deja vu – I just about to say that I could have sworn that I read about all of those studies before. And then you referenced thinking fast and slow in your audible promo and it made sense.

  6. I think it was Francis Galton…my favorite mathematician who postulated this.. "the measurer" because he walked around with a tape measure and was astonished by the various measurements throughout nature.. he was a quirk. But here's the best analogy…at least how Galton thought of it, if two parents are 6ft, and continue to have children that are increasingly taller.. eventually there will be a biological regression that's is explicitely observable over several generations forward as their children don't continue to raise their avarage expected height, but rather oscilate back to a standard global mean.

  7. The word "regression" i always thought interesting use of vocabulary, since you could probably observe many iterations of a controlled probabilistic bernoulli event, and while you could see some regression over time as n -> inf, and Coeficient of variation -> 0; i doubt that there could be any linear or non-linear lognormal or other type of regression line fitted to the cummulative sample mean. Just imagining the variation in results … your essentially putting a line thrift a series of zigzags for lack of better term.. not a very good regression..i guess if you compound the observations into the 10,000+ eventually your credibility could be reduced to immaterial value.

  8. Side topic: The gamblers fallacy confuses the sample size – if you were to flip a coin 100 times trying to get heads just once, the odds do add up that in this overall sequence of 100 flips you will get heads. But if the sequence is just one coin flip the odds of previous flips in previous sequences don't add up – gamblers confuse their continuous experience of prior flips as being the terms for the current sequence of flips. But if the sequence is 100 flips, dude, that adds up (still not a sure thing, but really high chance of getting a single heads in there). Odds (inside a sequence) do add up.

  9. I don't think this is accurate. Feedback absolutely affects how you attempt something on the second occasion. Yes, the RESULTS might be affected by external factors two times in a row. However attempting a challenge is not always about the results in the short run, but more importantly the long run. Transparent and reliable constructive criticism, in the long run, will bring out a better result. I guess you should really ask yourself what the end goal and timeframe is.

  10. Now how come we observe regression to the mean AFTER the speed camera's been installed? I mean if on average there are X accidents pre-camera, and we observe Y<X accidents post-camera… Is it just RTM ?

  11. The purpose of education is to minimize the effect of luck. The purpose of praise and criticism is to help students identify areas of weakness and strength.

    There should not be a need to randomly guess on a properly designed assessment, students should have received instruction and practice on the material prior to the test day. In the case of the random test there is nothing they can do to improve their performance on the next test, so feedback would be irrelevant.

    An increase in testosterone might help you improve your rugby performance but it is unlikely to improve your Algebra performance.

  12. Technically it is not about how we are fooled by statistics…it is just that the researcher didn't understand statistics. You can in theory use Bayesian conditional probability formula to calculate the actual probability. It is not so much regression to the mean, but rather by not using the original population and reporting the probability of a group scoring bottom 10% given that they scored bottom 10% previously.

    1. This statement is a lie.
    2. Veritasium is a real element.
    3. The fox says: Hatee-hatee-hatee-ho!
    4. The universe is infinite.
    5. "Many worlds" is the correct interpretation of Quantum Mechanics.
    6. Theres 6 continents on Earth.
    7. The repeating decimal 0.9999999… = 1.
    8. There are an infinite number of Twin Primes.
    9. Every even integer above 3 is the sum of two primes.
    10. Positive feedback is better than negative feedback.

  14. "we are hardwired to see patterns and causality everywhere"…this is why science and mathematics [when properly done], has changed the world. Unfortunately, religion and gut feel and click-bait and emotional appeals and media focus on sharks and killer bees and the Kardashians keeps holding us back…anti-vaxers in 2021…causing us to die

  15. LOL… randomness on the macro level is a correct assumption. Eventually all things slide into entropy and "even out". But on a micro level, human beings counteract entropy all the time. A classroom full of students may regress to "the" mean, but an individual will regress to his or her own personal mean, and then only so long as he or she does not apply cognition and effort to improve.

    How about a look at those gamblers. The simple fact is that no matter how well you play games in Vegas, if you play long enough the house always wins. BUT… if you play badly enough, you can most definitely lose worse and more consistently than the "average" player. And if you study the game and play as efficiently as possibly, you can most definitely lose more slowly than that same "average" player. In blackjack, the house edge is only .5%, which is the amount that is "baked in" to the game. Even a very good player will eventually lose at that rate. BUT, most players are not that skillful. Those players lose at a rate that is considerably higher than the house edge. They do not regress to the mean – they regress to their own level of skill (or lack thereof, if you prefer). Some at or approaching the house edge, and some considerably worse.

    I'm sorry, but you're just misleading people on this one. Each player may regress to his or her personal mean, based on skill. But the mean of the means will be a composite of those different skill levels, and not at all meaningful for predicting the outcome of any individual player or even group of players. And skill levels may well change over time, as players study and improve, which means that maintaining the "mean" depends totally on a new and constant supply of new, unskilled players.

  16. Don't really like the use of the word luck in this context, since it doesn't really exist. I get that you're trying to convey an idea, but this terminology used to bridge the gap is more misleading than helpful.

  17. Two facts: The Israeli military strategy involves a lot of regressing to be mean. The second is that the typical police/military presence after a suspected terrorist bomb blast is largely unneccesary.

  18. As a general rule I mix and match positive and negative feedback for my students about their progression and not their level, for me it's always putting them in the mean of an unknow scale and judging how they progress, I also promote self-feedback so my students are more able think of their progression and to push themselve to the limit they define.

  19. Showing an A-10 toy/model when talking about Israeli fighter pilot training really broke the immersion for me. Still a good video overall.

    For those who are unaware, not only is the A-10 only used by the USAF, it's an "attack plane", not a fighter (ground attack, not air attack). You would never intentionally engage a real fighter (like an F-16) in a dogfight with such a thing – ideally, your attack place would be escorted by a proper fighter. That being said, at low levels, an A-10 can be more agile than a fighter, but partially because it's much slower (and for other reasons), but an actual fighter wouldn't get in a turn-and-burn dogfight with it but would probably snipe it from dozens of miles away with a radar-guided missile, or if it had to get close, it would try to "zoom-and-boom" without even being detected (since the A-10 has no radar). The A-10 is capable of air-to-air combat, usually against helicopters, but this is not its primary mission.

  20. The difference between the coin and the pilots is that the coin tosses are (theoretically) independent, that is, past tosses don't affect the outcome of future tosses, whereas the pilots' performances obviously depend on the pilot, so future performances by a pilot are expected to be dependent of past performances by the same pilot. That's the difference between the gambler's fallacy and regression to the mean.

  21. This should be the test for first grade. You pass this you pass first grade. Maybe the would greatly
    reduce the manipulation by companies, politicians, scientists… you name it.
    I watch your stuff and this is the most informative video that could actually change the world
    for everybody!

  22. Great hypothesis…also, though, consider the role of the Ammygdala in conditioning, and repsonse to a negative environmental stimulus.

    Statistics are an important variable, but the neurobiology must be included in understanding the other variables of learning, and therefore the unique learning of each individual, or, in this case, the group.

    In this particular group of 'learners', fighter pilots are a special group because they're conditioned and trained to perform in a stressful and dangerous environment. This means the ammygdala will be stimulated by stessful situations so that the brain responds adequately and therefore can perform to the level of performance required to both survive, and execute fighter maneuvers in such a danagerous environment. The fighter pilots' ammgdala will respond more effectively, therefore, to negative stimuli, and therefore act cognitively to achieve the goal of improvement to maintain a successful 'kill ratio'….the work 'kill' does not necessarily mean they 'kill' the enemy, it means they have put the 'enemy' fighter pilot out of the fight…either by better maneuverability, successful locks on target (intimidating the enemy) or , unfortunately, a succesful splash…tactical jet goes down. They practice these skills continuesly, Maple Flag exercises in Canada, and Red Flag….both are aerial combat trianing engagements which include the 'aggressor' usually an F16, or sometimes a Sukhoi…as the jet to be targeted and successfully neutralized.

  23. I first heard of the Israeli fighter pilot tale in "The Drunkard's Walk" by Leonard Mlodinow, it's an excellent read – I may have to pick up Thinking, Fast and Slow now too.