5 Insights Learned From Psychology’s Most Studied Brain Teaser

If you think every new psychology experiment is different from the last, you’re sadly mistaken. The truth is that certain experimental protocols are often studied for years, even decades. Part of this is researchers’ laziness (it’s hard to invent new experimental protocols for every new study) and part has to do with the incremental nature of scientific discovery (it’s often easier to build upon an existing protocol to prove a point than to create something new).

One popular experimental protocol is called the “Cognitive Reflection Task” or, CRT, for short. The CRT has been used extensively to study intelligence, decision making, intuition, and a host of other topics. For instance, people’s scores on the CRT have been shown to correlate with risk preferences, patterns in statistical reasoning, and general intelligence.

Of course, the best way to learn is by doing. Complete the CRT below to see how you measure up to the thousands of people who have taken the test since it was first proposed in 2005. (Answers are provided at the end of the article.)

  1. A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
  2. If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?
  3. In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?

If you’ve reviewed the answers, you might notice that the CRT assesses people’s ability to forgo intuitive yet incorrect responses in favor of correct answers that require a deeper level of thinking.

A new study published in the Journal of Behavioral and Experimental Economics examined over a decade’s worth of research on the CRT (44,558 tests taken in 21 different countries, to be specific) to extract insights from psychology’s most popular brain teaser. Here’s what they found.

  1. How do people score? If you didn’t score well, don’t be alarmed. This is a difficult test. Approximately 40% of people don’t get any of the three questions right and approximately 20% of people get either one, two, or all three questions correct.
  2. Which question is easiest/most difficult? The “lillypad” question is the easiest of the three, with over 45% of people answering correctly. The “machine” question is second easiest and the “bat and ball” brain teaser is most difficult (just over 30% of people get this question right). Notice that none of the questions have higher than a 50% accuracy rate. Again, it is no stretch to say that this is a tough test.
  3. Is there a gender bias? Yes. On average, men outperform women on the CRT to a statistically significant degree. The authors write, “We find that: (i) males perform better in every single question, (ii) females are more likely to answer none of the questions correctly, and (iii) males are more likely to answer all three questions correctly. Importantly, gender differences persist even when we control for test characteristics (for example, monetary incentives, computerized, student samples, positioning of the experiment, etc.).”
  4. Does the ordering of the questions affect accuracy? No. The researchers did, however, find an effect of experimental fatigue. That is, participants who were asked to complete the CRT after completing other tasks showed decreases in accuracy. The researchers suggest this has to do with the natural depletion of glucose levels in the brain that occurs after completing effortful tasks.
  5. Do incentives increase accuracy? The effects of monetary incentives on human performance has been a hotly debated topic in psychology and economics. Evidence from CRT studies shows no clear effect of monetary incentives on performance. The researchers write, “paying subjects for correct answers on the CRT does not increase performance levels.”

The researchers also found that students outperformed non-students and that test accuracy was slightly higher when the test was administered on a computer versus pen and paper.

It will be interesting to see what the next decade has in store for research on the CRT.

Answers to the CRT:

  1. Intuitive answer 10, correct answer 5
  2. Intuitive answer 100, correct answer 5
  3. Intuitive answer 24, correct answer 47

Follow me on Twitter or LinkedIn.

Mark Travers, Ph.D., is a social psychologist by training and a writer by profession. He received his Ph.D. and M.A. in social psychology from the University of Co...