Statistical Sleuthing: Can you spot a problem?

Can you spot problems in a scientific study just by reading the published paper? We put our statistical sleuthing skills to the test with a famous study on procrastination and deadlines that was retracted more than 20 years after it was published. But were there clues hidden in plain sight in the published results—and would we have noticed anything suspicious at the time? We put on our data-detective hats and find a few statistical clues of our own before turning to Data Colada’s investigation of the original data, where the story gets even stranger. Along the way, we hunt for missing participants, question suspiciously enormous effects, meet some unlikely data twins, and discover why sometimes the best statistical tool is simply asking: Do these data behave like real people?

Statistical topics

• Statistical sleuthing
• Effect sizes
• Degrees of freedom
• Replication
• Missing data

Methodologic Morals

• “A data detective's toolbox is surprisingly simple. Check the degrees of freedom, look for twins, question giant effects, and ask whether the data actually make human sense.”
• “You don't have to be a statistician to catch a statistical criminal.”

References

• Data Colada blog posts:
  • https://datacolada.org/138
  • https://datacolada.org/139
• Data and code: https://researchbox.org/7135
• Retraction Watch article: https://retractionwatch.com/2026/09/03/procrastination-study-duke-dan-ariely-psychological-science-data-colada-tampering-retraction/
• Original 2002 paper: https://journals.sagepub.com/doi/10.1111/1467-9280.00441
• Retraction notice: https://journals.sagepub.com/doi/10.1177/09567976261488042
• Hyndman K, Bisin A. Replication of "Procrastination, Deadlines, and Performance: Self-Control by Precommitment". Psychol Sci. 2026;37(8):557-571. doi:10.1177/09567976261460772

Kristin and Regina’s online courses:

Demystifying Data: A Modern Approach to Statistical Understanding

Clinical Trials: Design, Strategy, and Analysis

Medical Statistics Certificate Program

Writing in the Sciences

Epidemiology and Clinical Research Graduate Certificate Program

Program that Kristin teaches in::

Epidemiology and Clinical Research Graduate Certificate Program

Kristin’s manuscript writing course:

https://instats.org/seminar/from-data-to-discussion-writing-a-scient

Find us on:

Kristin - LinkedIn & Twitter/X

Regina - LinkedIn & ReginaNuzzo.com

• (00:00) - Introduction
• (02:15) - The Procrastination Paper and Study Two
• (08:51) - Suspiciously Tidy Results
• (16:18) - Study One: Real Classroom Stakes
• (20:06) - Missing Data and First Red Flags
• (28:17) - How Data Colada Got the Data
• (33:43) - When the Replication Failed
• (35:36) - Red Flags: Data Twins and Sanity Checks
• (47:21) - Study One Gets the Spreadsheet Treatment
• (51:45) - The Grading Scheme Unravels
• (57:51) - Manuscript Versions Reveal the Truth
• (01:08:11) - Rating the Claim and Methodological Morals