AI Sychophancy with Steve Rathje

This week Dan and Nik sit down with Steve Rathje, a new HCII colleague who runs CMU's Psychology of Technology Lab, to talk about what sycophantic AI does to the people using it. Steve's team randomly assigned people to chat with bots that were flattering, disagreeable, or neutral. People liked the flatterers, shrugged at the neutral ones, and hated the bots that pushed back.

For designers, the news is mixed. Validation drives enjoyment, but cherry-picked facts do the persuading, so pairing a little warmth with opposing evidence makes disagreement easier to take (at some cost to its persuasive punch). The bad news: across six studies from three research teams, teaching people about sycophancy through warnings, videos, or quizzes made the chatbot seem less trustworthy but no less persuasive. That puts the little "AI can make mistakes" disclaimer in an unflattering light. We also cover AI dark patterns, whether the right dose of sycophancy depends on the task, and Dan's random idea of a group chat of dueling AI personalities.

Plus, Nik explains Jev and the new class of fast, cheap "System One" decision models, and we look at pieces from Microsoft and DoorDash design teams on why AI evals should measure what users experience as well as what models get right.

LINKS

Jev decision model touted as quicker, cheaper LLM alternative (TechTarget)
https://www.techtarget.com/it-infrastructure/news/366650696/Jev-decision-model-touted-as-quicker-cheaper-LLM-alternative

Putting the user back into AI Evaluation (Microsoft Design)
https://microsoft.design/articles/putting-the-user-back-into-ai-evaluation/

Building AI Evals: Who Gets to Decide What’s Good? (Doordash Design)
https://medium.com/design-doordash/building-ai-evals-who-gets-to-decide-whats-good-39671385f50d

Sycophantic AI increases attitude extremity and overconfidence (Rathje et al.)
https://osf.io/preprints/psyarxiv/vmyek_v1/

Observing sycophantic AI validate others reduces its appeal but not its persuasiveness (Ye, Kraut & Rathje)
https://arxiv.org/abs/2607.25166v1

Sycophancy in GPT-4o: What happened and what we're doing about it (OpenAI)
https://openai.com/index/sycophancy-in-gpt-4o/

Silicon sycophants: the effects of computers that flatter (Fogg & Nass, 1997)
https://www.sciencedirect.com/science/article/pii/S1071581996901044

Insincere Flattery Actually Works: A Dual Attitudes Perspective (Chan & Sengupta, 2010)
https://journals.sagepub.com/doi/abs/10.1509/jmkr.47.1.122

On the conversational persuasiveness of GPT-4 (Salvi et al., Nature Human Behaviour)
https://www.nature.com/articles/s41562-025-02194-6

@Grok Is This True? LLM-Powered Fact-Checking on Social Media (Renault, Mosleh & Rand)
https://osf.io/preprints/psyarxiv/85quw_v1

Steve Rathje's website
https://stevenrathje.com/

Steve Psychology on TikTok
https://www.tiktok.com/@stevepsychology