DigitalCoach: Gaps in Human and AI Software Coaching

This episode examines DIGITALCOACH, a dataset of 72 human coaching sessions (22,752 dialogue turns across Excel, FL Studio, Blender, Figma and Onshape) built to test whether multimodal models can coach novices through real software rather than just automate it. It separates what coaches say (dialogue acts), how they teach (methods drawn from cognitive apprenticeship, such as explanation, diagnosis and elicitation), and how they tie guidance to the learner's screen. Human coaches orient and explain as well as direct, and about 27% of their turns carry annotations drawn on the learner's screen. Current models mostly give bare instructions like "click Extrude". The discussion also questions how far the machine-labeled distributions can be trusted, since the labels were validated on only 200 utterances. It then sets up the offline next-utterance comparison and the live study of whether novices actually learn. Sources: 1. DigitalCoach: Gaps in Human and AI Software Coaching https://arxiv.org/pdf/2606.31980 2. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments — Tianbao Xie, Danyang Zhang, Jixuan Chen, et al., 2024 https://scholar.google.com/scholar?q=OSWorld%3A+Benchmarking+Multimodal+Agents+for+Open-Ended+Tasks+in+Real+Computer+Environments 3. Mind2Web: Towards a Generalist Agent for the Web — Xiang Deng, Yu Gu, Boyuan Zheng, et al., 2023 https://scholar.google.com/scholar?q=Mind2Web%3A+Towards+a+Generalist+Agent+for+the+Web 4. Principles of Mixed-Initiative User Interfaces — Eric Horvitz, 1999 https://scholar.google.com/scholar?q=Principles+of+Mixed-Initiative+User+Interfaces 5. Executing Instructions in Situated Collaborative Interactions — Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, Yoav Artzi, 2019 https://scholar.google.com/scholar?q=Executing+Instructions+in+Situated+Collaborative+Interactions 6. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering — Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh, 2017 https://scholar.google.com/scholar?q=Making+the+V+in+VQA+Matter%3A+Elevating+the+Role+of+Image+Understanding+in+Visual+Question+Answering 7. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents — Kanzhi Cheng, Qiushi Sun, Yougang Chu, et al., 2024 https://scholar.google.com/scholar?q=SeeClick%3A+Harnessing+GUI+Grounding+for+Advanced+Visual+GUI+Agents 8. Visual Dialog — Abhishek Das, Satwik Kottur, Khushi Gupta, et al., 2017 https://scholar.google.com/scholar?q=Visual+Dialog 9. Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering — Aishwarya Agrawal, Dhruv Batra, Devi Parikh, Aniruddha Kembhavi, 2018 https://scholar.google.com/scholar?q=Don%27t+Just+Assume%3B+Look+and+Answer%3A+Overcoming+Priors+for+Visual+Question+Answering 10. Learning from Human Tutoring — Michelene T. H. Chi, Stephanie A. Siler, Heisawn Jeong, Takashi Yamauchi, Robert G. Hausmann, 2001 https://scholar.google.com/scholar?q=Learning+from+Human+Tutoring 11. The ICAP Framework: Linking Cognitive Engagement to Active Learning Outcomes — Michelene T. H. Chi, Ruth Wylie, 2014 https://scholar.google.com/scholar?q=The+ICAP+Framework%3A+Linking+Cognitive+Engagement+to+Active+Learning+Outcomes 12. The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems — Kurt VanLehn, 2011 https://scholar.google.com/scholar?q=The+Relative+Effectiveness+of+Human+Tutoring%2C+Intelligent+Tutoring+Systems%2C+and+Other+Tutoring+Systems 13. Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech — Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, et al., 2000 https://scholar.google.com/scholar?q=Dialogue+Act+Modeling+for+Automatic+Tagging+and+Recognition+of+Conversational+Speech 14. Generative AI Can Harm Learning — Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı, Rei Mariman, 2024 https://scholar.google.com/scholar?q=Generative+AI+Can+Harm+Learning 15. Evaluating Human-Language Model Interaction — Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, et al., 2022 https://scholar.google.com/scholar?q=Evaluating+Human-Language+Model+Interaction 16. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference — Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, et al., 2024 https://scholar.google.com/scholar?q=Chatbot+Arena%3A+An+Open+Platform+for+Evaluating+LLMs+by+Human+Preference 17. Studying the Effect of AI Code Generators on Supporting Novice Learners in Introductory Programming — Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, Tovi Grossman, 2023 https://scholar.google.com/scholar?q=Studying+the+Effect+of+AI+Code+Generators+on+Supporting+Novice+Learners+in+Introductory+Programming 18. Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems: The Relative Effectiveness of Tutoring (VanLehn's review) — Kurt VanLehn, 2011 https://scholar.google.com/scholar?q=Human+Tutoring%2C+Intelligent+Tutoring+Systems%2C+and+Other+Tutoring+Systems%3A+The+Relative+Effectiveness+of+Tutoring+%28VanLehn%27s+review%29 19. MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors — Jakub Macina et al., 2025 https://scholar.google.com/scholar?q=MathTutorBench%3A+A+Benchmark+for+Measuring+Open-ended+Pedagogical+Capabilities+of+LLM+Tutors 20. From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning — David Dinucu-Jianu, Jakub Macina, Nico Daheim, Ido Hakimi, Iryna Gurevych, Mrinmaya Sachan, 2025 https://scholar.google.com/scholar?q=From+Problem-Solving+to+Teaching+Problem-Solving%3A+Aligning+LLMs+with+Pedagogy+using+Reinforcement+Learning 21. Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach (LearnLM) — Irina Jurenka et al. (Google), 2024 https://scholar.google.com/scholar?q=Towards+Responsible+Development+of+Generative+AI+for+Education%3A+An+Evaluation-Driven+Approach+%28LearnLM%29 22. Vid2Coach: Transforming How-To Videos into Task Assistants — Mina Huh, Zihui Xue, Ujjaini Das, Kumar Ashutosh, Kristen Grauman, Amy Pavel, 2025 https://scholar.google.com/scholar?q=Vid2Coach%3A+Transforming+How-To+Videos+into+Task+Assistants 23. Can Multi-modal LLMs Provide Live Step-by-Step Task Guidance? — Apratim Bhattacharyya et al., 2025 https://scholar.google.com/scholar?q=Can+Multi-modal+LLMs+Provide+Live+Step-by-Step+Task+Guidance%3F 24. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use — Kaixin Li et al., 2025 https://scholar.google.com/scholar?q=ScreenSpot-Pro%3A+GUI+Grounding+for+Professional+High-Resolution+Computer+Use 25. Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task — Nataliya Kosmyna et al., 2025 https://scholar.google.com/scholar?q=Your+Brain+on+ChatGPT%3A+Accumulation+of+Cognitive+Debt+when+Using+an+AI+Assistant+for+Essay+Writing+Task 26. Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation — Xinyi Lu, Xu Wang, 2024 https://scholar.google.com/scholar?q=Generative+Students%3A+Using+LLM-Simulated+Student+Profiles+to+Support+Question+Item+Evaluation Interactive Visualization: DigitalCoach: Gaps in Human and AI Software Coaching