Evaluating Large Language Models in the Assessment of Collaborative Problem-Solving Skills: Evidence from Classroom Dialogue

Resumen de los autores

The increasing use of large language models (LLMs) is reshaping how collaborative learning processes are analysed and assessed. In collaborative problem-solving (CPS), assessment commonly relies on human coding of dialogue for fine-grained social and cognitive skills. However, such judgments are difficult to scale, and their construct clarity is rarely examined. While recent studies suggest that LLMs may support qualitative coding of collaborative skills, limited research has examined whether different models apply CPS frameworks consistently or how their reasoning aligns with expert human judgments. To address these gaps, this study applies an ensemble of LLMs to code the presence of CPS subskills, defined by an established CPS framework, in transcripts of collaborative dialogue from CPS contexts. Our study shows that LLM-human alignment is achievable for some subskills but varies systematically across the framework. Analysis of model and human-generated rationales indicates that points of model disagreement correspond to persistent ambiguities in human coding rather than isolated model error. These findings highlight how LLM ensembles can be used not only to support CPS coding but also to diagnose the clarity and evidentiary limits of dialogue-based CPS frameworks. By synthesising agreement, disagreement, and model reasoning, this work positions LLM ensembles as transparent decision-support tools for educational research, offering a foundation for refining CPS frameworks and guiding responsible use of LLMs in collaborative learning assessment.

Áreas y palabras clave

Cómo citar (APA 7)

Zamecnik, A., de Laat, M., Claassen, A., & Baker, R. S. (2026). Evaluating Large Language Models in the Assessment of Collaborative Problem-Solving Skills: Evidence from Classroom Dialogue. https://doi.org/10.35542/osf.io/xh27a_v1