This series of studies set out to explore various ways that artificial intelligence (AI) could be integrated into higher education. Two main streams of research evolved from this initial concept. The first involved a longitudinal design-based research project that integrated generative AI into a 3D virtual simulation for teacher training. Our team examined how trainees interacted with AI-powered student agents using discourse analysis and think-aloud protocols.
Below is a brief tutorial video of the simulation software our team developed.
The second research stream investigated the efficacy of using ChatGPT and other large language models (LLMs) to automatically evaluate and assess English L2 writing. Our team designed a prompt and used it to provide both rubric-based and narrative feedback on student writing samples throughout a semester, closely monitoring student perceptions. We also compared multiple LLMs in this task longitudinally to get insights into the validity and reliability of LLM-basd automatic essay evaluation.
Highlights
- Preservice teachers’ use of ambitious talk moves leads to academically productive discourse with AI-powered student agents.
- Fine-tuning an LLM with authentic student discourse data leads to large similarities in simulation-base discourse, although AI-student agents tend to ask significantly more questions than human students do.
- LLMs are as reliable as humans in rubric-based essay evaluation.
- English L2 students perceive benefits to both LLM feedback as well as human feedback.
Next Steps
- How do AI-powered student agent’s affective profiles impact preservice teacher discourse interactions?
- How can hallucinations be reduced in LLM essay evaluation?
Representative Publications
Barrett, A., Ke, F., Zhang, N., Dai, C.-P., Bhowmik, S. & Yuan, X. (2025). Pattern analysis of ambitious science talk between preservice teachers and AI-powered student agents. The 15th International Learning Analytics & Knowledge Conference (LAK25), Dublin, Ireland. https://doi.org/10.1145/3706468.3706570
Barrett, A., Ke, F., Zhang, N. & Dai, C.-P. (2024). Comparing the science talk of AI and human students. Proceedings of the 18th International Conference of the Learning Sciences (ICLS), 2073-2074. https://doi.org/10.22318/icls2024.963739
Barrett, A. & Pack, A. (2023). Not quite eye to A.I.: Student and teacher perspectives on the use of generative artificial intelligence in the writing process. International Journal of Educational Technology in Higher Education, 20(59). https://doi.org/10.1186/s41239-023-00427-0
Barrett, A. J., Quaid, E. & Pack, A. (2021). Understanding learners’ acceptance of high-immersion virtual reality systems: Insights from confirmatory and exploratory PLS-SEM analyses. Computers & Education, 169, 104214. https://doi.org/10.1016/j.compedu.2021.104214
Works-in-Progress
Barrett. A., Ke, F., Zhang, N., Dai, C.-P., Bhowmik, S., Yuan, X. & Southerland, S. (Under Review). Preservice teachers’ dialogic interactions with AI-powered student agents: Patterns and perceptions.
Alex Barrett 