AERC Hosts: Dr Thorben Jansen "Writing with Generative AI: Studying AI-Supported Writing Processes and Products in PISA 2025"

Generative AI tools are increasingly available to students during writing, yet there is little large-scale evidence on how such tools change writing processes, outcomes, and equity in assessment contexts. This talk presents the study design and first analytic steps of an experimental extension to the foreign language assessment in PISA 2025, focusing on writing in English as a foreign language in a sample of 1700 students from Germany.

In the study, students complete two computer-based writing tasks: one without AI access and one with the opportunity to use generative AI to support planning, drafting, and revising. The design allows for within-student comparisons while linking writing performance and process data to the rich background, competence, and motivational measures available in PISA. During the assessment, writing processes are captured through detailed log data, including keystrokes, timing information, and interactions with the AI system.

The talk outlines the rationale behind the task design, the integration of AI into a standardized assessment environment, and the analytic strategy for evaluating both writing products and processes. I will discuss how writing quality is assessed using automated scoring approaches, how process indicators are derived from log data, and how early analyses examine shifts in planning, drafting, and revising when AI is available. The presentation concludes by highlighting open methodological and conceptual questions, particularly regarding validity, comparability, and educational equity, when generative AI becomes part of students’ writing practices and assessment.