AERC Hosts: Dr Paul Jewsbury "Measurement science for AI-enabled, digital-first assessment"

As assessment becomes digital-first and AI-enabled, new possibilities emerge: adaptive testing at scale, automatically generated items, and evolving item banks. Yet the resulting scores must still meet the psychometric standards that consequential decisions depend on. In this talk, I examine the challenges this shift creates and the methods it demands, using the Duolingo English Test (DET), a high-stakes test of English, as a case study.

The DET is item-level adaptive and draws on a large item bank, with new items added regularly. Ongoing turnover and sparse per-item response data pose psychometric challenges, including substantial uncertainty in item parameters, and call for methods that are principled and powerful, yet computationally feasible and efficient. In response, the DET estimates these parameters within a hierarchical Bayesian model, using AI-based predictions from each item's linguistic features as empirical priors. Scoring accounts for this uncertainty, attenuating the impact of items with less precise parameters. I then describe recent advances in Bayesian computational psychometrics, including methods for screening and calibrating new items, and for updating item parameters as data accumulate. Together, these illustrate how measurement can remain rigorous as assessment becomes increasingly AI-enabled.