Towards AI epidemiology: a measurement standardisation framework for prospective risk detection
Kit Tempest-Walters
Why It Matters
What makes this one worth your time
This framework could enhance the reliability and governance of AI systems, providing a basis for monitoring and improving alignment in various applications.
A framework for standardizing measurements in AI risk detection.
Summary
The paper proposes a measurement standardisation framework aimed at structuring expert-AI interactions for risk detection in AI systems, outlining a protocol for future empirical testing.
Key contributions
- Definition of a measurement standardisation framework for expert-AI interactions.
- Specification of a statistical protocol for empirical testing of the framework.
- Establishment of reliability, governance, and epidemiological claims related to AI alignment.
Notable insights
- The framework compresses expert-AI interactions into structured fields, which may facilitate better comparisons across different AI deployments.
- The introduction of 'AI epidemiology' suggests a novel approach to understanding AI risks through correlated variables rather than traditional mechanistic analysis.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2512.15783v3 Announce Type: replace Abstract: This paper proposes a measurement standardisation framework that compresses expert-AI interactions into structured, comparable fields for prospective risk detection in deployed AI systems, without access to model internals. The main aim of this concept paper is to define the scope of the framework, both semantically and statistically, and to specify a protocol for its empirical testing in future work. The population-level claims the framework is designed to support are therefore the subject of a staged research programme rather than results claimed in this paper. Measurement standardisation underpins all three claims that follow. The first is a reliability claim: under bounded conditions, large language models can produce reliable, standardised assessments of the evidential and policy alignment of expert-AI interactions. The second is a governance claim: alignment scores give experts an immediate signal during deployment and give institutions a basis for monitoring alignment patterns across mission types, models, and domains. The third is an epidemiological claim: once measurement standardisation is established, aggregate alignment scores could be used to study associations with downstream outcomes in regulated professional settings. This introduces the possibility of an "AI epidemiology" that detects risk based on correlated variables instead of mechanistic analysis. This paper addresses the first claim and specifies protocols for investigating the second and third. To enable empirical evaluation in future studies, this paper sets out a defined grammar, together with a statistical protocol based on paired bootstrap inference, DeLong's test for paired AUCs as a sensitivity check, a pre-specified one-sided non-inferiority margin of 0.05, and Holm-Bonferroni correction.