Theme
7. Digital transformation, Artificial Intelligence, and Systems Metrology
Conference150th anniversary of the Metre Convention — From Units to the Universe
Year2025
LocationPalais des congrès de Versailles, Versailles, France
PublisherBureau International des Poids et Mesures (BIPM)
LanguageEN
ABSTRACT
The adoption of machine learning (ML) and, more specifically, deep learning (DL) applications into all major areas of our lives is underway. However, the race for implementation and utilisation is currently outpacing comprehension of the technology. The complex and black box character of AI applications entails safety, security and privacy concerns. A lack of confidence in the transparency of AI prevents its utilisation for society and economic growth. It can lead to a slowed adoption of innovations and discourage innovators from unlocking the technology’s full potential. Hence, the demand for an improved understanding and validation of AI systems is ever increasing. Metrological evaluation of AI supports this effort, aiming to ensure reliability, trust and safety.
The development of trustworthy AI is especially important in medicine due to the large implications for patients’ lives. While trustworthiness concerns various aspects including ethical, transparency and safety requirements, we focus on the importance of data quality (training/test) in DL. Since data quality dictates the behaviour of ML products, evaluating data quality will play a key part in the regulatory approval of high-risk AI medical devices as required by the EU AI Act.
We have performed a systematic review following PRISMA guidelines in order to synthesise the existing knowledge on data quality frameworks and combine it with the perspective of ML applications in medicine. As a result, we propose the METRIC-framework, a specialised data quality framework for medical training data comprising 15 awareness dimensions divided into five clusters: measurement process, timeliness, representativeness, informativeness, consistency. Developers of medical ML applications should investigate the content of a dataset along this system. This knowledge helps to reduce biases as a major source of unfairness, increase robustness, facilitate interpretability and thus lays the foundation for trustworthy AI in medicine.
The mandatory AI-specific conformity assessment described in the EU AI Act (Articles 8 to 15) lays down data quality requirements (Article 10, paragraphs 3 and 4). It tasks, among others, metrology institutes with developing measurement methodologies for technical validation (Article 15, paragraph 2). We find the AI Act data requirements to map to the METRIC-framework in a straightforward, distinct and comprehensive way. Evaluations of datasets that utilize the METRIC-framework thus also show compliance with the EU AI Act. The METRIC-framework is therefore ideally positioned to serve as a base for systematically assessing training datasets during audits and conformity assessment, for establishing reference datasets, and for designing test datasets. The harmonization of such evaluations has great potential to accelerate the approval of AI medical devices.
Currently, the characteristics of the METRIC-framework are being equipped with quantitative metrics to enable operationalization of automatic data quality evaluation tests. The subsequent establishing of a seal of approval for datasets provided through neutral organizations, such as metrology institutes, aims to provide trust in the technology as well as to facilitate market access. Furthermore, our main task is to systematically investigate and quantify the influence data quality characteristics of the METRIC-framework have on characteristics of the AI system, including predictive performance, robustness, uncertainty, fairness and explainability.
CITE THIS POSTER
Include:
Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schäffter, T.. (2025). On the path to quality assurance of trustworthy AI in medicine: The METRIC-framework for systematically assessing medical training and test data. 150th anniversary of the Metre Convention — From Units to the Universe. Palais des congrès de Versailles, Versailles, France. Bureau International des Poids et Mesures (BIPM). https://doi.org/10.59161/IHSB2449
Schwabe, Daniel, et al.. “On the path to quality assurance of trustworthy AI in medicine: The METRIC-framework for systematically assessing medical training and test data.”. 150th anniversary of the Metre Convention — From Units to the Universe, 2025, Palais des congrès de Versailles, Versailles, France, Bureau International des Poids et Mesures (BIPM). DOI: 10.59161/IHSB2449
Schwabe, Daniel, et al.. 2025. “On the path to quality assurance of trustworthy AI in medicine: The METRIC-framework for systematically assessing medical training and test data.”. 150th anniversary of the Metre Convention — From Units to the Universe, Palais des congrès de Versailles, Versailles, France. Bureau International des Poids et Mesures (BIPM). https://doi.org/10.59161/IHSB2449
@inproceedings{schwabe2025on_the_path_to_quali,
author = {Schwabe, Daniel and Becker, Katinka and Seyferth, Martin and Klaß, Andreas and Schäffter, Tobias},
title = {On the path to quality assurance of trustworthy AI in medicine: The METRIC-framework for systematically assessing medical training and test data},
booktitle = {150th anniversary of the Metre Convention — From Units to the Universe},
year = {2025},
address = {Palais des congrès de Versailles, Versailles, France},
publisher = {Bureau International des Poids et Mesures (BIPM)},
doi = {10.59161/IHSB2449},
url = {https://thebipm150.org/poster-page/10euhzi},
}
TY - CONF
AU - Schwabe, Daniel
AU - Becker, Katinka
AU - Seyferth, Martin
AU - Klaß, Andreas
AU - Schäffter, Tobias
TI - On the path to quality assurance of trustworthy AI in medicine: The METRIC-framework for systematically assessing medical training and test data
T2 - 150th anniversary of the Metre Convention — From Units to the Universe
PB - Bureau International des Poids et Mesures (BIPM)
PY - 2025
CY - Palais des congrès de Versailles, Versailles, France
DO - 10.59161/IHSB2449
UR - https://thebipm150.org/poster-page/10euhzi
ER -