Designing for Qualitative Evaluation of Synthetic Medical Data

Machine learning in healthcare often struggles with data access for model training due to privacy restrictions, rare conditions, and high acquisition costs. Synthetic data offers a potential workaround, yet there are no agreed-upon gold standards for evaluating it. As quantitative metrics alone cannot fully assess the desired qualities of generative model outputs, human inspection is a key component of validation, warranting a “Doctor-in-the-loop” approach. However, research is scarce on best practices for interaction and user interface design in such systems. This paper presents preliminary designs for qualitative synthetic medical data evaluation, informed by four participatory workshops with seven doctors and nine machine learning engineers. Spanning tabular, image, and time series data, this study emphasised transparency and clear communication of the synthetic data generation. In addition to presenting the rationale behind the evaluation workflow design, we highlight challenges in the medical domain, including doctors’ limited familiarity and skepticism with synthetic data.