Harmonizing Case Validation for Vaccine Safety in Real-World Data: The VAC4EU Experience

Purpose: Applying standardized Brighton Collaboration (BC) case definitions retrospectively to heterogeneous real-world data (RWD) is challenging due to inconsistent clinical detail and data structures across settings. To address this, the Vaccine Monitoring Collaboration for Europe (VAC4EU) developed a structured validation pipeline that operationalizes BC case definitions for vaccine safety outcomes in RWD in a harmonized, scalable, and reusable manner. Methods: VAC4EU developed a systematic, stepwise approach to validate vaccine safety outcomes. BC case definitions were utilized when available, adapted for RWD as needed, and newly developed for outcomes without an existing BC definition. The approach involves: 1) Critical review and adaptation of BC definition by clinical and RWD experts; 2) Creation of dummy cases based on published reports; 3) Creation of REDCap electronic data collection forms (eDCF) incorporating decision logic to assigned levels of certainty (LOC); 4) Iterative testing of decision logic; and 5) comprehensive training of abstractors with real-time feedback. A dedicated task force assigned reference LOCs for dummy cases. Inter-rater reliability was measured using Fleiss’ kappa (κ) by comparing abstractor LOCs to the reference standard. Results: The pipeline was applied to 16 COVID-19 vaccine safety outcomes, of which 13 had existing BC definition. In total, 78 dummy case descriptions were developed across the outcomes for training of abstractors, and 15 REDCap eDCFs were created. Myocarditis and pericarditis shared an eDCF. Across 33 abstractors, 747 dummy case abstractions were completed. Agreement analysis showed 93 discrepancies (12.4%) and moderate overall concordance (κ = 0.55, across all outcomes), with the lowest for thrombosis with thrombocytopenia syndrome (κ = –0.05). Conclusion: The VAC4EU validation pipeline provides a standardized framework for training and validating vaccine safety outcome in RWD. By adapting BC case definitions and dedicated training of abstractors, we will reduce variability in outcome validation in post-authorization safety studies.