Real-world Inter-rater Agreement of PI-QUAL Version 2 for Prostate Magnetic Resonance Imaging Quality Assessment and Its Association with Diagnostic Accuracy

Background and objective: Magnetic resonance imaging (MRI) has been shown to improve the detection of prostate cancer, the second most diagnosed cancer among men. This study evaluates the inter-rater agreement for MRI quality using Prostate Imaging Quality (PI-QUAL) v2. It studies the association of PI-QUAL with diagnostic accuracy and the proportion of indeterminate Prostate Imaging Reporting and Data System (PI-RADS) 3 lesions with PI-QUAL scores. Methods: This multicenter cohort study included biopsy-naïve patients from the PCAVISION trial (NCT06281769) who underwent MRI for a suspicion of prostate cancer. Four radiologists independently scored PI-QUAL v2 and PI-RADS. PI-QUAL v2 scores were dichotomized as PI-QUAL 1 versus PI-QUAL ≥2. PI-RADS was dichotomized as PI-RADS ≤2 versus PI-RADS ≥3 and evaluated using percentage agreement, kappa, and Gwet's agree coefficient (AC) 1. The PI-QUAL consensus score was defined by the majority score or by a referee if no majority was reached. In cases with biopsy pathology, the association between PI-QUAL and diagnostic accuracy was assessed. Key findings and limitations: In total, 352 MRI scans were included, with 150 multiparametric MRI and 202 biparametric MRI scans. The percentage agreement for dichotomized PI-QUAL was 53% (95% confidence interval [CI] 51–56%) with AC1 of 0.11 (95% CI 0.06–0.16), mainly due to one reader who strictly applied technical criteria classifying 83% as inadequate (PI-QUAL 1), compared with 12–43% by others. Exclusion of this reader resulted in a percentage agreement of 69% (95% CI 65–73%) and an AC1 of 0.51 (95% CI 0.44–0.58). Consensus classified 24% of MRI scan as PI-QUAL 1 and 76% as PI-QUAL ≥2. In an exploratory analysis, the negative predictive value was 68% (95% CI 47–85%) for PI-QUAL 1 and 74% (95% CI 60–85%) for PI-QUAL ≥2, and the positive predictive value was 62% (95% CI 45–78%) and 50% (95% CI 42–59%), respectively. The proportion of PI-RADS 3 lesions was higher in PI-QUAL 1 scans than in PI-QUAL ≥2 scans (difference of –6.5%, 95% CI –11% to –1.4%, p = 0.011). Conclusions and clinical implications: Strict adherence to the technical PI-QUAL criteria led to poor inter-rater agreement, while visual-based assessment yielded moderate agreement. Importantly, higher PI-QUAL scores were linked to fewer indeterminate PI-RADS 3 lesions. Patient summary: We studied how doctors agree when judging the quality of prostate magnetic resonance imaging (MRI) scan using the Prostate Imaging Quality scoring system. When they focused too strictly on technical rules, agreement was poor; when scans were judged visually, agreement improved. Better-quality MRI scans led to fewer unclear (“indeterminate”) results, helping doctors make clearer biopsy decisions.