Research Article: An external peer review instrument for Korean Medicine clinical practice guideline drafts (KM-EPRI): development and content validation
Abstract:
Korean Medicine clinical practice guidelines (KM-CPGs) undergo mandatory external peer review of their drafts during development, yet reviewers lack a standardized, validated instrument to guide this task. Widely used appraisal tools assess the quality of completed guidelines rather than support the section-level review of a draft. We aimed to develop an external peer review instrument for KM-CPG drafts (KM-EPRI) and to assess its content validity.
In this two-phase methodological study, a draft instrument was first derived by directed qualitative content analysis of the external review comments produced for previously published KM-CPGs, with each comment coded into one of four predefined review modules: the clinical questions (CQ), the introductory components (CPG-I), the development methods (CPG-M), and the recommendations (CPG-R). The content validity and acceptability of the items were then assessed through a modified Delphi study with a multidisciplinary panel of KM clinicians and evidence-based medicine methodologists. Consensus was defined a priori as an item-level content validity index (I-CVI) ? 0.80. The scale-level content validity index (S-CVI/Ave), Kendall’s coefficient of concordance ( W ), and the interquartile range (IQR) were reported as supporting measures.
From 3,630 external review comments, 2,073 content-related comments were consolidated and refined into a finalized instrument of 117 review items across 4 modules and 33 subdomains. All 10 invited experts participated (response rate 100%, no attrition), and every item reached the consensus threshold in a single round under a pre-defined early-stopping rule. All 117 items had an I-CVI ? 0.80, the S-CVI/Ave was 0.976, and the IQR was ? 1 for every item (median 0.75). Kendall’s W was low (0.196), reflecting a ceiling effect from uniformly high ratings rather than disagreement among panelists.
KM-EPRI can serve as a consensus-based, structured reference for the external peer review of KM-CPG drafts during development. Its effect on review quality and inter-rater reliability remains to be evaluated in practice.
Introduction:
Korean Medicine clinical practice guidelines (KM-CPGs) undergo mandatory external peer review of their drafts during development, yet reviewers lack a standardized, validated instrument to guide this task. Widely used appraisal tools assess the quality of completed guidelines rather than support the section-level review of a draft. We aimed to develop an external peer review instrument for KM-CPG drafts (KM-EPRI) and to assess its content validity.
Read more