diff --git a/plugins/ce-datascience/README.md b/plugins/ce-datascience/README.md index fe0d5846..ab9a3585 100644 --- a/plugins/ce-datascience/README.md +++ b/plugins/ce-datascience/README.md @@ -1,6 +1,6 @@ # CE DataScience -Compound engineering for computational scientists. SAP management, statistical review, and full reporting guideline compliance for R and Python workflows — covering 16 standards across all study types and AI extensions. +Compound engineering for computational scientists. SAP management, statistical review, and full reporting guideline compliance for R and Python workflows — covering 35 standards across all study types and AI extensions. ## Getting Started @@ -127,7 +127,7 @@ Agents are specialized subagents invoked by skills. | `ce-methods-reviewer` | Statistical test selection and assumption verification | | `ce-multiplicity-reviewer` | Multiple comparisons, p-hacking, and selective reporting | | `ce-reproducibility-reviewer` | Seeds, package versions, paths, and environment specs | -| `ce-reporting-checklist-reviewer` | Reporting guideline compliance across 16 guidelines — auto-routes by study type, layers AI extensions, writes append-only compliance report (opt-in) | +| `ce-reporting-checklist-reviewer` | Reporting guideline compliance across 35 guidelines — auto-routes by study type, layers AI extensions, writes append-only compliance report (opt-in) | | `ce-sap-drift-detector` | Structural and semantic drift between SAP and analysis code; also flags blinding-state violations, missing amendment log entries, primary-endpoint changes after data lock, and code drift after amendments | | `ce-data-mapping-reviewer` | Codebook / SAP / extract column-mapping correctness — name drift, unit mismatches, level-set drift, derived-variable formulae, PHI in codebook | | `ce-phi-leak-reviewer` | HIPAA Safe Harbor identifier scan across data files, codebooks, notebooks, manuscripts, figure captions, and rendered output | diff --git a/plugins/ce-datascience/skills/ce-code-review/references/charms-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/charms-checklist.md new file mode 100644 index 00000000..3e0d03dd --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/charms-checklist.md @@ -0,0 +1,195 @@ +# CHARMS Checklist for Systematic Reviews of Prediction Modelling Studies + +CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS). +11-domain framework for data extraction and critical appraisal of studies reporting development, validation, or updating of clinical prediction models in systematic reviews. + +**Primary reference:** Moons KGM, de Groot JAH, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. 2014;11(10):e1001744. doi:10.1371/journal.pmed.1001744. PMID: 25314315. + +CHARMS is used alongside PRISMA when conducting systematic reviews of studies that develop, validate, or update prediction models. It guides both data extraction and critical appraisal. Each domain below describes the information to extract and appraise, and what to look for in analysis code and outputs when evaluating compliance during a code or analytic review of a prediction modelling systematic review. + +--- + +## Domain 1: Source of Data + +### Source of Data + +**Description:** Extract details about the data source used to develop or validate the prediction model, including study design, data collection method, and whether the data were collected prospectively or retrospectively. + +**What to look for in code/outputs:** +- Data extraction fields recording the data source type (randomized trial, prospective cohort, retrospective cohort, registry, electronic health record database, administrative claims) +- Code or notes distinguishing prospective data collection from retrospective use of existing data +- Documentation of the geographic setting, number of centers, and care setting (primary care, secondary care, emergency) for each included study +- Summary statistics describing the distribution of study designs across the systematic review's included studies +- Extraction of the time period over which data were collected for each study + +--- + +## Domain 2: Participants + +### Participants + +**Description:** Extract the participant eligibility criteria, recruitment method, and key baseline characteristics of the study population, and assess whether the study population is appropriate for the prediction model's intended use setting. + +**What to look for in code/outputs:** +- Inclusion and exclusion criteria for participants extracted and coded per study +- Code or notes documenting the setting and moment of intended model use (e.g., patients presenting to the emergency department with suspected sepsis) +- Baseline characteristic summaries extracted: age, sex, key clinical variables relevant to the prediction problem +- Assessment of whether the study population matches the target population in the review question +- Documentation of spectrum of disease severity and case mix in each included study + +--- + +## Domain 3: Outcome to be Predicted + +### Outcome to be Predicted + +**Description:** Extract the outcome variable being predicted, including its definition, measurement method, time horizon, and the handling of competing events. + +**What to look for in code/outputs:** +- Outcome variable definition extracted per study: clinical event, diagnosis, composite endpoint, or continuous measure +- Time horizon for prediction extracted (e.g., 30-day mortality, 5-year recurrence, in-hospital readmission) +- Measurement method for the outcome documented (administrative coding, clinical adjudication, registry linkage, laboratory confirmation) +- Competing events identified and their handling documented (e.g., censoring, cause-specific modeling, subdistribution hazard approach for competing risk outcomes) +- Assessment of whether the outcome definition is consistent with the review's target outcome and clinically meaningful +- Proportion of outcomes verified by blinded adjudication extracted and flagged when outcome verification was not blinded + +--- + +## Domain 4: Candidate Predictors + +### Candidate Predictors + +**Description:** Extract the candidate predictor variables considered during model development, including how they were measured, whether they were pre-specified or selected from the data, and any transformations applied. + +**What to look for in code/outputs:** +- Full list of candidate predictors considered during development extracted and coded per study +- Measurement method and timing of predictor assessment documented (e.g., vital signs at hospital admission, lab values from first blood draw) +- Documentation of whether candidate predictors were pre-specified based on clinical knowledge or identified by data-driven variable selection +- Continuous predictor handling extracted: categorization (and cut-points), linear assumption, spline or polynomial transformation +- Assessment of predictor availability at the intended moment of model use (some predictors may require results not available at decision time) +- Inter-predictor correlation or multicollinearity assessment documented when reported + +--- + +## Domain 5: Sample Size + +### Sample Size + +**Description:** Extract and critically appraise the number of participants and outcome events in each study, and assess whether the sample size was adequate for the number of predictors included in the model. + +**What to look for in code/outputs:** +- Number of participants, number of events, and events-per-variable (EPV) ratio extracted per study +- Code or notes flagging studies with EPV below commonly used thresholds (EPV < 10 for binary outcomes is a commonly cited concern; note that current guidance recommends events-per-parameter calculations over EPV thresholds) +- For validation studies: number of participants and events in the validation sample extracted separately from development sample +- Assessment of whether the sample size section provides a formal justification or power calculation +- Summary statistics describing the distribution of sample sizes and event counts across included studies + +--- + +## Domain 6: Missing Data + +### Missing Data + +**Description:** Extract how missing data were handled in each included study and assess whether the approach is appropriate given the extent and likely mechanism of missingness. + +**What to look for in code/outputs:** +- Missing data handling method extracted per study: complete case analysis, single imputation, multiple imputation (and number of imputations), missing indicator method +- Proportion of missing data for key predictors and the outcome extracted when reported +- Assessment of whether the missing data mechanism was considered (missing completely at random, missing at random, missing not at random) +- Code or notes flagging studies that performed complete case analysis with >5-10% missing data as potentially biased +- Documentation of whether imputation was done separately for the development and validation samples in internal validation studies + +--- + +## Domain 7: Model Development + +### Model Development + +**Description:** Extract and appraise the statistical method used to develop the prediction model, including predictor selection strategy, shrinkage or penalization, and the final model specification. + +**What to look for in code/outputs:** +- Modeling method extracted: logistic regression, Cox proportional hazards, Fine-Gray subdistribution hazard, random forest, gradient boosting, neural network, or other machine learning approach +- Predictor selection method extracted and appraised: all pre-specified predictors included, backward/forward/stepwise selection, LASSO, elastic net, or other penalization +- Stepwise selection based solely on statistical significance without clinical pre-specification flagged as a concern for overfitting +- Shrinkage or penalization applied (ridge, LASSO, elastic net, global shrinkage factor) extracted and documented +- Interaction terms included in the final model extracted and their clinical justification documented +- Final model presented as a score, nomogram, regression equation, or software tool extracted + +--- + +## Domain 8: Model Performance + +### Model Performance + +**Description:** Extract performance metrics reported for the prediction model, including discrimination, calibration, and overall performance, and assess whether the reported measures are appropriate and correctly computed. + +**What to look for in code/outputs:** +- Discrimination metrics extracted per study: C-statistic (AUC for binary outcomes, Harrell's C for survival), sensitivity and specificity at a reported threshold, D-statistic +- Calibration metrics extracted: calibration plot (visual), Hosmer-Lemeshow test, calibration slope, calibration-in-the-large (O/E ratio) +- Reliance on Hosmer-Lemeshow test alone as evidence of good calibration flagged (test has low power and is sensitive to sample size) +- Overall performance metrics extracted: Brier score, Nagelkerke R-squared, scaled Brier score +- Confidence intervals for performance metrics extracted when reported +- Code or notes documenting whether performance metrics were computed on the same data used for development (optimistic) vs. a held-out validation set (less biased) + +--- + +## Domain 9: Model Evaluation + +### Model Evaluation + +**Description:** Extract and appraise the type of validation performed, including internal validation (bootstrap, cross-validation), temporal validation, geographic validation, or independent external validation, and the extent of optimism correction applied. + +**What to look for in code/outputs:** +- Validation type extracted per study: apparent performance only, internal validation (bootstrap or cross-validation), internal-external cross-validation, temporal external validation, geographic external validation, independent external validation +- Studies reporting only apparent performance without any validation flagged as high risk of optimism +- Internal validation method extracted: bootstrap resampling with optimism correction, k-fold cross-validation, leave-one-out cross-validation +- Optimism-corrected performance estimates extracted separately from apparent estimates +- For external validation studies: the development study being validated identified and the degree of overlap in population and setting described +- Heterogeneity in external validation performance across geographic settings or time periods documented and analyzed + +--- + +## Domain 10: Results + +### Results + +**Description:** Extract the final model equation or scoring algorithm, reported predictor effects, and performance statistics in a format that enables inclusion in a meta-analysis of discrimination or calibration. + +**What to look for in code/outputs:** +- Final model equation, regression coefficients, or score weights extracted per study (enables model reproduction and meta-analysis of effects) +- Predictor-specific effect estimates (odds ratios, hazard ratios, regression coefficients) with confidence intervals extracted +- Meta-analysis code pooling C-statistics or calibration metrics across studies using appropriate variance-stabilizing transformations (logit transformation for C-statistics) +- Heterogeneity statistics (I-squared, tau-squared) reported for meta-analyzed performance metrics +- Forest plots of C-statistics or calibration slopes across included studies generated +- Studies reporting only graphical results without numerical estimates flagged as limiting meta-analysis inclusion + +--- + +## Domain 11: Interpretation and Discussion + +### Interpretation and Discussion + +**Description:** Extract and appraise whether the study authors' conclusions are appropriately qualified by the validation evidence, whether clinical utility was assessed, and whether model limitations are addressed. + +**What to look for in code/outputs:** +- Documentation of whether the included studies assessed clinical utility: decision curve analysis, net reclassification improvement (NRI), integrated discrimination improvement (IDI) +- Extraction of net benefit or decision curve analysis results when reported +- Assessment of whether authors' conclusions are proportionate to the strength of validation evidence (development-only studies should not claim external validity) +- Recommendations for intended clinical application qualified by the available validation evidence +- Limitations sections addressing overfitting, missing data handling, outcome definition, and generalizability extracted and summarized across included studies +- Recommendations for future validation or model updating identified as knowledge gaps in the systematic review discussion + +--- + +## Presentation and Summary for Systematic Reviews + +### Risk of Bias Assessment Across Domains* + +**Description:** Summarize critical appraisal findings across CHARMS domains for all included studies, identifying the prevalence of methodological concerns and their likely direction of bias on reported model performance. + +**What to look for in code/outputs:** +- Risk of bias or methodological concern coded per domain per study using a structured appraisal table (commonly using PROBAST alongside CHARMS for formal bias ratings) +- Code generating a summary visualization of methodological quality across included studies +- Sensitivity analyses in the meta-analysis restricting to studies with adequate sample size, external validation, or no stepwise selection +- Meta-regression examining the association between methodological quality indicators and reported discrimination or calibration +- Narrative synthesis discussing the overall quality of the prediction model literature for the target condition and how it limits confidence in pooled estimates diff --git a/plugins/ce-datascience/skills/ce-code-review/references/consort-adaptive-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/consort-adaptive-checklist.md new file mode 100644 index 00000000..e8d59f45 --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/consort-adaptive-checklist.md @@ -0,0 +1,205 @@ +# ACE Checklist for Adaptive Clinical Trial Designs + +Adaptive designs CONSORT Extension (ACE). +Supplements the core CONSORT 2010 checklist with items specific to randomized controlled trials that use pre-planned adaptations based on accumulating data. + +**Primary reference:** Dimairo M, Pallmann P, Wason J, et al. The Adaptive designs CONSORT Extension (ACE) statement: a checklist with explanation and elaboration guideline for reporting randomised trials that use an adaptive design. BMJ. 2020;369:m115. doi:10.1136/bmj.m115. PMID: 32554564. + +Apply this checklist alongside the core CONSORT 2010 checklist. Items below represent additions or modifications to base CONSORT items required for adaptive trial designs. Each item describes what must be reported and what to look for in analysis code and outputs. + +--- + +## Title and Abstract + +### Item 1 -- Identification as an adaptive design trial + +**Description:** Identify the study as a randomized adaptive design trial in the title and abstract, naming the specific type of adaptation used. + +**What to look for in code/outputs:** +- Report title or rendered document header explicitly stating "adaptive" and naming the design (e.g., "group sequential," "response-adaptive randomization," "seamless phase II/III") +- Abstract text naming the type of adaptation and summarizing the pre-specified adaptation rules +- Report metadata confirming the adaptive design designation and the primary adaptation type + +--- + +## Introduction + +### Item 2 -- Rationale for adaptive design* + +**Description:** Explain the scientific and statistical rationale for using an adaptive design, including why a non-adaptive design was insufficient to meet the objectives. + +**What to look for in code/outputs:** +- Documentation explicitly justifying the adaptive design over a fixed-sample design +- Report sections explaining the specific operational or scientific advantage gained (e.g., ethical imperative to stop early for harm, uncertainty about effect size requiring sample size re-estimation) +- Protocol or SAP text describing the adaptive design within the context of the overall development program + +--- + +## Methods + +### Item 3 -- Type of adaptation* + +**Description:** State the type(s) of adaptive design used, including the number of planned interim analyses, and describe each pre-specified adaptation rule in full operational detail. + +**What to look for in code/outputs:** +- Adaptation type clearly named: group sequential stopping (for efficacy or futility), sample size re-estimation, arm dropping, biomarker-adaptive enrichment, response-adaptive randomization, seamless phase II/III, or platform trial adaptation +- All planned interim looks listed with their timing (information fraction or calendar time) +- Each adaptation rule specified completely (stopping boundary values, enrichment criteria, re-estimation algorithm, arm-dropping criteria) +- Code or statistical software implementation of the adaptation rules (gsDesign, rpact, ADDPLAN, East) referenced or included + +### Item 4 -- Pre-specified adaptation rules* + +**Description:** Provide complete pre-specified decision rules for each adaptation, including the criteria that trigger adaptation, the decision algorithm, and the possible outcomes at each interim. + +**What to look for in code/outputs:** +- Decision rules documented in the SAP with exact threshold values (e.g., conditional power below 20% triggers futility stopping) +- Code implementing the decision boundaries with the exact parameter values from the SAP +- Documentation confirming rules were pre-specified before any unblinded data were accessed +- Version-controlled SAP files with timestamps confirming pre-specification relative to data access +- Possible outcomes at each interim analysis stage enumerated (continue unchanged, adapt allocation, stop for efficacy, stop for futility, enrich population) + +### Item 5 -- Interim analysis schedule* + +**Description:** Specify the timing of each planned interim analysis in terms of information fraction, sample size, or calendar time, and describe the process for conducting each interim analysis. + +**What to look for in code/outputs:** +- Information fractions or sample size milestones for each interim analysis stated in the SAP and analysis code +- Code that triggers or flags interim analyses at pre-specified data maturity thresholds +- Documentation of who conducts each interim analysis and their access to unblinded data +- Procedures for data lock and data transfer to the independent data monitoring committee (DMC) at each interim + +### Item 6 -- Type I error control across adaptations* + +**Description:** Describe the method used to control the overall type I error rate across all planned interim analyses and adaptations, including the alpha-spending function or group-sequential boundary used. + +**What to look for in code/outputs:** +- Alpha-spending function named and its parameters specified: O'Brien-Fleming, Pocock, Lan-DeMets with exact spending function parameters +- Code implementing the spending function (gsDesign `sfLDOF`, `sfHSD`, or custom function; rpact `getDesignGroupSequential`) +- Cumulative alpha spent at each interim analysis stated alongside the boundary +- Documentation confirming the overall one-sided or two-sided alpha level and how it is apportioned across interims +- Simulation results or analytic derivations confirming family-wise type I error control if non-standard adaptations are used + +### Item 7 -- Information fraction at each interim* + +**Description:** Provide the information fraction (proportion of maximum information) at each planned interim analysis and explain how information is defined and measured for the primary endpoint. + +**What to look for in code/outputs:** +- Information fraction defined (e.g., number of events for time-to-event endpoints, number of participants for continuous endpoints) in the SAP +- Code that computes the information fraction at each interim as a function of observed data +- Planned vs. observed information fractions compared in the results +- Documentation of how information was measured when the maximum information was unknown at design time (e.g., in sample size re-estimation designs) + +### Item 8 -- Conditional power and futility rules* + +**Description:** If futility stopping is included, describe the conditional power calculation method, the threshold below which the trial will stop for futility, and whether stopping is binding or non-binding. + +**What to look for in code/outputs:** +- Conditional power formula or computational approach documented in the SAP +- Threshold for futility stopping stated as a conditional power value (e.g., stop if conditional power < 20%) +- Code computing conditional power at each interim under the current observed treatment effect and under the assumed effect at design +- Documentation of whether futility boundaries are binding (protocol-specified mandatory stopping) or non-binding (advisory) +- DMC charter references confirming futility stopping authority and process + +### Item 9 -- Sample size re-estimation method* + +**Description:** If sample size re-estimation is used, describe the re-estimation algorithm, the blinding status during re-estimation, the timing, and the minimum and maximum sample size bounds. + +**What to look for in code/outputs:** +- Re-estimation algorithm fully specified: internal pilot, conditional power-based re-estimation, promising zone design +- Blinding status at re-estimation: blinded (using pooled variance) vs. unblinded (using observed treatment effect) +- Minimum and maximum allowable sample sizes after re-estimation defined in the SAP and coded as hard constraints +- Code that enforces the bounds and records the pre- and post-re-estimation sample sizes +- Documentation of the timing of re-estimation relative to information fraction + +### Item 10 -- Independence of DMC* + +**Description:** Describe the independence and operating procedures of the data monitoring committee (DMC) or analogous body that oversees interim analyses and adaptation decisions. + +**What to look for in code/outputs:** +- DMC charter referenced in the analysis documentation +- Documentation of DMC independence from the trial sponsor and investigators +- Description of the blinding firewall between the DMC statistician and the trial team +- Procedures for communicating DMC recommendations to the trial team without revealing unblinded interim data +- Code or workflow isolating unblinded analysis code from the main analysis repository + +### Item 11 -- Statistical analysis after adaptation* + +**Description:** Describe how the primary analysis accounts for the adaptive nature of the design, including any bias correction, conditional inference, or ordering of results across stages. + +**What to look for in code/outputs:** +- Final analysis code that combines data across all stages using the correct combining function (e.g., weighted inverse normal, Fisher combination) +- Bias-corrected point estimates or median unbiased estimates reported alongside naive estimates +- Confidence intervals and p-values adjusted for the sequential design (not standard fixed-sample intervals) +- Documentation confirming the primary analysis method was specified before any unblinded data were reviewed +- Stage-wise ordering of results for sequential trials or confidence interval computation method specified + +--- + +## Results + +### Item 12 -- Participant flow through adaptive stages* + +**Description:** Report the flow of participants through each stage of the adaptive trial, including the number at each interim, any adaptations implemented, and the reasons for stopping or continuing. + +**What to look for in code/outputs:** +- CONSORT flow diagram extended to show the number of participants at each stage and the outcome of each interim analysis +- Adaptation decisions documented at each interim: continuation unchanged, sample size increase, arm dropped, population enriched, or stopping +- Code tracking and reporting participant counts at each stage boundary +- Reasons for any unplanned deviations from the adaptation rules documented with rationale + +### Item 13 -- Results of interim analyses* + +**Description:** Report the results of each interim analysis, including the test statistic, boundary value, information fraction, and the adaptation decision made. + +**What to look for in code/outputs:** +- Table of interim analysis results: interim number, timing, observed statistic, boundary value (efficacy and/or futility), conditional power, and decision +- Code that records and outputs interim results in a reproducible log +- Documentation confirming whether each interim result triggered an adaptation or resulted in continuation +- Blinded vs. unblinded interim results clearly distinguished in reporting + +### Item 14 -- Final analysis results accounting for adaptation* + +**Description:** For each primary and secondary outcome, present the final analysis results including estimates, confidence intervals, and p-values that appropriately account for the adaptive design. + +**What to look for in code/outputs:** +- Adjusted (stage-combining or bias-corrected) estimates alongside naive final estimates +- Confidence intervals derived from the sequential or adaptive analysis procedure, not standard fixed-sample formulas +- Adjusted p-values consistent with the alpha-spending or combination test used +- Software and version used for the adaptive analysis reported (gsDesign, rpact, ADDPLAN, East, or custom code) + +--- + +## Discussion + +### Item 15 -- Limitations specific to adaptive design + +**Description:** Discuss limitations related to the adaptive design, including potential for operational bias, deviations from the adaptation plan, and how the adaptation may affect interpretation. + +**What to look for in code/outputs:** +- Limitation sections addressing operational bias risks (e.g., unblinded information leaking to the trial team, sample size changes revealing interim results) +- Documentation of any deviations from the pre-specified adaptation rules with explanation and impact assessment +- Discussion of how the adaptation affects generalizability or interpretation of the results +- Sensitivity analyses comparing adaptive analysis estimates to naive estimates that ignore the adaptive nature + +### Item 16 -- Generalizability after adaptation + +**Description:** Discuss the generalizability of the results in light of any population enrichment, arm selection, or other adaptation that changed the study population or design during the trial. + +**What to look for in code/outputs:** +- Explicit discussion of how population enrichment or arm dropping affects the population to which results apply +- Comparison of the enrolled population pre- and post-adaptation to contextualize generalizability +- Description of the final study population relative to the original target population + +--- + +## Other Information + +### Item 17 -- Adaptive design registration and protocol + +**Description:** State where the adaptive design protocol, SAP, and DMC charter can be accessed, confirming they were finalized before the first interim analysis. + +**What to look for in code/outputs:** +- Registration number referencing an adaptive trial registry entry (ClinicalTrials.gov, EudraCT) that includes the adaptive design details +- Version-controlled SAP and DMC charter with timestamps pre-dating the first interim analysis +- DOI or URL for the published adaptive design protocol if available +- Documentation confirming no post-hoc changes to adaptation rules diff --git a/plugins/ce-datascience/skills/ce-code-review/references/consort-cluster-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/consort-cluster-checklist.md new file mode 100644 index 00000000..4d49d63f --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/consort-cluster-checklist.md @@ -0,0 +1,188 @@ +# CONSORT Extension for Cluster Randomized Trials + +Consolidated Standards of Reporting Trials (CONSORT) extension for cluster randomized trials (CRTs). +Supplements the core CONSORT 2010 checklist with items specific to trials in which intact groups rather than individuals are randomized. + +**Primary reference:** Campbell MK, Piaggio G, Elbourne DR, Altman DG; CONSORT Group. Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012;345:e5661. doi:10.1136/bmj.e5661. PMID: 22951546. + +Apply this checklist alongside the core CONSORT 2010 checklist. Items below represent additions or modifications to base CONSORT items required for cluster randomized trials. Each item includes the base CONSORT item number where applicable, the description, and what to look for in analysis code and outputs. + +--- + +## Title and Abstract + +### Item 1 -- Identification as a cluster randomized trial + +**Description:** Identify the study as a cluster randomized trial in the title and abstract. Specify the unit of randomization (e.g., clinic, school, community). + +**What to look for in code/outputs:** +- Report title or rendered document header explicitly stating "cluster randomized trial" or "cluster randomised trial" +- Abstract text identifying the unit of randomization and the type of cluster (hospital, practice, village, etc.) +- Report metadata or template confirming the cluster design designation + +--- + +## Introduction + +### Item 2 -- Rationale for cluster design* + +**Description:** Explain why a cluster randomized design was chosen rather than individual randomization, including the scientific, logistical, or ethical justification. + +**What to look for in code/outputs:** +- Documentation explicitly justifying cluster randomization (e.g., intervention delivered at the group level, contamination prevention, administrative feasibility) +- Report sections distinguishing the unit of randomization from the unit of analysis +- Protocol or SAP text describing why individual randomization was not appropriate + +--- + +## Methods + +### Item 3 -- How clusters were defined* + +**Description:** Define the clusters used in the trial, including their nature (geographic, organizational, social) and how boundaries were determined. + +**What to look for in code/outputs:** +- Cluster definition documented in data dictionary, protocol, or analysis header (e.g., general practices defined by unique practice identifier) +- Code that assigns individuals to clusters using the cluster identifier variable +- Documentation of how cluster boundaries were established and whether clusters were pre-existing or defined for the trial +- Summary statistics describing cluster size distribution (mean, range, coefficient of variation) + +### Item 4 -- Eligibility criteria: clusters and individuals + +**Description:** Specify eligibility criteria for clusters and, where applicable, eligibility criteria for individuals within clusters. + +**What to look for in code/outputs:** +- Separate inclusion and exclusion criteria documented for clusters and for individuals within clusters +- Code applying cluster-level filters (e.g., minimum cluster size, geographic region) prior to individual-level filters +- Documentation of the two-stage eligibility process and the number excluded at each level +- Participant flowchart or attrition table distinguishing cluster-level from individual-level exclusions + +### Item 5 -- Rationale for cluster size and recruitment within clusters + +**Description:** Describe how many individuals were to be recruited within each cluster and how. + +**What to look for in code/outputs:** +- Target number of participants per cluster documented in the SAP or protocol +- Recruitment strategy within clusters described (consecutive enrollment, random sampling within cluster) +- Actual cluster size distribution in the data compared to the target in the sample size documentation +- Sensitivity analyses examining the effect of variable cluster sizes on the primary analysis + +### Item 6 -- Sample size accounting for clustering* + +**Description:** Describe how the sample size was calculated, including the intracluster correlation coefficient (ICC), the average cluster size, and the resulting design effect used to inflate the simple random sample size. + +**What to look for in code/outputs:** +- Sample size calculation code specifying the assumed ICC, average cluster size, and design effect (variance inflation factor) +- Formula or function used to compute the design effect: `1 + (m - 1) * ICC` where m is mean cluster size +- Source of the ICC estimate (prior literature, pilot data) documented with a citation or reference +- Sensitivity analyses varying the ICC assumption across plausible values +- Resulting inflated sample size and number of clusters stated alongside simple random sample equivalent + +### Item 7 -- Unit of randomization* + +**Description:** State the unit of randomization (i.e., the cluster) and the unit of analysis (i.e., individuals or clusters). + +**What to look for in code/outputs:** +- Clear documentation distinguishing unit of randomization from unit of analysis +- Code confirming that treatment assignment is at the cluster level (not individual level) +- Analysis code that treats the cluster as the unit of randomization for variance estimation +- Documentation of any cross-level analysis (e.g., cluster-level outcomes computed from individual-level data) + +### Item 8 -- Randomization: method and level + +**Description:** Describe the method used to generate the random allocation sequence for clusters and any stratification or restriction at the cluster level. + +**What to look for in code/outputs:** +- Randomization code operating on cluster identifiers, not individual patient identifiers +- Stratification factors applied at the cluster level (e.g., cluster size stratum, geographic region, baseline outcome rate) +- Block or restricted randomization within strata of clusters documented +- Random seed and allocation method recorded for reproducibility + +### Item 9 -- Blinding: which parties were blinded* + +**Description:** State whether participants, care providers, and outcome assessors were aware of the intervention assignment at the cluster level, and whether individual-level blinding was implemented. + +**What to look for in code/outputs:** +- Documentation specifying blinding status for clusters and for individuals within clusters separately +- Discussion of the practical constraints on blinding in cluster trials (e.g., open-label cluster-level intervention) +- Outcome assessor blinding procedures documented, especially for subjective outcomes +- Sensitivity analyses exploring the potential influence of unblinded outcome assessment + +### Item 10 -- Statistical methods for clustered data* + +**Description:** Describe the statistical methods used to account for clustering in the primary analysis, including the approach to handling the hierarchical structure of the data (individuals within clusters). + +**What to look for in code/outputs:** +- Primary analysis model explicitly accounting for clustering: mixed-effects models (lme4, nlme, lmer), generalized estimating equations (GEE), or cluster-robust standard errors +- Model specification distinguishing fixed effects (treatment, covariates) from random effects (cluster-level random intercept, random slope) +- GEE working correlation structure specified (exchangeable, independent, unstructured) with justification +- Sensitivity analysis comparing results from different approaches (e.g., GEE vs. mixed model, cluster-level analysis) +- Documentation that standard regression without clustering adjustment was NOT used as the primary analysis + +### Item 11 -- ICC reporting* + +**Description:** Report the estimated intracluster correlation coefficient (ICC) for the primary outcome and its confidence interval. + +**What to look for in code/outputs:** +- ICC estimation code for the primary outcome (e.g., from the null model or primary mixed model) +- Reported ICC value with 95% confidence interval in results tables or output +- Between-cluster and within-cluster variance components from the mixed model reported +- Comparison of estimated ICC to the value assumed in the sample size calculation + +--- + +## Results + +### Item 12 -- Participant flow: clusters and individuals* + +**Description:** For each group, provide the number of clusters allocated and the number of individuals recruited, followed by the numbers receiving the intended intervention, completing follow-up, and included in the primary analysis, all reported at both the cluster and individual level. + +**What to look for in code/outputs:** +- CONSORT flow diagram reporting cluster counts and individual counts at every stage separately +- Post-randomization exclusions documented at both the cluster and individual level with reasons +- Number of clusters and individuals analyzed for the primary outcome stated explicitly +- Code tracking attrition at the cluster level (e.g., cluster dropout, cluster contamination) alongside individual-level attrition + +### Item 13 -- Baseline characteristics: clusters and individuals* + +**Description:** Present baseline demographic and clinical characteristics for clusters and for individuals within clusters, by treatment arm. + +**What to look for in code/outputs:** +- Table 1 code generating cluster-level characteristics (cluster size, cluster-level outcome rates, cluster-level covariates) by arm +- Individual-level baseline characteristics table stratified by treatment arm +- Standardized mean differences or balance statistics reported at both levels +- No p-values for baseline comparisons (randomization makes differences due to chance) + +### Item 14 -- Numbers analyzed: clusters and individuals* + +**Description:** Report the number of clusters and individuals analyzed for each outcome and confirm that the analysis accounted for the cluster design. + +**What to look for in code/outputs:** +- Denominators stated at the cluster and individual level for each analysis +- Confirmation that the analysis population is consistent with the stated analysis plan (e.g., intention-to-treat) +- Documentation that all randomized clusters were included in the primary analysis unless exclusion was pre-specified +- Sensitivity analysis results for per-protocol or complete-case analyses alongside the primary analysis + +### Item 15 -- Treatment effects with clustering accounted for* + +**Description:** For each primary and secondary outcome, present results for each group, the estimated effect size, and its precision (confidence interval), confirming that the estimate accounts for the clustered design. + +**What to look for in code/outputs:** +- Treatment effect estimates with confidence intervals derived from cluster-adjusted models (not unadjusted individual-level analysis) +- Model output confirming the clustering structure was included (random effects or robust SE specification) +- Cluster-level effect estimates alongside individual-level estimates when both are informative +- Design effect or effective sample size reported alongside treatment effect to contextualize precision + +--- + +## Discussion + +### Item 16 -- Generalizability of cluster findings + +**Description:** Discuss the generalizability of the trial findings, including whether the results apply to the clusters studied, to individuals within those clusters, or to both. + +**What to look for in code/outputs:** +- Discussion distinguishing generalizability at the cluster level from generalizability at the individual level +- Description of the clusters and whether they represent the target population of clusters +- Acknowledgment of any unusual cluster characteristics that may limit generalizability +- Comment on how cluster size variability or cluster-level heterogeneity affects interpretation diff --git a/plugins/ce-datascience/skills/ce-code-review/references/consort-nof1-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/consort-nof1-checklist.md new file mode 100644 index 00000000..841cc93c --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/consort-nof1-checklist.md @@ -0,0 +1,208 @@ +# CONSORT Extension for N-of-1 Trials + +Consolidated Standards of Reporting Trials (CONSORT) extension for N-of-1 trials. +Supplements the core CONSORT 2010 checklist with items specific to single-patient multiple-crossover trials (N-of-1 trials) conducted individually or as a series. + +**Primary reference:** Vohra S, Shamseer L, Sampson M, et al. CONSORT extension for reporting N-of-1 trials (CENT) 2015 statement. J Clin Epidemiol. 2015;76:9-17. doi:10.1016/j.jclinepi.2015.05.004. PMID: 26272792. + +Apply this checklist alongside the core CONSORT 2010 checklist. Items below represent additions or modifications to base CONSORT items required for N-of-1 trial designs. Each item includes the description and what to look for in analysis code and outputs. + +--- + +## Title and Abstract + +### Item 1 -- Identification as an N-of-1 trial + +**Description:** Identify the study as an N-of-1 trial or series of N-of-1 trials in the title and abstract, specifying whether the report covers a single patient or an aggregated series. + +**What to look for in code/outputs:** +- Report title or rendered document header explicitly stating "N-of-1 trial," "single patient trial," or "series of N-of-1 trials" +- Abstract text clarifying whether results describe a single patient or an aggregated analysis across multiple N-of-1 trials +- Documentation distinguishing this design from a crossover trial, case study, or observational repeated-measures study + +--- + +## Introduction + +### Item 2 -- Rationale for N-of-1 design* + +**Description:** Explain the rationale for using an N-of-1 design, including the clinical or scientific circumstances that make individual-level treatment comparison appropriate. + +**What to look for in code/outputs:** +- Documentation justifying why a parallel-group or conventional crossover trial was not used (e.g., chronic stable condition, long-term treatment decision for an individual, patient preference) +- Report sections describing the clinical condition as one where group-average effects are expected to be heterogeneous or where individual-level decisions are the goal +- Protocol or SAP text explaining how the N-of-1 design addresses the specific clinical question + +--- + +## Methods + +### Item 3 -- Treatment periods and washout* + +**Description:** Describe the length of each treatment period, the criteria used to determine period length (e.g., expected time to treatment effect and offset), and the duration and nature of any washout periods between treatment periods. + +**What to look for in code/outputs:** +- Treatment period length specified in the SAP or protocol with justification based on pharmacokinetics or expected time to therapeutic effect +- Washout period duration specified with the rationale (minimum time for drug clearance, return to baseline) +- Code variables defining period start and end dates or time points relative to the washout schedule +- Outcome assessment windows within each treatment period documented and coded + +### Item 4 -- Sequence of treatments* + +**Description:** Describe the method used to determine the order of treatment periods (e.g., random assignment, alternating order) and whether the sequence was disclosed to the patient and clinician. + +**What to look for in code/outputs:** +- Randomization or sequence assignment code operating at the period level for each patient +- Documentation of whether a balanced incomplete block design, fully randomized order, or fixed alternating sequence was used +- Record of whether treatment sequence allocation was concealed from the patient and clinician prior to each period +- Sequence assignment recorded in the data alongside the outcome data for each period + +### Item 5 -- Number of treatment pairs or cycles* + +**Description:** State the planned number of treatment periods (or treatment pairs for alternating designs) and the rationale for this number, including any stopping rules for individual patients. + +**What to look for in code/outputs:** +- Pre-specified number of treatment cycles or pairs documented in the SAP (e.g., three pairs of two periods each) +- Rationale for the planned number of cycles based on power considerations or clinical decision requirements +- Stopping rules for individual patients defined (e.g., stop after achieving a clear treatment preference or after a pre-specified number of cycles regardless of outcome) +- Code recording the actual number of completed cycles per patient and flagging patients who stopped early with reasons + +### Item 6 -- Carryover effects assessment* + +**Description:** Describe how carryover effects (the influence of treatment received in one period on outcomes in subsequent periods) were assessed and accounted for in the analysis. + +**What to look for in code/outputs:** +- Code testing for period-by-treatment interaction or carry-over effects in the statistical model +- Washout monitoring variables (e.g., drug levels, symptom scores at the start of each period) used as indicators of incomplete washout +- Documentation of the assumption of no carry-over and the evidence supporting this assumption +- Sensitivity analyses excluding the first period of each pair or extending the washout period in a subset of patients +- Graphical displays of outcomes by period overlaid on treatment assignment to visually inspect period trends and carryover + +### Item 7 -- Outcome measurement within periods* + +**Description:** Describe how and when outcomes were measured within each treatment period, including the number of observations per period and how multiple measurements within a period were combined. + +**What to look for in code/outputs:** +- Timing of outcome assessments within each period documented (e.g., daily diary for days 5-14 of a 14-day period) +- Code averaging or summarizing multiple within-period observations to a single period-level outcome +- Rules for handling missing within-period observations documented and applied consistently +- Period summary statistics output alongside raw within-period measurements for review + +### Item 8 -- Blinding in N-of-1 context + +**Description:** State whether the patient, clinician, and outcome assessors were blinded to treatment assignment within each period, and describe how blinding was achieved (e.g., matching placebo, overencapsulation). + +**What to look for in code/outputs:** +- Documentation of blinding status for the patient, treating clinician, and outcome assessor +- Pharmaceutical preparation method ensuring treatment indistinguishability referenced (overencapsulation, matching placebo) +- Blinding check or success assessment code (patient's guess of treatment assignment) +- Procedures for emergency unblinding described in the protocol + +### Item 9 -- Statistical methods for within-patient analysis* + +**Description:** Describe the primary statistical method for estimating the within-patient treatment effect, including how data were analysed to compare treatment periods for the individual patient. + +**What to look for in code/outputs:** +- Within-patient analysis method specified: paired t-test, Wilcoxon signed-rank, linear mixed model with patient-as-random-effect, Bayesian hierarchical model +- Code implementing the primary within-patient comparison at the individual level before any aggregation +- Confidence intervals or credible intervals for the individual-level treatment effect computed and reported +- Documentation that the within-patient analysis is the primary inference target, not a group-average effect + +### Item 10 -- Aggregation across patients in a series* + +**Description:** If the report covers a series of N-of-1 trials, describe the method used to aggregate or synthesize individual patient results, including the rationale for aggregation and the model used. + +**What to look for in code/outputs:** +- Aggregation method described: meta-analytic pooling using random-effects model, Bayesian hierarchical model with patient-level random effects, or fixed-effects pooling +- Code implementing the aggregation with individual patient estimates as inputs +- Between-patient heterogeneity estimated and reported (tau-squared or I-squared for random-effects approaches) +- Rationale for aggregating results across patients documented (e.g., estimating average treatment effect across a heterogeneous population) +- Individual patient treatment effect estimates presented alongside the pooled estimate + +### Item 11 -- Patient-level treatment effect estimation* + +**Description:** Describe how patient-specific treatment effect estimates were derived and whether they were used to guide individual treatment decisions. + +**What to look for in code/outputs:** +- Code generating individual patient treatment effect estimates with uncertainty intervals (confidence or credible intervals) +- Documentation of how individual estimates were used clinically (shared with patient, used for prescribing decision) +- Empirical Bayes or shrinkage estimates reported when a hierarchical model is used, alongside the patient's own observed estimate +- Graphical display of individual patient estimates with uncertainty across the patient series + +--- + +## Results + +### Item 12 -- Participant flow per patient and per series + +**Description:** Report the number of treatment cycles completed for each patient and, for a series, the total number of patients enrolled, completing the full protocol, and analyzed. + +**What to look for in code/outputs:** +- Per-patient cycle completion table: planned cycles vs. completed cycles with reasons for early stopping +- For a series: CONSORT flow diagram showing number of patients enrolled, completing all cycles, and analyzed +- Reasons for missing periods or early stopping documented by patient +- Sensitivity analyses comparing completers to non-completers in a series + +### Item 13 -- Baseline characteristics per patient + +**Description:** Present baseline characteristics for each patient and, where relevant, compare characteristics across patients in a series. + +**What to look for in code/outputs:** +- Per-patient baseline table including relevant clinical characteristics and disease severity measures +- For a series: summary table of patient characteristics with variability statistics (mean, SD, range) +- Baseline outcome or symptom scores recorded before the first treatment period for each patient +- Code generating the baseline table from the per-patient data file + +### Item 14 -- Individual patient treatment effects* + +**Description:** Present the estimated treatment effect for each individual patient with its confidence or credible interval, using a format that allows comparison across patients in a series. + +**What to look for in code/outputs:** +- Forest plot or table of individual patient treatment effects with confidence or credible intervals +- Code generating per-patient estimates from the within-patient analysis model +- Graphical display of outcomes by treatment period for each patient (raw data plots) +- Clear labeling of the direction of effect (which treatment is favored) for each patient + +### Item 15 -- Aggregated results for a series* + +**Description:** For a series of N-of-1 trials, report the pooled treatment effect with its uncertainty, heterogeneity measures, and the number of patients for whom each treatment was favored. + +**What to look for in code/outputs:** +- Pooled treatment effect estimate with 95% confidence or credible interval from the aggregation model +- Between-patient heterogeneity statistics (tau-squared, I-squared, or posterior distribution of heterogeneity) +- Count or proportion of patients for whom treatment A was favored, treatment B was favored, or results were inconclusive +- Code confirming the aggregation model and its output are consistent with the described method + +### Item 16 -- Carryover and period effects* + +**Description:** Report the results of any tests for carryover or period effects and describe how these findings affected the analysis or interpretation. + +**What to look for in code/outputs:** +- Test statistics and p-values for carryover and period effect tests reported in results +- Code output for the period-by-treatment interaction test +- Discussion of how any detected carryover or period effects were handled (e.g., excluding first period, extending washout in future trials) +- Sensitivity analysis results when carryover was detected or suspected + +--- + +## Discussion + +### Item 17 -- Clinical application of individual results* + +**Description:** Discuss how the individual patient treatment effect estimates can be used to guide clinical decisions for the patient(s) included in the trial. + +**What to look for in code/outputs:** +- Discussion of how the individual estimates were communicated to patients and clinicians +- Description of the clinical decision made based on the N-of-1 trial results +- Acknowledgment of uncertainty in the individual estimate when cycles are few +- Consideration of whether the result for this individual generalizes to other similar patients + +### Item 18 -- Generalizability from a series + +**Description:** For a series of N-of-1 trials, discuss the extent to which the aggregated results generalize to the population of patients with the condition, and compare the N-of-1 series results to parallel-group trial evidence if available. + +**What to look for in code/outputs:** +- Discussion comparing the aggregate N-of-1 series estimate to evidence from parallel-group RCTs +- Acknowledgment of selection bias (patients motivated to participate in N-of-1 trials may differ from the general population) +- Comparison of the between-patient heterogeneity found in the series to heterogeneity assumptions made in the design +- Recommendations for which patient subtypes may benefit most based on the heterogeneity of treatment response observed diff --git a/plugins/ce-datascience/skills/ce-code-review/references/gramms-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/gramms-checklist.md new file mode 100644 index 00000000..991d667d --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/gramms-checklist.md @@ -0,0 +1,100 @@ +# GRAMMS Checklist for Mixed-Methods Studies + +Good Reporting of A Mixed Methods Study (GRAMMS). +6-item checklist for reporting studies that combine qualitative and quantitative methods. + +**Primary reference:** O'Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy. 2008;13(2):92-98. doi:10.1258/jhsrp.2007.007074. PMID: 18416914. + +Apply when reporting studies that deliberately combine qualitative and quantitative methods as part of a single investigation. GRAMMS focuses on whether the rationale for mixing methods is clear, whether the design is described coherently, and whether the integration of the two strands is explained and evaluated. It applies to published reports of empirical studies using mixed methods in health services research and related fields. + +--- + +## Justification + +### Item 1 -- Justification for using a mixed methods approach + +**Description:** Describe why a mixed methods approach was used for this study. Explain the specific value added by combining methods that could not be achieved by using either approach alone. + +**What to look for in code/outputs:** +- Explicit statement of why neither qualitative nor quantitative methods alone were sufficient +- Mixed methods purpose articulated (triangulation, complementarity, development, initiation, expansion) +- Description of how each methodological strand addresses a different aspect of the research question +- Reference to a mixed methods rationale framework or typology (Greene et al., Creswell, Tashakkori and Teddlie) + +--- + +## Design + +### Item 2 -- Description of the mixed methods design + +**Description:** Identify the mixed methods design used (e.g., convergent, explanatory sequential, exploratory sequential, embedded). Describe the timing of the two strands (concurrent or sequential), the relative emphasis or weight given to each strand (equal or dominant), and the point of integration. + +**What to look for in code/outputs:** +- Named mixed methods design type with citation supporting the design framework +- Timing specified (concurrent data collection vs. sequential with strand order described) +- Relative priority or weighting of qualitative and quantitative strands stated +- Point of integration identified (at data collection, analysis, interpretation, or all three) +- Visual diagram or notation describing the design (e.g., QUAL + quan, QUAN -> qual) + +--- + +## Data Collection + +### Item 3 -- Rationale for data collection techniques + +**Description:** Describe the specific qualitative and quantitative data collection techniques used and explain why these techniques were chosen for each strand of the study. + +**What to look for in code/outputs:** +- Quantitative data collection method described (survey instrument, administrative data extraction, randomized trial, cohort) +- Qualitative data collection method described (interviews, focus groups, observation, documents) +- Rationale for choosing each technique linked to the research question and design +- Sampling strategy for each strand described (purposive, random, theoretical, convenience) +- Sample sizes for each strand with justification (saturation, power, representativeness) + +--- + +## Analysis + +### Item 4 -- Description of analysis methods + +**Description:** Describe the methods used to analyze data in each strand. Explain any procedures used to integrate or link data from the two strands during the analysis phase. + +**What to look for in code/outputs:** +- Quantitative analysis method specified (regression, survival analysis, descriptive statistics, SPC) +- Qualitative analysis method specified (thematic analysis, grounded theory, framework analysis, IPA) +- Software tools for each strand named (R, Stata, SPSS for quantitative; NVivo, Atlas.ti, MAXQDA for qualitative) +- Integration analysis procedures described if applicable (joint display, case-ordered matrix, transformation) +- Inter-rater reliability or triangulation procedures for qualitative coding documented + +--- + +## Integration + +### Item 5 -- Description of how integration was achieved + +**Description:** Describe how the qualitative and quantitative strands were brought together (integrated) to address the research question. Report the findings from both strands in relation to each other. Identify any divergent or contradictory findings across strands. + +**What to look for in code/outputs:** +- Integration narrative linking qualitative themes to quantitative results +- Joint display table or matrix presenting findings from both strands side by side +- Convergent findings identified and described +- Divergent or contradictory findings across strands explicitly reported and discussed +- Meta-inferences drawn from integration stated separately from single-strand conclusions +- Mixed methods findings presented as distinct from the sum of individual strand findings + +--- + +## Quality + +### Item 6 -- Identification of the limitations of each approach and quality criteria applied + +**Description:** Discuss the quality of the qualitative and quantitative strands separately, and the quality of the integration. Address limitations of each method, including threats to internal validity, reliability, credibility, and transferability. Describe any quality criteria applied that are specific to mixed methods research. + +**What to look for in code/outputs:** +- Quantitative limitations addressed (selection bias, confounding, measurement error, generalizability) +- Qualitative limitations addressed (reflexivity, researcher positionality, transferability, confirmability) +- Integration quality discussed (completeness of integration, whether divergence was fully explored) +- Quality criteria specific to mixed methods applied (legitimation, inference quality, integration fidelity) +- Reflexivity statement documenting researchers' positionality relative to the study context +- Member checking or respondent validation described for qualitative strand +- Qualitative rigor strategies reported (audit trail, negative case analysis, thick description) diff --git a/plugins/ce-datascience/skills/ce-code-review/references/prisma-dta-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/prisma-dta-checklist.md new file mode 100644 index 00000000..b36e884f --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/prisma-dta-checklist.md @@ -0,0 +1,309 @@ +# PRISMA-DTA Checklist for Systematic Reviews of Diagnostic Test Accuracy + +Preferred Reporting Items for Systematic Reviews and Meta-Analyses of Diagnostic Test Accuracy (PRISMA-DTA). +Extension of PRISMA for systematic reviews and meta-analyses of diagnostic test accuracy studies. + +**Primary reference:** McInnes MDF, Moher D, Thombs BD, McGrath TA, Bossuyt PM; and the PRISMA-DTA Group. Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: The PRISMA-DTA Statement. JAMA. 2018;319(4):388-396. doi:10.1001/jama.2017.19163. PMID: 29362800. + +**Explanation and elaboration:** Salameh JP, Bossuyt PM, McGrath TA, et al. Preferred reporting items for systematic review and meta-analysis of diagnostic test accuracy studies (PRISMA-DTA): explanation, elaboration, and checklist. BMJ. 2020;370:m2632. doi:10.1136/bmj.m2632. PMID: 32816740. + +This checklist extends the 2009 PRISMA statement with items specific to diagnostic accuracy reviews. Items marked with an asterisk (*) are new additions or modifications from base PRISMA. Apply when reporting a systematic review or meta-analysis where the primary aim is to assess the accuracy of one or more index tests against a reference standard. + +--- + +## Title + +### Item 1 -- Title + +**Description:** Identify the report as a systematic review and/or meta-analysis of diagnostic test accuracy. * + +**What to look for in code/outputs:** +- Report title specifying systematic review or meta-analysis and naming the index test +- Title identifying the target condition or the clinical question +- Keywords including "diagnostic accuracy," "sensitivity," "specificity," and the test name +- Structured abstract consistent with the DTA review scope + +--- + +## Abstract + +### Item 2 -- Structured summary + +**Description:** Provide a structured summary including, as applicable: background, objectives, data sources, study eligibility criteria, participants, index test(s), reference standard(s), synthesis methods, results (including estimates of sensitivity and specificity), and conclusions. * + +**What to look for in code/outputs:** +- Abstract reporting pooled sensitivity and specificity with confidence intervals +- Index test and reference standard named explicitly in abstract +- Number of studies and total participant count in abstract +- Summary ROC or HSROC point estimates reported + +--- + +## Introduction + +### Item 3 -- Rationale + +**Description:** Describe the rationale for the review in the context of what is already known, specifically including mention of why accurate diagnosis is important in the context of this clinical question. + +**What to look for in code/outputs:** +- Documentation of clinical context and importance of accurate diagnosis +- Literature review of existing accuracy estimates or prior systematic reviews +- Articulation of the knowledge gap motivating the review +- Description of consequences of false-positive and false-negative results in this context + +### Item 4 -- Objectives + +**Description:** Provide an explicit statement of questions being addressed with reference to participants, index test(s), reference standard, and target condition (PIRT). * + +**What to look for in code/outputs:** +- PIRT framework (Participants, Index test, Reference standard, Target condition) specified +- Primary and secondary research questions documented +- Subgroup or threshold-specific questions prespecified +- Study outcomes (sensitivity, specificity, likelihood ratios, AUC) named + +--- + +## Methods + +### Item 5 -- Protocol and registration + +**Description:** Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number. + +**What to look for in code/outputs:** +- PROSPERO registration number or other review registry reference +- Published protocol DOI or URL +- Deviation log or amendment documentation if protocol was modified +- Date of registration relative to data collection + +### Item 6 -- Eligibility criteria + +**Description:** Specify study characteristics (e.g., PIRT, minimum sample size, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale. * + +**What to look for in code/outputs:** +- Eligibility criteria documented including study design restrictions +- Index test specification (modality, technology, threshold if fixed) +- Reference standard specification and acceptability criteria +- Language, publication status, and date restrictions with rationale + +### Item 7 -- Information sources + +**Description:** Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and the date last searched. + +**What to look for in code/outputs:** +- Database list with coverage dates (MEDLINE, EMBASE, Cochrane, others) +- Grey literature and conference abstract searches documented +- Date last searched specified +- Supplementary search strategies (reference lists, citation tracking, expert contact) + +### Item 8 -- Search + +**Description:** Present the full electronic search strategy for at least one database, including any limits used, such that it could be repeated. + +**What to look for in code/outputs:** +- Full reproducible search string for at least one database in supplementary materials +- MeSH terms, free-text terms, and Boolean operators documented +- Search filters applied (date, language, human subjects) specified +- Search strategy peer-reviewed or validated using known-item testing + +### Item 9 -- Study selection + +**Description:** State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis). + +**What to look for in code/outputs:** +- Two-stage screening process (title/abstract then full-text) documented +- Number of independent screeners and conflict resolution method described +- PRISMA flow diagram generation code with counts at each stage +- Cohen's kappa or percent agreement between screeners reported + +### Item 10 -- Data collection process + +**Description:** Describe the method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any process for obtaining and confirming data from investigators. + +**What to look for in code/outputs:** +- Data extraction form template or codebook +- Dual extraction with reconciliation process documented +- Author contact log for missing or unclear data +- Standardized variable names for TP, FP, FN, TN counts + +### Item 11 -- Data items + +**Description:** List and define all variables for which data were sought (e.g., PIRT, study characteristics, index test and reference standard details, sources of variability, risk of bias) and any assumptions made. * + +**What to look for in code/outputs:** +- Complete extraction variable list including TP, FP, FN, TN for each threshold +- Index test technical parameters extracted (manufacturer, protocol, reader training) +- Reference standard procedure and interpretation criteria extracted +- Threshold or cutoff values extracted per study + +### Item 12 -- Risk of bias in individual studies + +**Description:** Describe methods used for assessing the risk of bias of individual studies (including specification of whether this was done at the study or outcome level) and how this information is to be used in any data synthesis. * + +**What to look for in code/outputs:** +- QUADAS-2 implementation with domain ratings (Patient selection, Index test, Reference standard, Flow and timing) +- Risk of bias judgments (low, high, unclear) coded per domain per study +- Applicability concerns assessed separately from risk of bias +- Risk of bias summary plots or tables generated + +### Item 13 -- Summary measures + +**Description:** State the principal summary measures (e.g., sensitivity and specificity, likelihood ratios, diagnostic odds ratio, area under the ROC curve). Describe the rationale for the choice of cut-points of the index test, where applicable. * + +**What to look for in code/outputs:** +- Primary accuracy measures specified (paired sensitivity/specificity vs. DOR vs. AUC) +- Threshold selection rationale documented (prespecified vs. study-reported vs. optimal) +- Secondary measures (likelihood ratios, predictive values) defined +- AUC or SROC curve as a planned synthesis output + +### Item 14 -- Synthesis of results + +**Description:** Describe the methods of handling data and combining results of studies, including measures of consistency (for each meta-analysis). Describe any planned exploration of between-study heterogeneity, including subgroup analyses and meta-regression analyses in relation to the index test, reference standard, and clinical characteristics of the included studies. * + +**What to look for in code/outputs:** +- Bivariate model or HSROC (Hierarchical Summary ROC) implementation specified +- Random-effects meta-analysis code for sensitivity and specificity +- Heterogeneity quantification (I-squared, tau-squared for sensitivity and specificity) +- Subgroup analyses prespecified by index test variation, reference standard, or population +- Meta-regression code for covariates hypothesized to explain heterogeneity + +### Item 15 -- Risk of bias across studies + +**Description:** Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies). + +**What to look for in code/outputs:** +- Publication bias assessment code (Deeks' funnel plot asymmetry test for DTA) +- Selective threshold reporting assessment +- Comparison of registered vs. reported outcomes where applicable +- Sensitivity analyses excluding high-risk-of-bias studies + +### Item 16 -- Additional analyses + +**Description:** Describe methods of additional analyses (e.g., sensitivity analyses, subgroup analyses, meta-regression), if done, indicating which were pre-specified. + +**What to look for in code/outputs:** +- Sensitivity analysis code excluding studies with high risk of bias +- Subgroup analysis by index test modality, patient setting, or reference standard +- Meta-regression implementation with covariate definitions +- Pre-specified vs. post-hoc analysis labeling in code comments + +--- + +## Results + +### Item 17 -- Study selection + +**Description:** Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram. + +**What to look for in code/outputs:** +- PRISMA flow diagram generated with counts at each stage +- Exclusion reasons tabulated with frequencies +- Total studies included in qualitative synthesis vs. meta-analysis reported +- Studies included in each subgroup or threshold analysis counted + +### Item 18 -- Study characteristics + +**Description:** For each study, present characteristics for which data were extracted (e.g., study size, PIRT, index test details, reference standard, and prevalence of target condition) and provide the citations. * + +**What to look for in code/outputs:** +- Summary table of included studies with PIRT characteristics +- Index test technical parameters per study (modality, threshold, operator) +- Reference standard procedure and criteria per study +- Target condition prevalence per study reported + +### Item 19 -- Risk of bias within studies + +**Description:** Present data on risk of bias of each included study and, if available, any outcome-level assessment. * + +**What to look for in code/outputs:** +- QUADAS-2 domain ratings displayed per study (traffic light plot or table) +- Applicability concerns reported alongside risk of bias judgments +- Proportion of studies with high/unclear risk of bias per domain summarized +- Sensitivity analysis results comparing high vs. low risk-of-bias studies + +### Item 20 -- Results of individual studies + +**Description:** For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group, and (b) effect estimates and confidence intervals, ideally with a forest plot. * + +**What to look for in code/outputs:** +- 2x2 table data (TP, FP, FN, TN) or sensitivity/specificity per study with 95% CIs +- Forest plots of sensitivity and specificity across studies generated +- Study-level data displayed at each threshold analyzed +- Confidence intervals computed using exact binomial or score methods + +### Item 21 -- Synthesis of results + +**Description:** Present results of each meta-analysis done, including confidence intervals and measures of consistency. * + +**What to look for in code/outputs:** +- Pooled sensitivity and specificity with 95% confidence and prediction regions +- SROC curve with confidence and prediction ellipses generated as figure +- Heterogeneity statistics (I-squared, tau-squared) reported for both sensitivity and specificity +- Bivariate model parameter estimates (mu, sigma, rho) reported + +### Item 22 -- Risk of bias across studies + +**Description:** Present results of any assessment of risk of bias across studies (see Item 15). + +**What to look for in code/outputs:** +- Deeks' funnel plot asymmetry test result reported +- Egger's or related test for small-study effects implemented +- Sensitivity analysis results comparing subsets by risk-of-bias rating +- Publication bias discussion contextualizing test limitations for DTA reviews + +### Item 23 -- Additional analyses + +**Description:** Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]). + +**What to look for in code/outputs:** +- Subgroup and meta-regression results tabulated or plotted +- Pre-specified vs. exploratory status labeled for each additional analysis +- Comparison of pooled estimates across subgroups reported +- Covariate coefficients and confidence intervals from meta-regression reported + +--- + +## Discussion + +### Item 24 -- Summary of evidence + +**Description:** Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers). + +**What to look for in code/outputs:** +- Summary of pooled sensitivity and specificity with clinical interpretation +- Strength-of-evidence statement referencing heterogeneity and risk-of-bias findings +- Clinical utility framed in terms of false-positive and false-negative rates +- Comparison with prior systematic reviews or manufacturer-reported accuracy claims + +### Item 25 -- Limitations + +**Description:** Discuss limitations at study and outcome level (e.g., risk of bias), and at review level (e.g., incomplete retrieval of identified research, reporting bias). + +**What to look for in code/outputs:** +- Limitation discussion referencing QUADAS-2 domain findings +- Grey literature and language restriction impact acknowledged +- Threshold heterogeneity across studies discussed as limitation +- Between-study heterogeneity sources that could not be explained + +### Item 26 -- Conclusions + +**Description:** Provide a general interpretation of the results in the context of other evidence, and implications for future research. + +**What to look for in code/outputs:** +- Conclusion referencing both summary accuracy and certainty of evidence +- Implications for clinical use at the tested threshold(s) +- Recommended future research (prospective validation, threshold standardization, head-to-head comparisons) +- Explicit statement on readiness for clinical implementation + +--- + +## Funding + +### Item 27 -- Funding + +**Description:** Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review. + +**What to look for in code/outputs:** +- Funding acknowledgment section in report +- Conflict of interest disclosures for all authors +- Role of funder in search, selection, analysis, or reporting stated +- Independence of review from test manufacturer or commercial interest diff --git a/plugins/ce-datascience/skills/ce-code-review/references/prisma-ipd-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/prisma-ipd-checklist.md new file mode 100644 index 00000000..52252e66 --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/prisma-ipd-checklist.md @@ -0,0 +1,328 @@ +# PRISMA-IPD Checklist for Systematic Reviews Using Individual Participant Data + +Preferred Reporting Items for Systematic Reviews and Meta-Analyses of Individual Participant Data (PRISMA-IPD). +Extension of PRISMA for systematic reviews and meta-analyses that use individual participant data. + +**Primary reference:** Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Stewart G, Tierney JF; PRISMA-IPD Development Group. Preferred Reporting Items for Systematic Review and Meta-Analyses of Individual Participant Data: The PRISMA-IPD Statement. JAMA. 2015;313(16):1657-1665. doi:10.1001/jama.2015.3656. PMID: 25919529. + +This checklist extends base PRISMA with items specific to IPD meta-analyses (IPDMAs). Items marked with an asterisk (*) are new additions or modifications from base PRISMA. Apply when reporting a systematic review or meta-analysis that uses participant-level data obtained directly from trial or study investigators, rather than relying solely on published aggregate data. The base 27-item PRISMA 2009 checklist remains applicable alongside these extensions. + +--- + +## Title + +### Item 1 -- Title + +**Description:** Identify the report as a systematic review and/or meta-analysis of individual participant data. * + +**What to look for in code/outputs:** +- Report title identifying the review as an IPD meta-analysis or IPD systematic review +- Title naming the intervention(s), condition, and key outcome +- Keywords including "individual participant data," "IPD meta-analysis," or "individual patient data" +- Structured abstract consistent with IPD review scope + +--- + +## Abstract + +### Item 2 -- Structured summary + +**Description:** Provide a structured summary including, as applicable: background, objectives, data sources, eligibility criteria, participants, interventions and comparators, IPD collection, data checking and harmonization, statistical methods, results, and conclusions. * + +**What to look for in code/outputs:** +- Abstract stating the proportion of eligible IPD obtained (e.g., IPD from X of Y eligible studies) +- Summary of data harmonization and checking procedures +- Key efficacy or safety estimates with confidence intervals +- One- vs. two-stage analysis method named in abstract + +--- + +## Introduction + +### Item 3 -- Rationale + +**Description:** Describe the rationale for the review, specifically including why IPD were sought (rather than or in addition to aggregate data), in the context of what is already known. * + +**What to look for in code/outputs:** +- Justification for IPD approach over aggregate data (subgroup analyses, time-to-event, standardized outcomes) +- Documentation of limitations of published aggregate data for the research question +- Prior systematic reviews using aggregate data cited with their limitations +- Specific advantages of IPD for the stated objectives (participant-level subgroups, covariate adjustment) + +### Item 4 -- Objectives + +**Description:** Provide an explicit statement of questions being addressed with reference to participants, interventions, comparators, outcomes, and study design (PICOS), specifying whether an IPD meta-analysis was planned. + +**What to look for in code/outputs:** +- PICOS framework specified +- IPD meta-analysis designated as primary or secondary analysis +- Participant-level subgroup analyses prespecified +- Primary and secondary outcomes defined with measurement methods + +--- + +## Methods + +### Item 5 -- Protocol and registration + +**Description:** Indicate if a review protocol exists, if and where it can be accessed, and, if available, provide registration information including registration number. + +**What to look for in code/outputs:** +- PROSPERO registration number or other review registry reference +- Published protocol DOI or URL +- Deviation log for protocol amendments +- Date of registration relative to data collection and analysis + +### Item 6 -- Eligibility criteria + +**Description:** Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale. Specify whether IPD, aggregate data, or both were sought. * + +**What to look for in code/outputs:** +- Eligibility criteria specifying whether IPD were required for inclusion or merely preferred +- Criteria for accepting aggregate data in place of IPD for non-contributing studies +- Minimum follow-up or sample size thresholds with rationale +- Study design restrictions (RCTs, observational, or both) + +### Item 7 -- Information sources + +**Description:** Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and the date last searched. + +**What to look for in code/outputs:** +- Database list with coverage dates +- Unpublished and ongoing trial registries searched (ClinicalTrials.gov, WHO ICTRP) +- Author and sponsor contact for unreported studies or data +- Date last searched for each source + +### Item 8 -- Search + +**Description:** Present the full electronic search strategy for at least one database, including any limits used, such that it could be repeated. + +**What to look for in code/outputs:** +- Full reproducible search string for at least one database +- MeSH terms, free-text terms, and Boolean operators documented +- Search strategy sufficient to identify unpublished and ongoing studies +- Search peer-reviewed or validated using known-item testing + +### Item 9 -- Study selection + +**Description:** State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis). + +**What to look for in code/outputs:** +- Two-stage screening process with counts at each stage +- Study selection process documented separately from IPD collection outcome +- PRISMA flow diagram generated distinguishing studies providing IPD from those not +- Reasons for non-contribution of IPD documented per study + +### Item 10 -- IPD collection process + +**Description:** Describe all methods used to identify, request, receive, and manage the IPD, including any process for obtaining and confirming data from investigators. * + +**What to look for in code/outputs:** +- Data request process documented (initial contact, reminders, data sharing agreements) +- Data transfer method specified (secure file transfer, data repository) +- Data management plan or data custody documentation referenced +- Timeline from initial contact to data receipt per study + +### Item 11a -- Data items: study-level + +**Description:** List and define all study-level variables for which data were sought (e.g., PICOS, funding sources) and any assumptions made. + +**What to look for in code/outputs:** +- Study-level extraction form with all variables listed +- Study design, intervention details, and setting variables defined +- Risk of bias assessment variables captured at study level +- Statistical analysis plan variables to inform planned analyses + +### Item 11b -- Data items: participant-level + +**Description:** List and define all participant-level variables for which data were sought and any assumptions made. * + +**What to look for in code/outputs:** +- Participant-level variable list including demographic, prognostic, treatment, and outcome variables +- Variable definitions with units, coding schemes, and reference ranges +- Prespecified subgroup variables (age, sex, disease severity, biomarkers) listed +- Outcome variable definitions consistent across studies in the data collection plan + +### Item 12 -- Risk of bias in individual studies + +**Description:** Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level) and how this information is to be used in data synthesis. + +**What to look for in code/outputs:** +- Cochrane RoB 2.0 or RoB 1.0 domain ratings applied +- IPD-specific risk of bias items assessed (selective participant exclusion, differential outcome data availability) +- Risk of bias judgments coded per domain per study +- Sensitivity analyses excluding high-risk-of-bias studies planned + +### Item 13 -- Summary measures + +**Description:** State the principal summary measures (e.g., risk ratio, difference in means). + +**What to look for in code/outputs:** +- Primary effect measure specified (OR, HR, RR, mean difference) +- Absolute and relative measures both specified +- Treatment interaction terms for subgroup analyses defined +- Time-to-event outcome handling specified (Cox model, parametric survival) + +### Item 14 -- Planned methods of analysis + +**Description:** Describe the methods of handling data and combining results of studies, including any planned exploration of between-study heterogeneity. Describe the statistical method used (e.g., one-stage or two-stage approach). Specify any methods used to check, clean, and harmonize the IPD received from different studies. * + +**What to look for in code/outputs:** +- One-stage vs. two-stage analysis approach specified with rationale +- One-stage model specification (mixed-effects regression, stratified Cox, clustered standard errors) +- Two-stage approach with within-study estimate extraction method defined +- Data checking procedures specified (range checks, internal consistency, cross-variable logic) +- Harmonization procedures for differently coded variables across studies +- Missing participant data handling (within-study imputation, missing at random assumption) + +### Item 15 -- Assessment of risk of bias across studies + +**Description:** Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies). + +**What to look for in code/outputs:** +- Comparison of characteristics between contributing and non-contributing studies +- Funnel plot asymmetry test planned +- Comparison of results using IPD vs. published aggregate data where both available +- Selection bias assessment for non-contributing studies + +### Item 16 -- Additional analyses + +**Description:** Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified. + +**What to look for in code/outputs:** +- Participant-level subgroup analyses coded (treatment-by-covariate interaction terms) +- Aggregate vs. IPD result comparison coded as sensitivity analysis +- Meta-regression for between-study moderators specified +- Pre-specified vs. post-hoc labeling for all additional analyses + +--- + +## Results + +### Item 17 -- Study selection + +**Description:** Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram. Report the number of studies and participants (and events, if applicable) for which IPD were obtained. * + +**What to look for in code/outputs:** +- PRISMA flow diagram distinguishing studies providing IPD from non-contributing studies +- Total participant count and event count across IPD-contributing studies +- Reasons for IPD non-contribution per study (refused, unreachable, data lost) +- Proportion of eligible IPD obtained reported (studies and participants) + +### Item 18 -- Study characteristics + +**Description:** For each study, present characteristics for which data were extracted and provide the citations. Present comparisons of characteristics between studies that did and did not provide IPD. * + +**What to look for in code/outputs:** +- Study characteristics table stratified by IPD contributor vs. non-contributor +- Statistical comparison of key characteristics between contributing and non-contributing studies +- Missing IPD impact assessment (would non-contributors change conclusions) +- Treatment arm descriptions and dosing per study + +### Item 19 -- Risk of bias within studies + +**Description:** Present data on risk of bias of each included study and, if available, any outcome-level assessment. + +**What to look for in code/outputs:** +- Risk of bias table or traffic light plot per study and per domain +- IPD-specific risk of bias items reported (completeness of IPD, selective participant exclusion) +- Comparison of risk of bias between IPD contributors and non-contributors +- Sensitivity analysis results for high-risk-of-bias studies + +### Item 20a -- Results of data checking and harmonization + +**Description:** Report on any important issues found when checking and harmonizing the IPD received from different studies, and how these were resolved. * + +**What to look for in code/outputs:** +- Data cleaning log documenting issues found and resolution decisions +- Variable harmonization decisions documented (recoding, combining categories, unit conversion) +- Participant-level data integrity checks and error correction procedures +- Final variable coding scheme with decisions documented per study + +### Item 20b -- Results of individual studies + +**Description:** For all outcomes considered, present, for each study: (a) simple summary data for each intervention group and (b) effect estimates and confidence intervals, ideally with a forest plot. + +**What to look for in code/outputs:** +- Arm-level event rates, means, or survival summaries per study +- Study-level effect estimates with 95% CIs in forest plots +- Participant flow within each contributing study (randomized, analyzed, events) +- Consistency between IPD-derived estimates and published aggregate data checked + +### Item 21 -- Synthesis of results + +**Description:** Present the results of each meta-analysis done, including confidence intervals and measures of consistency. + +**What to look for in code/outputs:** +- Pooled effect estimate with 95% CI from IPD meta-analysis +- Heterogeneity statistics (I-squared, tau-squared) reported +- One-stage and two-stage estimates compared if both computed +- Prediction intervals reported alongside summary estimates + +### Item 22 -- Risk of bias across studies + +**Description:** Present results of any assessment of risk of bias across studies (see Item 15). + +**What to look for in code/outputs:** +- Comparison of pooled estimates from IPD vs. published aggregate data +- Funnel plot and asymmetry test results reported +- Characteristics comparison between contributing and non-contributing studies +- Sensitivity analysis excluding non-contributing studies reported + +### Item 23 -- Additional analyses + +**Description:** Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]). + +**What to look for in code/outputs:** +- Participant-level subgroup interaction estimates with p-values for interaction reported +- Aggregate vs. IPD sensitivity analysis results compared +- Meta-regression covariate estimates tabulated +- Pre-specified vs. exploratory labeling for each analysis + +--- + +## Discussion + +### Item 24 -- Summary of evidence + +**Description:** Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups. + +**What to look for in code/outputs:** +- Summary of pooled IPD estimates with clinical interpretation +- Participant-level subgroup findings contextualized +- IPD advantages over prior aggregate analyses discussed +- Certainty of evidence (GRADE) assessed and reported + +### Item 25 -- Limitations + +**Description:** Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete retrieval of identified research, reporting bias). Describe any limitations of the IPD meta-analysis (e.g., not all eligible IPD obtained, between-study heterogeneity in variable definitions). * + +**What to look for in code/outputs:** +- Missing IPD impact discussed (studies and participant proportions not contributing) +- Variable harmonization limitations and residual heterogeneity in variable definitions +- Missing participant-level data patterns and potential bias impact +- Generalizability limitations from contributing vs. non-contributing study differences + +### Item 26 -- Conclusions + +**Description:** Provide a general interpretation of the results in the context of other evidence, and implications for future research. + +**What to look for in code/outputs:** +- Conclusions referencing both summary estimates and subgroup findings +- Clinical implications of participant-level subgroup analyses +- Recommended future research based on IPD findings +- Data sharing recommendations for future trials to facilitate prospective IPD-MA + +--- + +## Funding + +### Item 27 -- Funding + +**Description:** Describe sources of funding for the systematic review and other support; role of funders for the systematic review. + +**What to look for in code/outputs:** +- Funding acknowledgment in report +- Conflict of interest disclosures for all authors +- Role of funder in IPD collection, analysis, or reporting stated +- Independence of analysis from original study sponsors confirmed diff --git a/plugins/ce-datascience/skills/ce-code-review/references/prisma-nma-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/prisma-nma-checklist.md new file mode 100644 index 00000000..6e6ac06c --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/prisma-nma-checklist.md @@ -0,0 +1,318 @@ +# PRISMA-NMA Checklist for Network Meta-Analyses + +Preferred Reporting Items for Systematic Reviews and Meta-Analyses incorporating Network Meta-Analysis (PRISMA-NMA). +Extension of PRISMA for systematic reviews that include a network meta-analysis. + +**Primary reference:** Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, Ioannidis JPA, Straus S, Thorlund K, Jansen JP, Mulrow C, Catala-Lopez F, Gotzsche PC, Dickersin K, Boutron I, Altman DG, Moher D. The PRISMA Extension Statement for Reporting of Systematic Reviews Incorporating Network Meta-analyses of Health Care Interventions: Checklist and Explanations. Ann Intern Med. 2015;162(11):777-784. doi:10.7326/M14-2385. PMID: 26030634. + +This checklist extends base PRISMA with items specific to network meta-analysis. Items marked with an asterisk (*) are new additions or modifications from base PRISMA. Apply when reporting a systematic review incorporating indirect or mixed treatment comparisons via network meta-analysis (NMA). The base 27-item PRISMA 2009 checklist remains applicable alongside these extensions. + +--- + +## Title + +### Item 1 -- Title + +**Description:** Identify the report as a systematic review incorporating a network meta-analysis. * + +**What to look for in code/outputs:** +- Report title specifying network meta-analysis or indirect treatment comparison +- Title naming the condition, intervention class, or decision context +- Structured abstract consistent with NMA scope and network geometry +- Keywords including "network meta-analysis," "indirect comparisons," and the therapeutic area + +--- + +## Abstract + +### Item 2 -- Structured summary + +**Description:** Provide a structured summary including, as applicable: background, objectives, data sources, study eligibility criteria, participants, interventions, study appraisal and synthesis methods (including any assumptions about transitivity and consistency), results (including the network geometry and ranking of interventions), and conclusions. * + +**What to look for in code/outputs:** +- Abstract reporting the number of treatments in the network and number of studies per comparison +- Summary of ranking results (e.g., SUCRA or P-scores) or selected pairwise estimates +- Transitivity and consistency assessment methods named in abstract +- Network plot or reference to it in abstract + +--- + +## Introduction + +### Item 3 -- Rationale + +**Description:** Describe the rationale for the review in the context of what is already known and specifically why a network meta-analysis is appropriate, including justification that transitivity assumption is likely to hold. * + +**What to look for in code/outputs:** +- Justification for indirect comparisons (absence of head-to-head trials, decision-making need) +- Argument for transitivity based on shared effect modifiers across comparisons +- Documentation of clinical homogeneity across the treatment network +- Reference to existing pairwise meta-analyses and their limitations for decision-making + +### Item 4 -- Objectives + +**Description:** Provide an explicit statement of questions being addressed with reference to participants, interventions, comparators, outcomes, and study design (PICOS). Include the perspective of the network meta-analysis (e.g., decision-making, research synthesis). + +**What to look for in code/outputs:** +- PICOS framework specified for the network +- All treatments (nodes) in the planned network listed +- Primary and secondary outcomes defined +- Target comparisons or decision context specified (e.g., ranking, which treatment is best) + +--- + +## Methods + +### Item 5 -- Protocol and registration + +**Description:** Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number. + +**What to look for in code/outputs:** +- PROSPERO registration number or other review registry reference +- Published protocol DOI or URL +- Deviation log documenting changes from the protocol +- Date of registration relative to data collection + +### Item 6 -- Eligibility criteria + +**Description:** Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale. + +**What to look for in code/outputs:** +- Inclusion criteria specifying which comparisons are eligible (active vs. active, active vs. placebo) +- Outcome measurement time point standardization across the network +- Study design restrictions (RCTs only, or including observational) +- Minimum follow-up or sample size thresholds with rationale + +### Item 7 -- Information sources + +**Description:** Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and the date last searched. + +**What to look for in code/outputs:** +- Database list with coverage dates (MEDLINE, EMBASE, Cochrane CENTRAL, others) +- Grey literature and conference abstract searches documented +- Date last searched for each database +- Author or manufacturer contact for unpublished data + +### Item 8 -- Search + +**Description:** Present the full electronic search strategy for at least one database, including any limits used, such that it could be repeated. + +**What to look for in code/outputs:** +- Full reproducible search string for at least one database +- MeSH terms, free-text terms, and Boolean operators documented +- All relevant intervention terms included to capture the full network +- Search strategy peer-reviewed or validated + +### Item 9 -- Study selection + +**Description:** State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis). + +**What to look for in code/outputs:** +- Two-stage screening process with counts at each stage +- Number of independent screeners and conflict resolution method +- PRISMA flow diagram generated with counts +- Separate reporting of studies excluded from NMA vs. narrative synthesis + +### Item 10 -- Data collection process + +**Description:** Describe the method of data extraction from reports and any process for obtaining and confirming data from investigators. + +**What to look for in code/outputs:** +- Data extraction form for arm-level data (treatment, sample size, events, means, SDs) +- Dual extraction with reconciliation documented +- Author contact log for multi-arm trial data needed for NMA +- Handling of multi-arm trials in extraction (all arms captured) + +### Item 11 -- Data items + +**Description:** List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions made. * + +**What to look for in code/outputs:** +- Complete variable list including arm-level outcome data (events/n for binary, mean/SD for continuous) +- Treatment coding scheme with reference treatment designated +- Study-level potential effect modifier variables extracted for transitivity assessment +- Multi-arm trial flag for appropriate handling in analysis + +### Item 12 -- Risk of bias in individual studies + +**Description:** Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level) and how this information is to be used in data synthesis. + +**What to look for in code/outputs:** +- Cochrane RoB 2.0 or RoB 1.0 domain ratings per study +- Risk of bias judgments coded (low, some concerns, high) per domain +- Sensitivity analysis excluding high-risk-of-bias studies planned +- Risk of bias summary figures or tables generated + +### Item 13 -- Summary measures + +**Description:** State the principal summary measures (e.g., risk ratio, difference in means) including the network meta-analysis summary measures (e.g., odds ratio, mean difference, relative risk, absolute risk difference). * + +**What to look for in code/outputs:** +- Effect measure specified (odds ratio, relative risk, mean difference, hazard ratio) +- Reference treatment for all pairwise comparisons defined +- Absolute and relative effect measures both specified where relevant +- Ranking measures defined (SUCRA, P-score, probability of being best) + +### Item 14 -- Planned methods of analysis + +**Description:** Describe the methods of handling data and combining results of studies including any planned exploration of heterogeneity and inconsistency. Describe the statistical model used (e.g., Bayesian or frequentist), the specific model used for the network meta-analysis (e.g., consistency model), any assumptions made in the analysis, and the methods used to assess transitivity (comparability across treatment comparisons). * + +**What to look for in code/outputs:** +- Statistical framework specified (Bayesian with prior distributions, or frequentist) +- Consistency model specified (common heterogeneity vs. comparison-specific) +- Transitivity assessment method coded (comparison of effect modifier distributions across comparisons) +- Inconsistency tests planned (node-splitting, design-by-treatment interaction, loop-specific inconsistency) +- Network geometry description method (network plot, contribution matrix) + +### Item 15 -- Assessment of risk of bias across studies + +**Description:** Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies). + +**What to look for in code/outputs:** +- Comparison-adjusted funnel plots generated for publication bias assessment +- Selective reporting assessment using registered vs. reported outcomes +- Small-study effects assessment in network context +- Sensitivity analyses for potential publication bias + +### Item 16 -- Additional analyses + +**Description:** Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified. + +**What to look for in code/outputs:** +- Subgroup and meta-regression analyses prespecified with effect modifier rationale +- Sensitivity analyses for key assumptions (consistency, heterogeneity priors) +- Network meta-regression code with treatment-covariate interaction terms +- Pre-specified vs. post-hoc analysis designation in code comments + +--- + +## Results + +### Item 17 -- Study selection + +**Description:** Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram. + +**What to look for in code/outputs:** +- PRISMA flow diagram with counts at each stage +- Studies by comparison (pairwise breakdown of included studies) +- Reasons for exclusion of full-text articles tabulated +- Multi-arm trials identified and counted + +### Item 18 -- Study characteristics + +**Description:** For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations. * + +**What to look for in code/outputs:** +- Summary table with study-level characteristics (design, sample size, comparisons, outcomes) +- Treatment arms and dosing for each study +- Effect modifiers (e.g., baseline risk, disease severity, duration) tabulated per study +- Multi-arm trial indicator noted in study characteristics table + +### Item 19 -- Risk of bias within studies + +**Description:** Present data on risk of bias of each included study and, if available, any outcome-level assessment. + +**What to look for in code/outputs:** +- Risk of bias table or traffic light plot per study and per domain +- Overall risk of bias judgment per study +- Proportion of studies with high or unclear risk per domain summarized +- Sensitivity analysis results for high-risk-of-bias studies + +### Item 20 -- Results of individual studies + +**Description:** For all outcomes considered, present, for each study: (a) simple summary data for each intervention group and (b) effect estimates and confidence intervals, ideally with a forest plot. + +**What to look for in code/outputs:** +- Arm-level event rates or means per study tabulated +- Pairwise effect estimates with CIs displayed in forest plots +- Direct evidence forest plots by comparison pair generated +- Multi-arm trial data displayed with all arms reported + +### Item 21 -- Synthesis of results + +**Description:** Present the results of each analysis, the network geometry (including a network plot), results of the network meta-analysis (including a summary treatment effect for each pairwise comparison, with credible or confidence intervals), and the results for ranking of interventions. * + +**What to look for in code/outputs:** +- Network plot generated with nodes sized by sample size and edges by study count +- League table or matrix of all pairwise treatment effects with 95% CrI/CI +- SUCRA values or P-scores with uncertainty quantification reported +- Rankograms or cumulative ranking plots generated for primary outcome +- Contribution matrix showing direct vs. indirect evidence weights + +### Item 22 -- Exploration of inconsistency + +**Description:** Present the results of any investigation of inconsistency (see Item 14). * + +**What to look for in code/outputs:** +- Node-splitting results with direct, indirect, and network estimates compared +- Design-by-treatment inconsistency test (chi-squared, p-value) reported +- Loop-specific inconsistency factors for identified loops reported +- Overall inconsistency test result (Cochran's Q for design-by-treatment interaction) + +### Item 23 -- Assessment of risk of bias across studies + +**Description:** Present results of any assessment of publication bias across studies (see Item 15). + +**What to look for in code/outputs:** +- Comparison-adjusted funnel plot displayed +- Egger's test or rank correlation test results for asymmetry +- Qualitative discussion of small-study effects in the network +- Sensitivity analysis results for scenarios with selective reporting + +### Item 24 -- Additional analyses + +**Description:** Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]). + +**What to look for in code/outputs:** +- Subgroup and meta-regression estimates tabulated +- Sensitivity analyses (excluding studies, alternative priors, alternative heterogeneity assumptions) reported +- Pre-specified vs. exploratory labeling for each additional analysis +- Comparison of rankings across sensitivity scenarios + +--- + +## Discussion + +### Item 25 -- Summary of evidence + +**Description:** Summarize the main findings including the strength of evidence for each main outcome, and considering its relevance to key groups (e.g., healthcare providers, users, and policy makers). Interpret the results in the context of the transitivity assumption. * + +**What to look for in code/outputs:** +- Summary of NMA estimates with clinical interpretation +- Transitivity evaluation results discussed (similarity of effect modifier distributions) +- Consistency assessment findings integrated into conclusions +- Certainty of evidence (GRADE for NMA) assessed and reported + +### Item 26 -- Limitations + +**Description:** Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete retrieval of identified research, reporting bias). Describe any limitations of the network meta-analysis (e.g., assumptions about transitivity and consistency). * + +**What to look for in code/outputs:** +- Transitivity assumption limitations discussed with supporting evidence +- Inconsistency sources identified and their impact discussed +- Sparse network limitations (few studies per comparison, disconnected nodes) addressed +- Heterogeneity impact on ranking uncertainty discussed + +### Item 27 -- Conclusions + +**Description:** Provide a general interpretation of the results in the context of other evidence, and implications for future research. + +**What to look for in code/outputs:** +- Conclusion referencing summary estimates and ranking with appropriate uncertainty language +- Implications for clinical decision-making and guideline development +- Recommended future research (direct head-to-head trials for key comparisons) +- Explicit statement on conditions under which transitivity assumption holds + +--- + +## Funding + +### Item 28 -- Funding + +**Description:** Describe sources of funding for the systematic review and other support; role of funders for the systematic review. + +**What to look for in code/outputs:** +- Funding acknowledgment section in report +- Conflict of interest disclosures for all authors +- Role of funder in search, selection, analysis, or reporting stated +- Independence of review from pharmaceutical or device industry stated diff --git a/plugins/ce-datascience/skills/ce-code-review/references/prisma-scr-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/prisma-scr-checklist.md new file mode 100644 index 00000000..953fb348 --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/prisma-scr-checklist.md @@ -0,0 +1,255 @@ +# PRISMA-ScR Checklist for Scoping Reviews + +Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR). +22-item checklist applicable to scoping reviews that map the extent, range, and nature of evidence on a topic. + +**Primary reference:** Tricco AC, Lillie E, Zarin W, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467-473. doi:10.7326/M18-0850. PMID: 30178033. + +Each item includes the section, description, and what to look for in analysis code and outputs +when evaluating compliance. Items marked with an asterisk (*) are additions to the base PRISMA checklist. + +--- + +## Title + +### Item 1 -- Title + +**Description:** Identify the report as a scoping review. + +**What to look for in code/outputs:** +- Report title or rendered document header explicitly including "scoping review" +- Title generation code or manuscript template that includes the study type designation +- Output documents whose title clearly differentiates the work from a systematic review or narrative review + +--- + +## Abstract + +### Item 2 -- Structured summary + +**Description:** Provide a structured summary that includes the background, objectives, eligibility criteria, sources of evidence, charting methods, results, and conclusions that relate to the review questions and objectives. + +**What to look for in code/outputs:** +- Abstract template or generation code with structured sections covering all required elements +- Explicit mention of the concept being mapped and the context (population, concept, context framework) +- Key quantitative findings (number of sources, date range, study designs) in the abstract output +- Rendered summary that identifies knowledge gaps or areas requiring future research + +--- + +## Introduction + +### Item 3 -- Rationale* + +**Description:** Describe the rationale for the review in the context of what is already known, explaining why the review questions or objectives lend themselves to a scoping review approach. + +**What to look for in code/outputs:** +- Documentation explicitly justifying the scoping review methodology over a systematic review or meta-analysis +- Text distinguishing the current goal (mapping evidence, identifying gaps) from synthesizing intervention effectiveness +- References to prior reviews and the specific gap or breadth of topic that motivates the scoping approach +- Report sections or comments explaining the appropriateness of scoping review for an emerging or heterogeneous evidence base + +### Item 4 -- Objectives* + +**Description:** Provide an explicit statement of the questions and objectives being addressed with reference to their key elements (e.g., population or participants, concept, and context) or other relevant key elements used to conceptualize the review questions and/or objectives. + +**What to look for in code/outputs:** +- PICO or PCC (population, concept, context) framework documented in protocol, report preamble, or config file +- Explicit research question statements mapping to population/participants, concept, and context +- Search strategy documentation that operationalizes the stated objectives +- Objectives framed as mapping or describing rather than assessing effectiveness or synthesizing effect sizes + +--- + +## Methods + +### Item 5 -- Protocol and registration + +**Description:** Indicate whether a review protocol exists; state if and where it can be accessed (e.g., a web address); and if available, provide registration information including the registration number. + +**What to look for in code/outputs:** +- Protocol document path or URL referenced in analysis code or report +- PROSPERO, OSF, or institutional registration number in report metadata or methods section +- Version-controlled protocol files in the project repository +- Documentation of any deviations from the registered protocol with explanations + +### Item 6 -- Eligibility criteria + +**Description:** Specify characteristics of the sources of evidence used as eligibility criteria (e.g., years considered, language, and publication status) and provide a rationale. + +**What to look for in code/outputs:** +- Inclusion and exclusion criteria for study designs, languages, date ranges, and publication types in code or documentation +- Rationale for chosen eligibility criteria referencing the review scope and concept being mapped +- Code or decision rules that apply eligibility criteria consistently across sources +- Documentation of whether grey literature, conference abstracts, or unpublished studies were included + +### Item 7 -- Information sources + +**Description:** Describe all information sources in the search (e.g., databases with dates of coverage and contact with authors to identify additional sources) and the date the most recent search was executed. + +**What to look for in code/outputs:** +- List of databases searched (MEDLINE, Embase, CINAHL, etc.) with coverage dates and search execution dates +- Code or documentation covering supplementary search methods (hand-searching, citation chasing, grey literature sources) +- Expert contact or key informant consultation documented in the methods +- Date of the most recent search execution recorded in code metadata or report header + +### Item 8 -- Search + +**Description:** Present the full electronic search strategy for at least one database, including any limits used, such that it could be repeated. + +**What to look for in code/outputs:** +- Full search string for at least one database included in supplementary materials or methods appendix +- Boolean operators, MeSH terms, free-text terms, and field tags documented +- Any limits applied (date, language, publication type) specified in the search strategy +- Search strings stored as reproducible code or query files + +### Item 9 -- Selection of sources of evidence* + +**Description:** State the process for selecting sources of evidence (i.e., screening and eligibility) included in the scoping review. + +**What to look for in code/outputs:** +- Documentation of the screening process: title/abstract screening followed by full-text review +- Number of reviewers at each stage and any conflict resolution process described +- Screening tool or data extraction platform referenced (Covidence, Rayyan, DistillerSR) +- Pilot testing of eligibility criteria documented prior to full screening + +### Item 10 -- Data charting process* + +**Description:** Describe the methods of charting data from the included sources of evidence (e.g., calibrated forms or forms that have been pilot-tested by the team before widespread use, and whether data charting was done independently or in duplicate) and any processes for obtaining and confirming data from investigators. + +**What to look for in code/outputs:** +- Data charting form or template referenced or included in supplementary materials +- Documentation of whether charting was done independently by two reviewers and how conflicts were resolved +- Pilot testing of the charting form before full extraction documented +- Process for contacting study authors to confirm or supplement charted data described + +### Item 11 -- Data items* + +**Description:** List and define all variables for which data were sought and any assumptions and simplifications made. + +**What to look for in code/outputs:** +- Data dictionary or codebook listing all charted variables with definitions +- Code that maps extracted data fields to analysis variables +- Documentation of assumptions made when source information was incomplete or ambiguous +- Simplifications applied to heterogeneous data types (e.g., collapsing outcome categories) justified and recorded + +### Item 12 -- Critical appraisal of individual sources of evidence* + +**Description:** If done, provide a rationale for conducting a critical appraisal of included sources of evidence; describe the methods used and how this information was used in any data synthesis. + +**What to look for in code/outputs:** +- Documentation of whether formal quality appraisal was performed and the stated rationale +- Quality appraisal tool referenced when appraisal was conducted (noting that quality appraisal is optional in scoping reviews) +- Explanation of how quality appraisal findings influenced data synthesis, narrative, or conclusions +- Acknowledgment that scoping reviews do not typically exclude studies based on methodological quality + +### Item 13 -- Synthesis of results + +**Description:** Describe the methods of handling and summarizing the data that were charted. + +**What to look for in code/outputs:** +- Narrative synthesis approach described (thematic analysis, descriptive mapping, content analysis) +- Code or documentation for tabulating study characteristics (design, setting, population, concept) +- Quantitative summary methods (counts, frequencies, proportions) used to characterize the evidence base +- Absence of pooled meta-analysis consistent with scoping review purpose; if meta-analysis is performed, rationale provided + +--- + +## Results + +### Item 14 -- Selection of sources of evidence + +**Description:** Give numbers of sources of evidence screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally using a flow diagram. + +**What to look for in code/outputs:** +- PRISMA-style flow diagram generation code with counts at each stage (records identified, deduplicated, screened, assessed, included) +- Reasons for exclusion at full-text stage with counts per reason +- Final included source count consistent across flow diagram, results text, and data tables +- Kappa or percent agreement statistics for inter-rater reliability at screening stages + +### Item 15 -- Characteristics of sources of evidence* + +**Description:** For each source of evidence, present characteristics for which data were charted and provide the citations. + +**What to look for in code/outputs:** +- Table summarizing characteristics of all included sources (author, year, country, design, population, concept, context, key findings) +- Code generating the characteristics table from the charted data file +- Complete citation information for all included sources +- Logical consistency between the characteristics table and the synthesis narrative + +### Item 16 -- Critical appraisal within sources of evidence + +**Description:** If done, present data on critical appraisal of included sources of evidence (see item 12). + +**What to look for in code/outputs:** +- Quality appraisal scores or ratings table if appraisal was performed +- Narrative interpretation of quality findings and their implications for the review conclusions +- Consistency between reported appraisal approach in methods and results presented + +### Item 17 -- Results of individual sources of evidence + +**Description:** For each included source of evidence, present the relevant data that were charted that relate to the review questions and objectives. + +**What to look for in code/outputs:** +- Per-source data table or appendix with charted data aligned to review objectives +- Code that outputs source-level results alongside aggregate summaries +- Data items presented consistently for all sources, with missing data noted +- Source-level data traceable to the final synthesis narrative + +### Item 18 -- Synthesis of results + +**Description:** Summarize and/or present the charting results as they relate to the review questions and objectives. + +**What to look for in code/outputs:** +- Narrative synthesis organized by the concept dimensions, population subgroups, or contexts identified in the objectives +- Tables or figures summarizing the distribution of evidence by design, geography, date, setting, or population +- Counts and proportions describing the evidence landscape (e.g., proportion of studies from high-income countries) +- Identification of patterns, gaps, and heterogeneity across included sources + +--- + +## Discussion + +### Item 19 -- Summary of evidence* + +**Description:** Summarize the main results (including an overview of concepts, themes, and types of evidence available), link to the review questions and objectives, and consider the relevance to key stakeholders. + +**What to look for in code/outputs:** +- Discussion sections linking findings back to the stated objectives and concept framework +- Summary of the nature and extent of the evidence base mapped +- Explicit identification of knowledge gaps and areas where evidence is sparse or absent +- Consideration of implications for policy, practice, or future research in the context of the mapped concept + +### Item 20 -- Limitations + +**Description:** Discuss the limitations of the scoping review process. + +**What to look for in code/outputs:** +- Limitation sections addressing search comprehensiveness, language restrictions, grey literature coverage, and charting consistency +- Discussion of the impact of excluding quality appraisal on the conclusions +- Acknowledgment of heterogeneity in how the concept is defined across included sources +- Documentation of any deviations from the protocol and their potential effect on results + +### Item 21 -- Conclusions + +**Description:** Provide a general interpretation of the results with respect to the review questions and objectives, as well as potential implications and/or next steps. + +**What to look for in code/outputs:** +- Conclusion statements that directly address the stated objectives and concept being mapped +- Explicit identification of areas requiring systematic review, primary research, or policy action +- Language framing conclusions as mapping the evidence landscape rather than establishing effectiveness +- Recommended next steps grounded in the identified knowledge gaps + +--- + +## Funding + +### Item 22 -- Funding + +**Description:** Describe sources of funding for the included sources of evidence, as well as sources of funding for the scoping review. Describe the role of the funders of the scoping review. + +**What to look for in code/outputs:** +- Funding acknowledgment section in report templates naming the funding source and grant numbers +- Explicit role-of-funder statement (data access, analytical oversight, publication rights) +- Conflict of interest disclosures for the review team +- Funding sources of included primary studies summarized or noted when relevant to interpretation diff --git a/plugins/ce-datascience/skills/ce-code-review/references/quadas-2-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/quadas-2-checklist.md new file mode 100644 index 00000000..e043287e --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/quadas-2-checklist.md @@ -0,0 +1,154 @@ +# QUADAS-2 Tool for Diagnostic Accuracy Studies + +Quality Assessment of Diagnostic Accuracy Studies, second version (QUADAS-2). +Structured tool for assessing the quality and applicability of primary diagnostic accuracy studies included in systematic reviews, comprising four domains each evaluated for risk of bias and applicability concerns. + +**Primary reference:** Whiting PF, Rutjes AWS, Westwood ME, et al; QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529-536. doi:10.7326/0003-4819-155-8-201110180-00009. PMID: 22007046. + +QUADAS-2 is applied study-by-study during data extraction for a systematic review of diagnostic accuracy. Each of the four domains is assessed for (1) risk of bias and (2) applicability concerns (where applicable). Signaling questions guide the risk of bias judgment. This checklist describes each domain, its signaling questions, and what to look for in analysis code and outputs when evaluating compliance during a code or analytic review. + +--- + +## Domain 1: Patient Selection + +### Risk of Bias -- Patient Selection + +**Description:** Assess whether the selection of patients was likely to introduce bias. Key concerns are case-control design (which artificially inflates spectrum by selecting known cases and known controls) and inappropriate exclusions that restrict the spectrum of disease or conditions seen in practice. + +Signaling questions: +- Was a consecutive or random sample of patients enrolled? +- Was a case-control design avoided? +- Did the study avoid inappropriate exclusions? + +**What to look for in code/outputs:** +- Data extraction fields recording enrollment method (consecutive, random, convenience, case-control) populated for each study +- Code that flags studies using a case-control design and applies a high risk of bias rating for this domain +- Exclusion criteria from each study documented in the extraction database and coded for appropriateness +- Summary statistics tabulating the proportion of studies using consecutive/random enrollment vs. other methods +- Risk of bias judgment coded as low, high, or unclear with the signaling question responses recorded as the basis + +### Applicability Concerns -- Patient Selection + +**Description:** Assess whether the patients included in the study match the review question in terms of the intended use population (spectrum of disease, setting, prior test results, and disease stage). + +**What to look for in code/outputs:** +- Applicability judgment (low, high, or unclear concern) coded separately from the risk of bias judgment +- Data extraction fields capturing the study population's setting, disease spectrum, and severity compared to the review's target population +- Code or notes documenting the specific applicability concern when the study population departs from the review question +- Narrative summary in the review discussing applicability concerns across the body of evidence + +--- + +## Domain 2: Index Test + +### Risk of Bias -- Index Test + +**Description:** Assess whether the conduct or interpretation of the index test (the test being evaluated) could have introduced bias. Key concerns are failure to pre-specify the positivity threshold and knowledge of the reference standard result when interpreting the index test. + +Signaling questions: +- Were the index test results interpreted without knowledge of the results of the reference standard? +- If a threshold was used, was it pre-specified? + +**What to look for in code/outputs:** +- Data extraction fields recording whether index test interpretation was blinded to reference standard results +- Code flagging studies where blinding to the reference standard was not maintained or not reported +- Threshold pre-specification status extracted and coded for each study (pre-specified from prior literature, from a training set, or determined in the same dataset) +- Studies using data-driven optimal cutoff thresholds from the same dataset flagged as high risk of bias +- Risk of bias judgment coded as low, high, or unclear with signaling question responses recorded + +### Applicability Concerns -- Index Test + +**Description:** Assess whether the index test, its conduct, or its interpretation differ from the way it is described in the review question (e.g., different equipment, operator training, or scoring system). + +**What to look for in code/outputs:** +- Applicability judgment coded for the index test separately from risk of bias +- Data extraction fields capturing the index test version, protocol, operator training level, and equipment compared to the target use scenario +- Code or notes flagging studies using a prototype, research-grade, or early-generation version of a test when the review covers the commercial version +- Narrative discussion of applicability concerns related to index test implementation across studies + +--- + +## Domain 3: Reference Standard + +### Risk of Bias -- Reference Standard + +**Description:** Assess whether the reference standard is likely to correctly classify the target condition and whether knowledge of the index test result influenced the reference standard interpretation. + +Signaling questions: +- Is the reference standard likely to correctly classify the target condition? +- Were the reference standard results interpreted without knowledge of the results of the index test? + +**What to look for in code/outputs:** +- Data extraction fields recording the reference standard used in each study and an assessment of its accuracy for classifying the target condition +- Code flagging studies where the reference standard is imperfect (e.g., a single biopsy when composite histopathology plus follow-up is the accepted standard) +- Blinding of reference standard interpretation to index test results extracted and coded per study +- Studies using differential verification (different reference standards applied to index-positive and index-negative patients) identified and flagged +- Risk of bias judgment coded as low, high, or unclear for this domain + +### Applicability Concerns -- Reference Standard + +**Description:** Assess whether the target condition as defined by the reference standard matches the target condition of the review question. + +**What to look for in code/outputs:** +- Applicability judgment coded for the reference standard separately from risk of bias +- Data extraction fields capturing the reference standard definition compared to the review's target condition definition +- Code or notes flagging discordant definitions (e.g., review targets clinically significant disease but study uses any pathological abnormality) +- Summary of applicability concerns related to reference standard definition variation across the body of evidence + +--- + +## Domain 4: Flow and Timing + +### Risk of Bias -- Flow and Timing + +**Description:** Assess whether the time interval between the index test and reference standard was appropriate (short enough that disease status could not change), whether all patients received the same reference standard, and whether all enrolled patients were included in the analysis. + +Signaling questions: +- Was there an appropriate interval between the index test and reference standard? +- Did all patients receive a reference standard? +- Did patients receive the same reference standard? +- Were all patients included in the analysis? + +**What to look for in code/outputs:** +- Data extraction fields recording the interval between index test and reference standard for each study +- Code flagging studies with prolonged intervals where disease progression or resolution between tests is plausible +- Partial verification identified: studies where only a subset of patients (typically index-positive patients) received the reference standard flagged and coded +- Differential verification identified: studies where different reference standards were applied to different patient subgroups coded as high risk +- Proportion of enrolled patients included in the final analysis extracted and studies with >10-15% exclusion from the analysis flagged +- Risk of bias judgment coded as low, high, or unclear with all four signaling question responses recorded + +--- + +## Presentation and Summary + +### Risk of Bias Summary Table* + +**Description:** Present a summary of risk of bias judgments across all four domains for each included study, typically as a colored matrix (green/low, yellow/unclear, red/high). + +**What to look for in code/outputs:** +- Code generating a risk of bias summary table or figure (e.g., using robvis, ggplot2, or equivalent tool) with one row per study and one column per domain +- Color coding or symbol coding for low, high, and unclear risk of bias +- Both risk of bias and applicability concern judgments included in the table or presented in parallel tables +- All included studies represented in the table with no missing entries +- Source data file (CSV, spreadsheet) with the individual signaling question responses for each study available for audit + +### Risk of Bias Across Studies* + +**Description:** Summarize the proportion of studies at low, high, and unclear risk of bias for each domain across the included studies, and interpret the impact of the risk of bias profile on the overall strength of evidence. + +**What to look for in code/outputs:** +- Code computing the proportion of studies at each risk level per domain +- Bar chart or proportion table summarizing the risk of bias distribution across the body of evidence +- Sensitivity analyses in the meta-analysis restricting to studies at low risk of bias for one or more domains +- Discussion of how risk of bias findings affect confidence in the pooled sensitivity and specificity estimates +- Funnel plot asymmetry or meta-regression examining the association between risk of bias domain scores and diagnostic accuracy estimates + +### Applicability Summary* + +**Description:** Summarize applicability concerns across the four domains for all included studies and discuss how applicability findings affect the generalizability of the review's conclusions to the target population and setting. + +**What to look for in code/outputs:** +- Separate applicability summary table or figure alongside the risk of bias summary +- Code or notes documenting the specific applicability gaps identified (population mismatch, different test version, different reference standard definition) +- Narrative discussion explicitly addressing each applicability domain and its implications for the review conclusions +- Subgroup analyses or sensitivity analyses by applicability rating when sufficient studies allow diff --git a/plugins/ce-datascience/skills/ce-code-review/references/squire-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/squire-checklist.md new file mode 100644 index 00000000..fad234f2 --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/squire-checklist.md @@ -0,0 +1,220 @@ +# SQUIRE 2.0 Checklist for Quality Improvement Studies + +Standards for QUality Improvement Reporting Excellence (SQUIRE 2.0). +18-item checklist for reporting quality improvement work in healthcare. + +**Primary reference:** Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf. 2016;25(12):986-992. doi:10.1136/bmjqs-2015-004411. PMID: 25505110. + +**Explanation and elaboration:** Goodman D, Ogrinc G, Davies L, Baker GR, Barnsteiner J, Foster TC, et al. Explanation and elaboration of the SQUIRE (Standards for Quality Improvement Reporting Excellence) Guidelines, V.2.0: examples of SQUIRE elements in the healthcare improvement literature. BMJ Qual Saf. 2016;25(12):e7. doi:10.1136/bmjqs-2015-004480. PMID: 26385936. + +Apply when reporting studies that describe healthcare quality, safety, or value improvement interventions in local or organizational contexts. SQUIRE 2.0 acknowledges that QI work is complex, context-dependent, and iterative. Items should be addressed as fully as their relevance allows; not all items apply to every study. + +--- + +## Title and Abstract + +### Item 1 -- Title + +**Description:** Indicate that the manuscript concerns an initiative to improve healthcare (broadly defined to include the quality, safety, effectiveness, patient-centeredness, timeliness, efficiency, and equity of healthcare). + +**What to look for in code/outputs:** +- Report title naming the improvement initiative, care setting, and target outcome +- Title indicating the healthcare context (e.g., hospital, clinic, health system) +- Keywords including improvement-relevant terms (quality improvement, patient safety, implementation) +- Language distinguishing from a traditional clinical research study + +### Item 2 -- Abstract + +**Description:** Summarize the background and importance of the problem, the aim, setting, participant/patient population, study design and analytical methods, key findings, and conclusions. + +**What to look for in code/outputs:** +- Abstract including the specific problem and its measured baseline magnitude +- Aim and primary measure of improvement stated in abstract +- Setting and population described in abstract +- Key results with pre/post or time-series comparison reported + +--- + +## Introduction + +### Item 3 -- Problem description + +**Description:** Nature and significance of the local problem. Describe what is known about the problem, why it is important to address, and how it affects patients, staff, or the organization. + +**What to look for in code/outputs:** +- Baseline performance data or measure of the problem's magnitude +- Patient harm, inefficiency, or inequity quantified at the local level +- Local context differentiated from the general literature +- Stakeholder identification (who is affected and how) + +### Item 4 -- Available knowledge + +**Description:** Summary of what is currently known about the problem, including relevant previous studies. + +**What to look for in code/outputs:** +- Literature review documenting prior QI or implementation studies addressing the same problem +- Evidence base for the proposed intervention (clinical guidelines, systematic reviews) +- Identification of successful implementations elsewhere and their key features +- Gaps between evidence and current local practice documented + +### Item 5 -- Rationale + +**Description:** Informal or formal frameworks, models, concepts, and/or theories used to explain the problem, any reasons or assumptions that were used to develop the intervention(s), and reasons why the intervention(s) was expected to work. + +**What to look for in code/outputs:** +- Theoretical framework cited (e.g., Model for Improvement, Lean, Theory of Constraints, Behavior Change Wheel) +- Driver diagram, logic model, or causal map linking intervention components to outcomes +- Assumptions about local factors that will support or hinder the intervention +- Mechanism of action described (how the intervention will change behavior or process) + +### Item 6 -- Specific aims + +**Description:** Purpose of the project and of this report. + +**What to look for in code/outputs:** +- Specific, measurable aim statement with target, timeline, and population +- SMART aim format (Specific, Measurable, Achievable, Relevant, Time-bound) +- Primary and secondary aims distinguished +- Alignment between aim statement and analytic approach + +--- + +## Methods + +### Item 7 -- Context + +**Description:** Contextual elements considered important at the outset of introducing the intervention(s). Describe the setting, the local and system factors that were important for understanding the problem, and the people involved in delivering care. + +**What to look for in code/outputs:** +- Care setting described (hospital unit, ambulatory clinic, community setting) +- Local organizational factors documented (staffing, resources, culture, competing priorities) +- Patient population characteristics relevant to the improvement aim +- Historical context of prior improvement efforts at the site + +### Item 8 -- Intervention(s) + +**Description:** Description of the intervention(s) in sufficient detail that others could reproduce it. Describe what was done for whom, by whom, in what setting and when (the what, who, where, and when of the intervention). + +**What to look for in code/outputs:** +- Intervention components described with sufficient detail for replication +- Stakeholders involved in intervention delivery identified +- Timing and duration of the intervention specified +- Change package or bundle components documented +- Iterative refinements to the intervention over time described + +### Item 9 -- Study of the intervention + +**Description:** Approach used to assess the impact of the intervention(s). Describe the type of study design used (e.g., pre-post, interrupted time series, run chart, control chart). + +**What to look for in code/outputs:** +- Study design specified (pre-post, interrupted time series, stepped wedge, concurrent control) +- Unit of analysis identified (patient encounter, individual patient, provider, unit) +- Run charts or statistical process control charts generated +- Baseline and follow-up measurement periods defined + +### Item 10 -- Measures + +**Description:** Measures chosen for studying processes and outcomes of the intervention(s), including rationale for choosing them, their operational definitions, and how data were collected. + +**What to look for in code/outputs:** +- Outcome measures operationally defined (numerator, denominator, data source) +- Process measures defined alongside outcome measures +- Balancing measures identified (to detect unintended consequences) +- Measurement frequency and data collection method specified +- Reliability or validity of measures documented + +### Item 11 -- Analysis + +**Description:** Approach to assessing the impact of the intervention(s). Include the approach to analyzing quantitative data (e.g., statistical methods and software used) and qualitative data (e.g., content analysis, thematic analysis). + +**What to look for in code/outputs:** +- Statistical method specified and appropriate for study design (SPC, ITS regression, difference-in-differences) +- Run chart or SPC rules for special cause variation defined and applied in code +- Segmented regression or ITS model code for interrupted time series +- Qualitative analysis method described for mixed-methods components +- Software and package versions specified + +### Item 12 -- Ethical considerations + +**Description:** Ethical aspects of implementing and studying the improvement, including potential conflicts of interest, how they were managed, and the process for obtaining any required institutional permissions. + +**What to look for in code/outputs:** +- IRB or ethics board determination (QI exemption vs. full review) documented +- Informed consent process described or waiver documented +- Conflict of interest disclosures +- Potential dual-role conflicts (clinician-researcher) acknowledged and managed + +--- + +## Results + +### Item 13 -- Results + +**Description:** Initial steps of the intervention and the context in which it was implemented; changes to the intervention during the project (with rationale); details of the process measures and outcome measures; and any harms to participants and unintended consequences. + +**What to look for in code/outputs:** +- Timeline of intervention implementation with PDSA cycles or phases documented +- Run charts or SPC charts for primary process and outcome measures over time +- Baseline and post-intervention descriptive statistics reported +- Any modifications to the intervention documented with rationale +- Balancing measures reported to detect unintended consequences +- Participant flow (patients or encounters included vs. excluded from measurement) + +--- + +## Discussion + +### Item 14 -- Summary + +**Description:** Key findings, including relevance to the rationale and specific aims. Consider the overall implications with respect to the aim. + +**What to look for in code/outputs:** +- Summary of observed improvement magnitude relative to the stated aim +- Process measure findings linked to outcome measure findings +- Alignment between theoretical framework predictions and observed results +- Practical significance of improvement assessed alongside statistical signal + +### Item 15 -- Interpretation + +**Description:** Nature of the association between the intervention(s) and the outcomes; any observed mechanisms; unanticipated findings; sustainability; and transferability of findings. + +**What to look for in code/outputs:** +- Causal pathway between intervention components and outcomes discussed +- Rival explanations (secular trends, concurrent changes) addressed +- Unanticipated consequences identified and explained +- Sustainability of improvement over time assessed in time series data +- Transferability to other settings discussed with relevant contextual factors + +### Item 16 -- Limitations + +**Description:** Limits to the generalizability of the work. Include factors that may limit internal validity such as confounding, bias, or imprecision in the design, methods, measurement, or analysis. + +**What to look for in code/outputs:** +- Internal validity limitations (confounding, selection bias, measurement bias) discussed +- Absence of a concurrent control group acknowledged as limitation +- Hawthorne effect or regression to the mean addressed +- Local context features that may limit transferability identified + +### Item 17 -- Conclusions + +**Description:** Usefulness of the work; sustainability; potential for spread to other contexts; implications for practice and for further study in the field. + +**What to look for in code/outputs:** +- Actionable conclusions for similar settings +- Spread strategy for scaling the intervention +- Recommended adaptations for different contexts +- Implications for future QI research or evaluation + +--- + +## Other Information + +### Item 18 -- Funding + +**Description:** Sources of funding that supported this work. Role, if any, of the funding organization in the design, implementation, interpretation, and reporting. + +**What to look for in code/outputs:** +- Funding acknowledgment in report +- Grant numbers or organizational support documented +- Role of funder in QI project design or data access stated +- Conflict of interest disclosures for all authors diff --git a/plugins/ce-datascience/skills/ce-code-review/references/stard-ai-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/stard-ai-checklist.md new file mode 100644 index 00000000..fa67d13a --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/stard-ai-checklist.md @@ -0,0 +1,121 @@ +# STARD-AI Checklist for AI-Assisted Diagnostic Accuracy Studies + +Standards for Reporting of Diagnostic Accuracy Studies -- Artificial Intelligence (STARD-AI). +Extension of STARD for studies evaluating AI/ML-based diagnostic tests against a reference standard. + +**Primary reference:** Sounderajah V, Ashrafian H, Aggarwal R, De Fauw J, Denniston AK, Greaves F, Karthikesalingam A, King D, Liu X, Markar SR, McInnes MDF, Panch T, Pearson-Stuttard J, Ting DSW, Golub RM, Moher D, Bossuyt PM, Darzi A. Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: The STARD-AI Steering Group. Nat Med. 2020;26(6):807-808. doi:10.1038/s41591-020-0941-1. PMID: 32514173. + +**Companion:** Defined based on Sounderajah V et al. A quality assessment tool for artificial intelligence-centered diagnostic test accuracy studies: QUADAS-AI. Nat Med. 2024. And the STARD-AI extension items published alongside the STARD 2015 update. + +This checklist supplements STARD 2015. Apply alongside the base STARD checklist when the index test uses AI/ML. + +--- + +## Title and Abstract + +### STARD-AI 1 -- AI in title + +**Description:** Identify the study as evaluating an AI-based diagnostic test. Specify the type of AI (e.g., deep learning, machine learning, NLP). + +**What to look for in code/outputs:** +- Title includes "artificial intelligence", "machine learning", "deep learning", or the specific AI technique +- Abstract specifies the AI method and its intended diagnostic task + +--- + +## Methods + +### STARD-AI 4a -- AI system description + +**Description:** Describe the AI system including: input data type and format, model architecture, training data source and size, output type (binary, probabilistic, segmentation), and software/hardware used. + +**What to look for in code/outputs:** +- Model architecture documented (e.g., ResNet-50, BERT, XGBoost) +- Training dataset described: source, size, class distribution, inclusion/exclusion criteria +- Input preprocessing pipeline documented (resizing, normalization, augmentation) +- Output format specified (class label, probability, heatmap, segmentation mask) +- Software framework and version (PyTorch, TensorFlow, scikit-learn) documented + +### STARD-AI 4b -- AI version and threshold + +**Description:** Report the specific version of the AI system evaluated, including any threshold or operating point used to convert continuous output to a diagnostic decision. + +**What to look for in code/outputs:** +- Model version or checkpoint identifier documented +- Classification threshold documented (e.g., probability > 0.5 for positive) +- If multiple thresholds were evaluated, report which was pre-specified vs post-hoc + +### STARD-AI 5 -- Intended use and clinical role + +**Description:** State the intended clinical role of the AI system: triage, standalone replacement, add-on to clinical assessment, or second reader. + +**What to look for in code/outputs:** +- Intended use statement in documentation or report preamble +- Clinical workflow integration described (where in the diagnostic pathway does the AI fit?) +- Target user population specified (radiologist, GP, screening program) + +### STARD-AI 7 -- Reference standard independence + +**Description:** Describe whether the reference standard was determined independently from the AI system output. Report whether clinicians establishing the reference standard were blinded to AI results. + +**What to look for in code/outputs:** +- Blinding protocol documented for reference standard assessors +- Independence of reference standard from AI predictions verified +- If reference standard was partially informed by AI output, this is documented as a limitation + +### STARD-AI 9 -- Data partitioning + +**Description:** Describe how data were partitioned for development and evaluation. Report whether temporal, geographic, or institutional splitting was used. + +**What to look for in code/outputs:** +- Train/validation/test split strategy documented (random, temporal, institutional, geographic) +- Temporal validation: training data predates test data +- External validation: test data from a different institution or population +- No data leakage between partitions + +### STARD-AI 10 -- Subgroup specification + +**Description:** Pre-specify subgroups for analysis of diagnostic performance, including demographic groups and clinical subtypes. + +**What to look for in code/outputs:** +- Subgroup analysis plan documented before analysis +- Subgroups by age, sex, race/ethnicity, disease severity, imaging device, or site +- Performance metrics reported separately per subgroup + +--- + +## Results + +### STARD-AI 19 -- Performance metrics + +**Description:** Report standard diagnostic accuracy metrics: sensitivity, specificity, PPV, NPV, AUC/AUROC, and calibration. Report with confidence intervals. + +**What to look for in code/outputs:** +- Sensitivity, specificity, PPV, NPV with 95% CIs +- AUC/AUROC with 95% CI (bootstrapped or DeLong) +- Calibration plot or calibration-in-the-large / calibration slope +- Operating point on the ROC curve identified + +### STARD-AI 21 -- Failure analysis + +**Description:** Report cases where the AI system failed (false positives, false negatives). Describe patterns in failure cases. + +**What to look for in code/outputs:** +- Error analysis code examining false positives and false negatives +- Pattern identification in failure cases (e.g., specific subtypes, image quality, demographics) +- Confusion matrix at the operating point +- Examples of failure cases when possible (with appropriate de-identification) + +--- + +## Discussion + +### STARD-AI 25 -- Generalizability of AI + +**Description:** Discuss the generalizability of the AI system performance to populations, settings, and workflows not represented in the evaluation data. + +**What to look for in code/outputs:** +- Discussion of domain shift risks (different scanners, populations, disease prevalence) +- Comparison of training/test data demographics +- Limitations related to single-site evaluation, narrow demographics, or specific hardware +- Plans for prospective or multi-site validation diff --git a/plugins/ce-datascience/skills/ce-code-review/references/start-rwe-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/start-rwe-checklist.md new file mode 100644 index 00000000..af794370 --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/start-rwe-checklist.md @@ -0,0 +1,237 @@ +# STaRT-RWE Checklist for Real-World Evidence Studies + +Structured Template for Planning and Reporting on the Implementation of Real World Evidence (STaRT-RWE). +Checklist for planning and reporting real-world evidence studies intended for regulatory decision-making. + +**Primary reference:** Wang SV, Schneeweiss S, RWE DUPLICATE Initiative Working Group, et al. Emulation of Randomized Clinical Trials With Nonrandomized Database Studies: Results of 32 Clinical Trials. JAMA. 2023;330(16):1557-1566. doi:10.1001/jama.2023.19572. For the STaRT-RWE checklist specifically: Wang SV, Pottegard A, Crown W, Norgaard M, Ekstrom CT, Feldman W, et al. HARmonized Protocol Template to Enhance Reproducibility of Hypothesis Evaluating Real-World Evidence Studies on Treatment Effects: A Good Practices Report of a Joint ISPE/ISPOR Task Force. Pharmacoepidemiol Drug Saf. 2021;30(1):44-55. doi:10.1002/pds.5169. PMID: 33210386. For the original STaRT-RWE statement: Wang SV, Pottegard A, Crown W, et al. STaRT-RWE: Structured Template for Planning and Reporting on the Implementation of Real World Evidence Studies. BMJ. 2021;372:n48. doi:10.1136/bmj.n48. PMID: 33472779. + +Apply when planning or reporting a real-world evidence (RWE) study on treatment effects using non-interventional data (electronic health records, claims, registries) in contexts where results may inform regulatory, coverage, or clinical guideline decisions. STaRT-RWE is designed to support transparency, pre-specification, and replicability of RWE analyses. + +--- + +## Regulatory and Decision Context + +### Item 1 -- Regulatory question specification + +**Description:** Specify the regulatory or decision-making question that the RWE study is designed to address. State the intended use of the study results and the relevant regulatory pathway or decision context. + +**What to look for in code/outputs:** +- Explicit statement of regulatory or coverage question (label expansion, post-market requirement, coverage decision) +- Intended use of evidence documented (FDA submission, HTA submission, payer negotiation) +- Decision context described (confirmatory evidence, supplementary evidence, exploratory) +- Alignment between the regulatory question and the study's estimand +- Pre-specification of analysis relative to data access (protocol finalized before data lock) + +### Item 2 -- Estimand specification + +**Description:** Define the target estimand precisely, including the treatment contrast, the target population, the variable of interest (outcome), and the handling of intercurrent events (e.g., treatment discontinuation, competing events). + +**What to look for in code/outputs:** +- Estimand framework components documented (treatment, population, outcome, intercurrent events, summary measure) +- Treatment contrast specified (new user of drug A vs. new user of drug B) +- Handling of intercurrent events specified (treatment-policy, hypothetical, composite, while on treatment) +- Estimand distinguishes between the effect of initiating vs. continuing treatment +- Alignment between estimand and the corresponding RCT estimand being emulated (if applicable) + +--- + +## Data Source + +### Item 3 -- Data source fitness for purpose + +**Description:** Describe the data source(s) used and evaluate their fitness for addressing the regulatory question, including coverage, completeness, accuracy of key variables, and representativeness of the target population. + +**What to look for in code/outputs:** +- Data source description (EHR system, claims database, registry, name, coverage dates, population) +- Validation studies referenced for key variable accuracy (diagnosis codes, drug exposure, outcomes) +- Completeness assessment for key covariates (laboratory values, vital signs, clinical variables) +- Population representativeness assessed relative to the indicated patient population +- Linkage to other data sources documented (mortality registry, cancer registry, dispensing data) + +### Item 4 -- Data source limitations relevant to the regulatory question + +**Description:** Describe specific limitations of the data source that are relevant to the study question, including known misclassification, unmeasured confounders, and potential biases inherent to the data. + +**What to look for in code/outputs:** +- Known misclassification rates for exposure or outcome variables documented +- Unmeasured or incompletely measured confounders identified by name +- Channeling bias potential assessed (reasons for treatment selection likely correlated with outcome) +- Healthy user bias and depletion of susceptibles documented as applicable +- Duration of data availability relative to the treatment exposure and outcome window + +--- + +## Study Design + +### Item 5 -- Study design selection and justification + +**Description:** Specify the study design (cohort, case-control, self-controlled) and justify its appropriateness for the regulatory question and data source. Describe the target trial emulation framework if applicable. + +**What to look for in code/outputs:** +- Study design specified with justification relative to the regulatory question +- Active comparator, new user design implemented in cohort construction code +- Target trial emulation protocol documented (eligibility criteria, treatment assignment, follow-up, outcomes) +- Design choice rationale addressing alternative designs and why they were not used +- Prevalent user exclusion implemented and documented in cohort entry code + +### Item 6 -- Index date and follow-up definition + +**Description:** Define the index date (cohort entry date) for each study participant and the follow-up period, including start, end, and censoring rules. Describe how these definitions align with the regulatory question. + +**What to look for in code/outputs:** +- Index date definition coded and documented (first prescription, first diagnosis, first clinical encounter) +- Washout period applied and duration specified +- Follow-up end date rules documented (treatment discontinuation, switching, death, database end, administrative censoring) +- Grace period or treatment gap tolerance specified for exposure continuation definition +- Censoring rules pre-specified and implemented consistently in code + +### Item 7 -- Eligibility criteria + +**Description:** Define inclusion and exclusion criteria for study participants, specifying how they were operationalized in the data. Justify criteria in relation to the target population of interest. + +**What to look for in code/outputs:** +- Inclusion criteria coded with look-back window specifications +- Exclusion criteria coded with diagnosis, procedure, or dispensing code lists +- Minimum continuous enrollment period prior to index date implemented +- Contraindication and prior use exclusions documented +- Age, sex, and clinical eligibility criteria aligned with the approved indication or trial population + +--- + +## Exposure and Comparator + +### Item 8 -- Exposure and comparator definition + +**Description:** Define the treatment exposure and the comparator, including drug classes, dose ranges, route of administration, and the time window used to classify exposure. + +**What to look for in code/outputs:** +- Drug codes (NDC, RxNorm, ATC) for treatment and comparator fully specified +- Dose range restrictions applied and documented +- Route of administration specified where relevant +- New user definition implemented (absence of exposure in washout period confirmed in code) +- Active comparator rationale documented (why this comparator reduces confounding by indication) + +### Item 9 -- Handling of treatment changes + +**Description:** Define how treatment switching, augmentation, discontinuation, and gap tolerance are handled in the exposure definition and follow-up. + +**What to look for in code/outputs:** +- Treatment discontinuation definition coded (days supply plus grace period) +- Switching and augmentation handling specified (censoring, ITT-like, per-protocol-like) +- Grace period duration specified and clinically justified +- Sensitivity analyses with alternative grace period durations planned +- Intent-to-treat and per-protocol analytic approaches compared if applicable + +--- + +## Outcome + +### Item 10 -- Outcome definition and validation + +**Description:** Define the primary and secondary outcomes, including the code lists and algorithms used for ascertainment. Describe any validation of the outcome definition in the data source. + +**What to look for in code/outputs:** +- Outcome definition code lists (ICD-9, ICD-10, CPT, HCPCS, lab codes) fully specified +- Positive predictive value or sensitivity/specificity of outcome algorithm cited or measured +- Outcome ascertainment window defined relative to index date +- Competing events identified and handling strategy specified +- Secondary and safety outcomes defined with the same specificity as the primary outcome + +--- + +## Confounding + +### Item 11 -- Confounding assessment and control + +**Description:** Describe the approach to identifying, measuring, and controlling for confounding, including the covariate list, measurement windows, and analytic method. + +**What to look for in code/outputs:** +- Full covariate list with measurement windows and data source tables +- Confounding control method specified (propensity score, multivariable regression, high-dimensional propensity score, instrumental variable) +- Propensity score model specification documented (variables included, model type) +- Propensity score trimming, matching, or weighting approach specified +- Covariate balance assessment code (standardized mean differences, Love plot) +- High-dimensional propensity score (hdPS) implementation if used + +### Item 12 -- Unmeasured confounding assessment + +**Description:** Describe methods used to assess the potential impact of unmeasured confounding, including sensitivity analyses, quantitative bias analyses, or negative control analyses. + +**What to look for in code/outputs:** +- E-value calculation or quantitative bias analysis code implemented +- Negative control outcome or exposure analysis coded +- Instrument-based sensitivity analysis (e.g., instrumental variable) if applicable +- Array approach or rule-out analysis for unmeasured confounders +- Comparison with RCT results using the target trial emulation framework + +--- + +## Statistical Analysis + +### Item 13 -- Primary analysis specification + +**Description:** Specify the primary statistical analysis method, the effect measure, and how the analysis accounts for the study design and confounding approach. + +**What to look for in code/outputs:** +- Primary analysis model specified (Cox regression, logistic regression, Poisson, G-computation) +- Effect measure defined (hazard ratio, risk ratio, odds ratio, risk difference, rate difference) +- Confidence interval method specified (robust variance, bootstrap, profile likelihood) +- Covariate adjustment or propensity score incorporation in the outcome model documented +- Absolute risk estimates planned alongside relative measures + +### Item 14 -- Sensitivity analyses for regulatory purposes + +**Description:** Pre-specify sensitivity analyses that address the key assumptions and data limitations of the RWE study, including design choices, exposure definitions, and potential biases. + +**What to look for in code/outputs:** +- Sensitivity analyses listed and pre-specified before data access +- Alternative exposure definitions tested (different grace periods, dose thresholds) +- Alternative comparator groups analyzed +- Alternative confounder adjustment approaches compared (PS matching vs. weighting, hdPS vs. standard PS) +- On-treatment vs. intent-to-treat analytic approaches compared +- Restriction analyses for high-adherence subpopulations + +### Item 15 -- Subgroup and interaction analyses + +**Description:** Pre-specify any subgroup or effect modification analyses, with rationale for the subgroups examined. + +**What to look for in code/outputs:** +- Prespecified subgroup variables listed with clinical rationale +- Interaction terms specified in the outcome model for effect modification assessment +- Sample size adequacy for planned subgroup analyses addressed +- Multiplicity adjustment approach specified for subgroup analyses + +--- + +## Transparency and Reproducibility + +### Item 16 -- Protocol pre-specification and transparency + +**Description:** Confirm that the study protocol was finalized and, ideally, publicly registered before accessing the study data. Describe any deviations from the pre-specified protocol. + +**What to look for in code/outputs:** +- Protocol registration reference (ENCEPP, ClinicalTrials.gov, OSF, or other registry) +- Protocol version number and date relative to data access documented +- Deviation log documenting any post-data-access protocol changes with rationale +- Protocol and SAP availability in supplementary materials or repository + +### Item 17 -- Analytical transparency + +**Description:** Describe the availability of the analysis code, the data, and any analytic tools to support replication or validation of the study results. + +**What to look for in code/outputs:** +- Analysis code availability stated (public repository, supplementary materials, available on request) +- Data availability statement including data use agreement constraints +- Software and package versions documented in code or supplementary materials +- Study protocol and analysis plan linked or appended to the report + +### Item 18 -- Assumptions documentation + +**Description:** Explicitly state the key assumptions underlying the study design, analysis, and interpretation, and describe how each was evaluated or addressed. + +**What to look for in code/outputs:** +- No unmeasured confounding assumption stated and evaluated (negative controls, E-value) +- Positivity assumption assessed (covariate overlap between treatment groups) +- Consistency assumption addressed (treatment version irrelevance discussed) +- Exchangeability assumption evaluated (covariate balance statistics reported) +- Any target trial emulation assumptions explicitly stated and justified diff --git a/plugins/ce-datascience/skills/ce-code-review/references/strega-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/strega-checklist.md new file mode 100644 index 00000000..387c6b00 --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/strega-checklist.md @@ -0,0 +1,112 @@ +# STREGA Checklist for Genetic Association Studies + +STrengthening the REporting of Genetic Association Studies (STREGA). +Extension of STROBE for observational studies reporting on gene-disease associations. + +**Primary reference:** Little J, Higgins JP, Ioannidis JP, Moher D, Gagnon F, von Elm E, Khoury MJ, Cohen B, Davey Smith G, Grimshaw J, Scheet P, Gwinn M, Williamson RE, Zou GY, Hutchings K, Johnson CY, Tait V, Wiens M, Golding J, van Duijn C, McLaughlin J, Paterson A, Wells G, Fortier I, Freedman M, Zecevic M, King R, Infante-Rivard C, Stewart A, Birkett N. STrengthening the REporting of Genetic Association Studies (STREGA)--an extension of the STROBE statement. Genet Epidemiol. 2009;33(7):581-598. doi:10.1002/gepi.20410. PMID: 19278015. + +**Also published in:** Eur J Epidemiol. 2009;24(1):37-55. PMID: 19082745. + +This checklist supplements STROBE. Apply alongside the base STROBE 22-item checklist when reporting genetic association studies (GWAS, candidate gene, or gene-environment interaction). + +--- + +## Methods + +### STREGA 3 -- Objectives (genetic-specific) + +**Description:** State whether the study is a discovery (hypothesis-generating) or replication (hypothesis-testing) study. If replication, identify the original study. + +**What to look for in code/outputs:** +- Study classified as discovery GWAS, candidate gene, replication, or meta-analysis +- If replication: original study cited, same SNPs and outcome definitions used +- Hypothesis stated if candidate gene study + +### STREGA 6 -- Participants (genetic-specific) + +**Description:** Describe the source population, sampling strategy, and any population stratification concerns. Report ancestry/ethnicity of participants. + +**What to look for in code/outputs:** +- Ancestry or ethnicity of study participants reported +- Population stratification assessment (genomic inflation factor, principal components) +- Sample recruitment strategy described (population-based, clinic-based, family-based) +- Related individuals handling documented (kinship pruning, mixed models) + +### STREGA 7 -- Variables (genetic-specific) + +**Description:** Report the genotyping platform, SNP selection criteria, quality control filters, and imputation methods. + +**What to look for in code/outputs:** +- Genotyping platform and array specified (e.g., Illumina Global Screening Array, Affymetrix Axiom) +- SNP selection: candidate gene rationale, or genome-wide with significance threshold +- Quality control filters: call rate (per-sample and per-SNP), HWE threshold, MAF threshold, heterozygosity, sex check +- Imputation: reference panel (1000 Genomes, TOPMed, HRC), software (IMPUTE2, Minimac4, BEAGLE), imputation quality filter (INFO > 0.3 or R2 > 0.3) + +### STREGA 8 -- Data sources (genetic-specific) + +**Description:** Describe the laboratory methods for genotyping, including DNA extraction, genotyping protocol, and quality assurance. + +**What to look for in code/outputs:** +- DNA source (blood, saliva, buccal) documented +- Genotyping batch and plate design documented +- Genotype calling algorithm specified (GenomeStudio, zCall, optiCall) +- Quality control concordance for duplicated samples + +### STREGA 11 -- Quantitative variables (genetic-specific) + +**Description:** Describe how genetic variants were modeled: additive, dominant, recessive, or genotypic model. + +**What to look for in code/outputs:** +- Genetic model specified (additive coding per-allele, dominant, recessive, or all three tested) +- Rationale for model choice if not additive (which is standard for GWAS) +- Haplotype analysis methods if used + +### STREGA 12 -- Statistical methods (genetic-specific) + +**Description:** (a) Describe methods for multiple testing correction. (b) Report methods for population stratification adjustment. (c) Report methods for gene-environment interaction testing. + +**What to look for in code/outputs:** +- Multiple testing correction: Bonferroni, FDR (Benjamini-Hochberg), permutation testing, or genome-wide significance threshold (5e-8) +- Population stratification: principal component adjustment, genomic control, mixed models (BOLT-LMM, SAIGE, REGENIE) +- Gene-environment interaction: interaction terms, stratified analyses, G x E testing framework +- Software documented (PLINK, REGENIE, SAIGE, BOLT-LMM, GCTA) + +--- + +## Results + +### STREGA 13 -- Participants (genetic-specific) + +**Description:** Report the number of participants with genotype data available after quality control, and the number of SNPs analyzed. + +**What to look for in code/outputs:** +- Sample size before and after QC +- Number of genotyped SNPs, number after QC, number after imputation +- Genomic inflation factor (lambda) reported +- QQ plot included + +### STREGA 16 -- Main results (genetic-specific) + +**Description:** Report effect estimates (odds ratios, beta coefficients) with confidence intervals for each genetic variant. Report allele frequencies in cases and controls. + +**What to look for in code/outputs:** +- Per-SNP effect estimates with 95% CI and p-values +- Effect allele and other allele specified for each reported SNP +- Allele frequencies in cases and controls (or exposed and unexposed) +- Manhattan plot and QQ plot included +- Regional association plots (LocusZoom) for significant loci + +--- + +## Discussion + +### STREGA 19 -- Limitations (genetic-specific) + +**Description:** Discuss limitations specific to genetic studies: winner's curse, population stratification residual, cryptic relatedness, and genotyping error. + +**What to look for in code/outputs:** +- Winner's curse (effect size inflation in discovery) acknowledged +- Residual population stratification discussed +- Linkage disequilibrium patterns and fine-mapping limitations +- Generalizability to other ancestries discussed +- Functional annotation of associated variants (or lack thereof) diff --git a/plugins/ce-datascience/skills/ce-code-review/references/strobe-mr-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/strobe-mr-checklist.md new file mode 100644 index 00000000..64176750 --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/strobe-mr-checklist.md @@ -0,0 +1,107 @@ +# STROBE-MR Checklist for Mendelian Randomization Studies + +Strengthening the Reporting of Mendelian Randomization Studies (STROBE-MR). +Extension of STROBE for observational studies using genetic variants as instrumental variables to estimate causal effects. + +**Primary reference:** Skrivankova VW, Richmond RC, Woolf BAR, Yarmolinsky J, Davies NM, Swanson SA, VanderWeele TJ, Timpson NJ, Higgins JPT, Dimou N, Langenberg C, Golub RM, Loder EW, Gallo V, Tybjaerg-Hansen A, Davey Smith G, Egger M, Richards JB. Strengthening the reporting of observational studies in epidemiology using Mendelian randomization: the STROBE-MR Statement. JAMA. 2021;326(16):1614-1621. doi:10.1001/jama.2021.18236. PMID: 34698778. + +This checklist supplements STROBE. Apply alongside the base STROBE 22-item checklist when genetic variants are used as instrumental variables. + +--- + +## Introduction + +### STROBE-MR 2 -- Background (MR-specific) + +**Description:** Explain the rationale for using MR and the assumptions required (relevance, independence, exclusion restriction). + +**What to look for in code/outputs:** +- Rationale for MR over conventional observational analysis +- The three IV assumptions stated and discussed in context of the specific exposure-outcome pair +- Prior MR studies on the same topic referenced + +--- + +## Methods + +### STROBE-MR 4 -- Data sources + +**Description:** Describe the data sources for exposure GWAS, outcome GWAS, and (if applicable) the sample used for instrument selection. State whether one-sample or two-sample MR was used. + +**What to look for in code/outputs:** +- GWAS summary statistics sources named with sample sizes and populations +- One-sample vs two-sample design documented +- If two-sample: non-overlapping samples verified or overlap estimated +- GWAS catalog accession numbers or URLs provided + +### STROBE-MR 5 -- Instrument selection + +**Description:** Describe the criteria for selecting genetic instruments: p-value threshold, LD clumping parameters, genome build, and number of SNPs retained. + +**What to look for in code/outputs:** +- P-value threshold for instrument selection (e.g., 5e-8) +- LD clumping parameters: r2 threshold and distance window (e.g., r2 < 0.001, 10 Mb) +- Genome build specified (GRCh37/GRCh38) +- Number of SNPs before and after clumping +- F-statistic or R2 for instrument strength + +### STROBE-MR 7 -- Statistical methods + +**Description:** Describe the MR estimation method(s) used and the rationale for their selection. + +**What to look for in code/outputs:** +- Primary MR method stated (inverse-variance weighted, Wald ratio, MR-Egger, weighted median, MR-PRESSO, CAUSE) +- Rationale for method choice based on pleiotropy assumptions +- Software and version documented (TwoSampleMR, MendelianRandomization, MR-Base, CAUSE) + +### STROBE-MR 8 -- Sensitivity analyses + +**Description:** Describe MR-specific sensitivity analyses to assess the robustness of causal estimates to violations of IV assumptions. + +**What to look for in code/outputs:** +- Pleiotropy assessment: MR-Egger intercept test, MR-PRESSO global test, Cochran's Q +- Outlier detection: MR-PRESSO outlier test, leave-one-out analysis, radial MR +- Heterogeneity: Cochran's Q statistic, funnel plot +- Directionality: Steiger test for correct causal direction +- Multivariable MR if confounding by LD or horizontal pleiotropy suspected +- Colocalization analysis (coloc, eCAVIAR) to distinguish pleiotropy from shared causal variant + +--- + +## Results + +### STROBE-MR 12 -- Instrument characteristics + +**Description:** Report the characteristics of the genetic instruments including number of SNPs, variance explained, and F-statistic. + +**What to look for in code/outputs:** +- Number of SNPs used as instruments +- Variance explained (R2) in the exposure +- Mean F-statistic (or per-SNP F-statistics) +- Instrument-exposure associations with effect sizes and standard errors + +### STROBE-MR 16 -- MR estimates + +**Description:** Report causal effect estimates from all MR methods used, with confidence intervals and p-values. + +**What to look for in code/outputs:** +- IVW estimate with 95% CI and p-value +- Sensitivity method estimates (MR-Egger, weighted median, MR-PRESSO) with 95% CIs +- Scatter plot of SNP-exposure vs SNP-outcome associations +- Forest plot of individual SNP Wald ratios +- Funnel plot for assessing directional pleiotropy +- Leave-one-out plot + +--- + +## Discussion + +### STROBE-MR 19 -- Assumption assessment + +**Description:** Discuss the plausibility of the three IV assumptions for the specific exposure-outcome pair studied. + +**What to look for in code/outputs:** +- Relevance: F-statistic above 10, biological plausibility of genetic association +- Independence: potential for population stratification, assessment of confounding by ancestry +- Exclusion restriction: pleiotropy assessment results, biological pathways of instruments +- Comparison with conventional observational estimates and/or RCT evidence where available diff --git a/plugins/ce-datascience/skills/ce-code-review/references/target-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/target-checklist.md new file mode 100644 index 00000000..76d36018 --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/target-checklist.md @@ -0,0 +1,103 @@ +# TARGET Checklist for Target Trial Emulation Studies + +TARGET (TArgeted Reporting of observational studies emulating a randomized trial using the TARGET framework). +Reporting checklist for observational studies that explicitly emulate a hypothetical target trial. + +**Primary reference:** Hansford HJ, Cashin AG, Jones MD, Swanson SA, Islam N, Douglas SR, Camacho EM, Furlan AD, Golub RM, Hopewell S, Lee H, Maas ET, Mathieson S, McCord KA, Moher D, Moreno-Betancur M, Rijnhart JJW, Rogers KD, Sandbrink F, Schriger DL, Turk DC, Williamson E, McAuley JH; TARGET Working Group. The TARGET statement: a framework for the reporting of studies that emulate a target trial using observational data. BMJ. 2025;388:e081680. + +This checklist applies to observational studies that frame their analysis as an emulation of a hypothetical randomized trial. Use alongside STROBE and, where applicable, RECORD. + +--- + +## Protocol Components + +### TARGET 1 -- Eligibility criteria + +**Description:** Specify the eligibility criteria for the target trial and describe how they were emulated using the observational data. + +**What to look for in code/outputs:** +- Target trial eligibility stated explicitly (who would be enrolled in the hypothetical trial) +- Emulation eligibility criteria in code: how the observational cohort was filtered to approximate the trial population +- Differences between target trial and emulated eligibility documented + +### TARGET 2 -- Treatment strategies + +**Description:** Specify the treatment strategies being compared in the target trial and describe how they were defined in the observational data. + +**What to look for in code/outputs:** +- Treatment strategies fully specified (not just "treated vs untreated" but exact regimens, doses, durations) +- Operationalization in data: code lists, prescription rules, or algorithm defining each strategy +- Handling of treatment switching, discontinuation, and non-adherence documented + +### TARGET 3 -- Treatment assignment + +**Description:** Describe the assumed treatment assignment mechanism in the target trial and the method used to adjust for confounding in the emulation. + +**What to look for in code/outputs:** +- Confounding adjustment method documented (IPTW, matching, g-estimation, standardization) +- Covariates used for adjustment listed with rationale +- Propensity score model specification if applicable +- Positivity assessment: overlap in propensity scores between groups + +### TARGET 4 -- Outcome + +**Description:** Define the primary and secondary outcomes and how they were ascertained in both the target trial and its emulation. + +**What to look for in code/outputs:** +- Outcome definition matching the target trial outcome (same ascertainment window, same severity threshold) +- ICD/procedure codes or clinical criteria used for outcome identification +- Outcome ascertainment period and censoring rules + +### TARGET 5 -- Time zero (baseline) + +**Description:** Define time zero (the point of randomization in the target trial) and describe how it was emulated. Time zero must align treatment assignment, eligibility, and the start of follow-up. + +**What to look for in code/outputs:** +- Time zero defined explicitly in documentation and code +- Alignment verified: eligibility assessed at time zero, treatment assigned at time zero, follow-up starts at time zero +- No immortal time between eligibility and treatment assignment +- Grace period documented if used (and justified) + +### TARGET 6 -- Follow-up + +**Description:** Specify the follow-up period, censoring rules, and end-of-follow-up definition. + +**What to look for in code/outputs:** +- Follow-up start (time zero) and end (outcome, censoring, administrative end) documented +- Censoring rules: what events trigger censoring, handling of competing risks +- Per-protocol vs intention-to-treat emulation documented +- Artificial censoring for per-protocol emulation explained + +### TARGET 7 -- Statistical analysis + +**Description:** Describe the causal estimand and the statistical methods used to estimate it. + +**What to look for in code/outputs:** +- Estimand stated explicitly (ATE, ATT, per-protocol effect, ITT effect) +- Statistical method consistent with the estimand (IPTW for ATE, matching for ATT) +- Variance estimation method (bootstrap, robust SE, sandwich estimator) +- Sensitivity analyses for unmeasured confounding (E-value, quantitative bias analysis) +- If cloning-censoring-weighting used, all three steps documented + +--- + +## Reporting + +### TARGET 8 -- Table mapping target trial to emulation + +**Description:** Include a structured table that maps each protocol component (eligibility, treatment, assignment, outcome, time zero, follow-up, analysis) from the target trial to its observational emulation. + +**What to look for in code/outputs:** +- Two-column table: "Target Trial Protocol" vs "Emulation Using Observational Data" +- All 7 protocol components addressed +- Differences and compromises documented in the table + +### TARGET 9 -- Assumptions and limitations + +**Description:** Discuss the assumptions required for the emulation to be valid, and the limitations of the observational data relative to the hypothetical trial. + +**What to look for in code/outputs:** +- Exchangeability (no unmeasured confounding) assumption discussed +- Positivity assumption assessed (treatment available to all eligible) +- Consistency assumption discussed (well-defined interventions) +- Data-specific limitations: measurement error, misclassification, missing data diff --git a/plugins/ce-datascience/skills/ce-code-review/references/tripod-checklist.md b/plugins/ce-datascience/skills/ce-code-review/references/tripod-checklist.md new file mode 100644 index 00000000..aff5330d --- /dev/null +++ b/plugins/ce-datascience/skills/ce-code-review/references/tripod-checklist.md @@ -0,0 +1,290 @@ +# TRIPOD Checklist for Prediction Model Studies + +Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD). +22-item checklist for studies developing, validating, or updating multivariable prediction models using regression methods. + +**Primary reference:** Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, Vickers AJ, Ransohoff DF, Collins GS. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1-73. doi:10.7326/M14-0698. PMID: 25560730. + +**Companion statement:** Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015;162(1):55-63. doi:10.7326/M14-0697. PMID: 25560714. + +This checklist applies to regression-based prediction models. For studies using machine learning or AI methods, apply the TRIPOD+AI extension instead. Items marked D apply to model development, V to validation, and D+V to both. + +--- + +## Title and Abstract + +### Item 1 -- Title + +**Description:** Identify the study as developing and/or validating a multivariable prediction model, the target population, and the outcome to be predicted. (D+V) + +**What to look for in code/outputs:** +- Report title or header that names the prediction target (e.g., 30-day mortality, 5-year cardiovascular event) +- Title that identifies whether the study is developing, validating, or updating a model +- Keywords in document metadata that include the model type and clinical domain +- Structured abstract or summary section consistent with the stated title + +### Item 2 -- Abstract + +**Description:** Provide a summary of objectives, study design, setting, participants, sample size, predictors, outcome, statistical analysis, results, and conclusions. (D+V) + +**What to look for in code/outputs:** +- Abstract sections reporting model development or validation objectives +- Sample sizes for development and validation datasets stated in abstract +- Key performance metrics (AUC/C-statistic, calibration measure) reported in abstract +- Event rates or outcome frequencies summarized in abstract + +--- + +## Introduction + +### Item 3 -- Background and objectives + +**Description:** Explain the medical context (including whether diagnostic or prognostic) and rationale for developing or validating the multivariable prediction model, including references to existing models. State the objectives, including whether the study describes the development or validation of the model or both. (D+V) + +**What to look for in code/outputs:** +- Documentation of the clinical question (diagnostic vs. prognostic prediction) +- Literature review referencing existing models for the same clinical question +- Stated rationale for developing a new model or validating an existing one +- Primary objectives defined in analysis header, SAP, or documentation + +--- + +## Methods -- Source of Data + +### Item 4 -- Source of data + +**Description:** Describe the study design or data source (such as a randomized trial, a cohort, or registry data), separately for the development and validation data sets, if applicable. (D+V) + +**What to look for in code/outputs:** +- Data source documentation (clinical trial, prospective cohort, registry, EHR extract, claims) +- Study design classification (prospective vs. retrospective data collection) +- Separate description of development and validation data sources when both are used +- Date range filters in data extraction code (enrollment period, follow-up cutoff) + +### Item 5 -- Participants + +**Description:** Specify the key elements of the study setting (e.g., primary, secondary, tertiary care, community) including number and location of centers. Describe the eligibility criteria for participants. (D+V) + +**What to look for in code/outputs:** +- Inclusion/exclusion criteria implemented in cohort construction code +- Care setting and institutional context documented +- Number of centers or sites with contribution counts +- Participant flow from source population through final analytic sample + +--- + +## Methods -- Outcome and Predictors + +### Item 6 -- Outcome + +**Description:** Clearly define the outcome that is predicted by the prediction model, including how and when the outcome was measured. (D+V) + +**What to look for in code/outputs:** +- Outcome variable derivation code with ICD/CPT/LOINC code definitions +- Prediction horizon specification (e.g., in-hospital, 30-day, 1-year) +- Outcome ascertainment window defined relative to index date in code +- Binary, time-to-event, or ordinal outcome type specified with appropriate model family + +### Item 7 -- Predictors + +**Description:** Clearly define all predictors used in developing or validating the multivariable prediction model, including how and when they were measured. (D+V) + +**What to look for in code/outputs:** +- Complete predictor list with variable derivation code and source tables +- Temporal relationship documented (all predictors available before the prediction time) +- Measurement units, reference ranges, and coding schema specified +- Look-back window definitions for time-varying predictors + +--- + +## Methods -- Sample Size + +### Item 8 -- Sample size + +**Description:** Explain how the study size was arrived at. (D+V) + +**What to look for in code/outputs:** +- Sample size justification based on events-per-variable (EPV) rule or simulation +- Documentation of minimum EPV threshold applied (e.g., 10-20 events per predictor) +- Power calculations for validation studies +- Comments when sample size is limited by available data + +--- + +## Methods -- Missing Data + +### Item 9 -- Missing data + +**Description:** Describe how missing data were handled (e.g., complete-case analysis, single imputation, multiple imputation) with details of any imputation method. (D+V) + +**What to look for in code/outputs:** +- Missing data summary table showing count and percentage missing per variable +- Imputation method implemented in code (multiple imputation, single imputation, indicator) +- Imputation model specified with predictors of missingness +- Comparison of complete cases vs. full sample to assess potential bias + +--- + +## Methods -- Statistical Analysis + +### Item 10a -- Statistical analysis methods (development) + +**Description:** Describe how predictors were handled in the analyses. (D) + +**What to look for in code/outputs:** +- Continuous predictor handling (linear assumption testing, splines, categorization) +- Cut-point selection rationale if continuous variables were categorized +- Interaction term specification and clinical rationale +- Non-linear transformation code (fractional polynomials, restricted cubic splines) + +### Item 10b -- Statistical analysis methods (model type) + +**Description:** Specify the type of model, all model-building procedures (including any predictor selection), and method for internal validation. (D) + +**What to look for in code/outputs:** +- Model family and link function specified (logistic, Cox, competing risks, multinomial) +- Predictor selection method documented (full model, backward elimination, LASSO, clinical prespecification) +- Internal validation method coded (bootstrap, cross-validation, apparent vs. optimism-corrected) +- Optimism correction applied if predictor selection was performed + +### Item 10c -- Statistical analysis methods (validation) + +**Description:** For validation, describe how the predictions were calculated. (V) + +**What to look for in code/outputs:** +- Prediction calculation code applying the original model formula or intercept/slope recalibration +- Coefficient values or model object from development dataset applied to validation data +- Recalibration method specified if applied (intercept update, slope recalibration) +- No re-estimation of predictor effects in the validation dataset + +### Item 10d -- Statistical analysis methods (performance measures) + +**Description:** Specify all measures used to assess model performance and, if relevant, to compare multiple models. (D+V) + +**What to look for in code/outputs:** +- Discrimination measures coded (C-statistic/AUC with confidence intervals) +- Calibration measures coded (calibration-in-the-large, calibration slope, calibration plot) +- Overall performance measures (Brier score, R-squared equivalents) +- Decision curve analysis or net benefit calculations if clinical utility is assessed + +### Item 10e -- Statistical analysis methods (risk groups) + +**Description:** Describe any model updating (e.g., recalibration) arising from the validation, if done. (V) + +**What to look for in code/outputs:** +- Risk group definitions applied (low/medium/high) with cut-points specified +- Clinical or statistical basis for cut-point selection documented +- Distribution of participants across risk groups reported +- Sensitivity and specificity at each threshold calculated + +--- + +## Results + +### Item 11 -- Participants + +**Description:** Describe the flow of participants through the study, including the number of participants with and without the outcome and, if applicable, a summary of the follow-up time. A diagram may be helpful. (D+V) + +**What to look for in code/outputs:** +- Participant flow from eligible population to final analytic sample with counts +- Event count and event rate in development and validation cohorts +- Follow-up time summary (median, IQR, person-years) for time-to-event outcomes +- Flowchart or attrition table generated in code + +### Item 12 -- Model development + +**Description:** Describe the characteristics (e.g., demographic, clinical, predictor values) of the participants in the development and any validation data sets. (D+V) + +**What to look for in code/outputs:** +- Table 1 generation code with descriptive statistics for predictor and outcome variables +- Stratification by development vs. validation cohort where applicable +- Missing data counts per variable in the descriptive table +- Continuous variables summarized as mean/SD or median/IQR as appropriate + +### Item 13 -- Model specification + +**Description:** Present the full prediction model to allow predictions for individuals (i.e., all regression coefficients, and model intercept or baseline survival at a given time point). (D) + +**What to look for in code/outputs:** +- Final model coefficients or hazard ratios reported with confidence intervals +- Model intercept or baseline hazard/survival function reported +- Predictor reference categories and coding scheme documented +- Sufficient information to reproduce predictions without the original dataset + +### Item 14 -- Model performance + +**Description:** Report performance measures (with confidence intervals) for the prediction model. (D+V) + +**What to look for in code/outputs:** +- Discrimination statistics (C-statistic/AUC) with bootstrap or cross-validated confidence intervals +- Calibration plot (observed vs. predicted probabilities) generated as figure +- Calibration metrics (calibration slope, calibration-in-the-large) reported numerically +- Apparent and optimism-corrected performance reported separately for development + +### Item 15 -- Model updating + +**Description:** Report the results from any model updating (i.e., model specification, model performance after updating). (V) + +**What to look for in code/outputs:** +- Updated intercept or recalibration slope applied and reported +- Performance before and after updating compared in results tables +- Reason for updating documented (poor calibration, different outcome rate) +- Updated model coefficients or recalibrated predictions saved and reported + +--- + +## Discussion + +### Item 16 -- Limitations + +**Description:** Discuss any limitations of the study (such as non-representative sample, few events per predictor, missing data). (D+V) + +**What to look for in code/outputs:** +- Limitation section referencing sample representativeness and generalizability +- EPV ratio or events-per-predictor calculation reported +- Missing data mechanism and potential impact discussed +- Overfitting risk discussed relative to internal validation method used + +### Item 17 -- Interpretation + +**Description:** Give an overall interpretation of the results, considering objectives, limitations, results from similar studies, and other relevant evidence. (D+V) + +**What to look for in code/outputs:** +- Conclusion section contextualizing model performance relative to existing models +- Clinical relevance of discrimination and calibration metrics discussed +- Comparison with other published models for the same prediction task +- Risk group cut-points justified in clinical terms + +### Item 18 -- Implications + +**Description:** Discuss the potential clinical use of the model and implications for future research. (D+V) + +**What to look for in code/outputs:** +- Intended clinical application described (triage, treatment selection, shared decision-making) +- Recommended next steps articulated (external validation, prospective evaluation, impact study) +- Net benefit or decision curve analysis interpretation in clinical context +- Generalizability of findings to target deployment population discussed + +--- + +## Other Information + +### Item 19 -- Supplementary information + +**Description:** Provide information about the availability of supplementary resources, such as study protocol, Web calculator, and datasets. (D+V) + +**What to look for in code/outputs:** +- Data availability statement in report +- Web calculator or online implementation referenced or provided +- Analysis code availability (GitHub, supplementary materials, OSF) +- Protocol or pre-registration reference + +### Item 20 -- Funding + +**Description:** Give the source of funding and the role of the funders for the present study. (D+V) + +**What to look for in code/outputs:** +- Funding acknowledgment in report template +- Grant numbers or sponsor documentation +- Conflict of interest disclosures +- Role of funder in study design, analysis, or reporting stated