Repository navigation
fix(skills): correct verify, genome-build and pubmed guidance; guard sprint signoff and review scope - #82
Merged
Conversation
…reproducibility checks - Missing-data gate no longer passes/fails on percent missing; checks mechanism, SAP handling, outcome in the imputation model, and m >= % incomplete cases (Madley-Dowd 2019 PMID 30878639; White, Royston & Wood 2011 PMID 21225900) - Reproducibility: data paths must be project-relative or config-driven, not absolute; NumPy default_rng and parallel R RNG seeds - Leakage: feature selection/SMOTE before split, test-set tuning, duplicate encounters, post-index predictors (Kapoor & Narayanan 2023 PMID 37720327) - Effect direction: check reference level and event coding before warning - Figure check follows the ce-figure style profile, not JAMA by default
- Example pairs GENCODE 44 with its own chr-named FASTA (Ensembl 110 equivalent), counts 62,700 genes / 252,835 transcripts (GENCODE stats) - hg19 chrM 16,571 vs b37 MT 16,569 (UCSC chrom.sizes; Ensembl GRCh37 REST) - Build from filename assembly token, not Ensembl release number; vNNlift37 - GDC GENCODE v22 through DR31, v36 from DR32 (GDC GENCODE update FAQ) - Load detection rules on demand instead of @-inlining
- Human limit uses the Cochrane double negative NOT (animals[mh] NOT humans[mh]) instead of humans[MeSH], which dropped in-process records (Cochrane Handbook ch. 4 Box 4.4.a); same query returned 1,073 vs 776 hits - English[lang] is opt-in via --english-only (Cochrane Handbook 4.4) - Add rct-sensitive (Cochrane HSSS) and label prediction/diagnostic filters as high-precision discovery aids (Geersing 2012, PMID 22393453) - Biopython via uv run or the project environment, not a shared interpreter - Drop journal-prestige ranking and rationale prose
- 'significant' needs a prespecified test with estimate and CI, 'substantial' an effect size, 'robust' named sensitivity analyses (ASA statement 2016, principles 3 and 5); a contradicting nearby statistic no longer passes - Example P value follows the target journal's convention instead of P<0.0001
Move the generic PubMed/Paperclip bullets for ce-grill, ce-literature-search, ce-manuscript-citations, and ce-pubmed into each skill's own references/evidence-access.md, keeping only the skill-specific lookup in SKILL.md. Use the real default-server prefix (mcp__pubmed-mcp-server__pubmed_search_articles) and hand off to the ce-pubmed skill instead of reaching into its bundled script.
…guard Best CV/validation score after many iterations is optimistic (Cawley & Talbot, JMLR 2010); report the held-out test score, use nested CV when the test set is small. Leakage guard adds feature selection/SMOTE, test-set thresholds, patient-level grouping, and prediction-time availability (Kapoor & Narayanan 2023, PMID 37720327). Label the worktree cap and hypothesis count as defaults.
sprint.py signoff NAME --by REVIEWER replaces the hand-edited YAML step; it refuses unless the audit passed and the signer is the named reviewer. The example now scopes analyses to SAP-5.x and uses an integer sap_version, as ce-plan defines them.
Each scope snippet that writes PR metadata or path lists under $REVIEW_SCOPE_DIR now starts with a :? guard, so a fresh shell call stops instead of writing to /paths or /pr-preflight.json. The PHI-safe preflight is a numbered checklist (classify, capture privately, risk check, authorization marker). Adds a structure test for the guard.
- ce-plan: confirm the mode when SAP keywords appear in a code-change request - ce-update: keep the two operative rules, drop design-history rationale - ce-worktree: drop the 'most harnesses now' time-sensitive claim
|
Warning Review limit reachedThis review is too large to run within your organization's remaining usage spending cap. Raise or remove your spending cap in the billing tab, then retry. Next included review available in 58 minutes. View limit details
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 16 of 17 (statistical-correctness audit). Base:
stats-stack/14-descriptions. Review and merge bottom-up, starting with #66.36 files changed, 320 insertions(+), 123 deletions(-)Commits
6024fee8fix(verify): judge missing data by mechanism and tighten leakage and reproducibility checks824a6d9afix(genome-build): correct GENCODE 44 example and build-detection rulesec2013bafix(pubmed): keep unindexed and non-English records by default4082c3c5fix(clinical-research-voice): reserve significant for prespecified tests2ba8a483fix(skills): move evidence-access boilerplate into per-skill references8cee04a7fix(optimize): flag selection-biased CV scores and widen the leakage guardbce702e2fix(sprint): add a guarded signoff subcommand and fix SAP example refs4f9420b9fix(code-review): fail closed when the private scope dir is unsetfa0f6223fix(skills): trim rationale prose and time-sensitive wording767eebf1fix(effect-size): name REML/Hartung-Knapp pooling and pool trigger in descriptione1161ec5fix(skills): keep edited SKILL.md files under the 16 KB budgetChanges and sources
Each source was read at the primary text before citing (NCBI E-utilities, Paperclip full text, Browserbase fetch, regulatory PDFs).
Claimrows require all five ICH E9(R1) attributes, with a strategy for each intercurrent event. New guardrail: hypothetical and principal-stratum strategies are flagged as untestable and need a sensitivity analysis. Sources: ICH E9(R1) Step 4, A.3.2-A.3.3 (adopted 20 Nov 2019); Kahan et al., BMJ 2024;384:e076316, PMID 38262663.references/estimands.mdcovers the attributes, the strategies, why analysis sets are not estimands, sensitivity versus supplementary analyses (E9(R1) A.5.2-A.5.3; Morris, Kahan & White, BMC Med Res Methodol 2014, PMID 24456267), and missing data defined relative to the estimand (A.5.2.2). It also covers conditional versus marginal summaries, the target-trial protocol, and a worked Claim/Analysis/Decision_Evidence example. Conditional/marginal sources: FDA Adjusting for Covariates in RCTs, May 2023 (prespecify before unblinding, include strata variables, no post-randomization covariates, conditional vs unconditional, bootstrap SEs); CONSORT 2025 item 25.references/evidence-access.md, identical to PR fix(stats): correct power, effect-size and trial-design methods #66 and using fully qualified MCP names. README row updated.pmsampsize), CI width for precision studies, and no post-hoc power (Hoenig & Heisey, Am Stat 2001;55:19-24, doi:10.1198/000313001300339897).early-clinical-ai-evaluationandprotocol_stageto the frontmatter.sap-mode-workflow.mdandSKILL.mdnow treat an unresolved estimand as a blocker.ce-plan-rct-estimand-sap.bun run eval:validate: all cases valid, including the newce-plan-rct-estimand-sap.Implements the approved "fix all" audit for the reporting/omics scope (
reporting-omics.md), plus three items fromrecent-changes.md: phenotype verification bias, GEE MCAR vs mixed-model MAR, and the ecology-heavy mixed-model reading list. Branch:fix/stats-reporting-omics. No version bumps or CHANGELOG entries.mcp__pubmed-mcp-server__…andmcp__paperclip__…names, and loads thece-pubmedskill instead of reaching into its script.Branch:
fix/stats-skill-craft. Implements the skill-craft audit (skill-craft.md, except A1, A2, A10, A11, A17), the Skill-craft description findings from all six reports, and two items fromrecent-changes.md(PHI scope guard, flaky test timeouts). No version bumps or CHANGELOG release entries; the one CHANGELOG edit is the requested 3.3.0 header link./ce-datascience:ce-planand/plugin, becauseskillOverridesdoes not apply to plugin skills (docs: "Plugin skills are not affected byskillOverrides").default_rng(NumPy docs recommend Generator over legacy RandomState) and parallel R RNG.chrnaming against a GENCODE GTF./info/assembly/homo_sapiens/MTlength 16569).Homo_sapiens.GRCh37.87.gtf.gzreturns HTTP 200 on the Ensembl GRCh37 archive);vNNlift37is GRCh37; "v20+ without lift37" replaces the time-sensitive v44-46 list.NOT (animals[mh] NOT humans[mh]), verified against Cochrane Handbook ch. 4, Box 4.4.a.humans[MeSH]filter dropped unindexed records. Live check, "sepsis bundle" over the last 10 years: 1,073 vs 776 hits.English[lang]is now opt-in (--english-only), per Cochrane Handbook §4.4: "searches should not be restricted by language".--study-type rct-sensitive(Cochrane HSSS, Box 4.4.a feat: ce-datascience 40-skill release with workflow navigator and competitive ports #1-fix(integrations): harden installs and MCP project roots #8).uv run --with biopythonor the project environment.P<.001and defers to the target journal's convention. I did not attribute this rule to JAMA: jamanetwork.com blocked curl, tavily, Browserbase, and the PDF, and browser-harness needs remote debugging enabled.execution.max_worktreeskey in the spec schema, so none was invented.sprint.py signoff NAME --by REVIEWER. It refuses unless the sprint isclosed_pending_signoffand the signer is the named reviewer, which replaces the hand-edited YAML step. A test covers it.sap_versionis now the integer 2.$REVIEW_SCOPE_DIRstarts with: "${REVIEW_SCOPE_DIR:?...}". A fresh shell call now stops instead of writing PR metadata to/pathsor/pr-preflight.json. A structure test enforces this.references/evidence-access.md. SKILL.md keeps a pointer plus "Look up here".mcp-routing.md):mcp__pubmed-mcp-server__pubmed_search_articles(confirmed against this session's connected server);ce-pubmedskill" replaces reaching into its script.Rejected or changed after checking
Validation
Run at this exact stack position:
bun test --timeout 60000,bun run release:validate,bun run eval:validateall pass (see stack summary on #66). Live skill-creator behavioral runs were not performed.