Repository navigation
feat(method-citations): gate bibliographies for fabricated references - #138
Merged
Merged
Conversation
Add scripts/verify_references.py, a deterministic reference-integrity audit for existing bibliographies. It parses BibTeX, RIS, delimited, and numbered free-text reference lists, resolves every DOI via CrossRef then OpenAlex and every PMID via PubMed, then classifies each entry: - OK: identifier resolves and claimed title/year/author match - MISMATCH: identifier resolves but a claimed field differs - UNVERIFIED: no identifier or no record fetched - FABRICATED: DOI/PMID registered nowhere; the audit exits 2 Free-text entries carry guessed titles and years, so only their identifier is compared and the resolved record is attached for a human to check (the fields_trustworthy flag marks the distinction). --lookup-cache records and reuses lookups across runs; --offline classifies from a populated cache with no network, which is also how the tests run. Regression coverage pins the parsing bugs found while smoke-testing: DOIs containing parentheses, bracketed numbering splits, and zero title overlap reported as a MISMATCH instead of skipped as a falsy similarity. SKILL.md documents the audit as step 4 of the evidence workflow with the anchored script invocation and a python-fallback row in failure handling. The method-citations eval set gains the integrity-gate expectation, and the plugin README row and one trigger query cover the new capability.
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configuration
📒 Files selected for processing: 6
✨ Finishing Touches
Comment |
Merged
sajor2000
added a commit
that referenced
this pull request
Oct 21, 2026
Close the journal-selection half of audit gap #5 with a new skill that matches a finished manuscript or abstract to candidate journals: - builds a 5-8 candidate pool from each journal's own aims-and-scope page read via web tools, never from memory or a third-party summary; - gathers scope-fit evidence by searching PubMed restricted to each candidate journal for similar recent studies (count and recency); - runs a legitimacy screen verifying each candidate's indexing claims against the NLM Catalog or DOAJ, flagging unverifiable claims, copycat names, and APC-only venues instead of silently dropping or accepting them; - applies the author's must-haves (indexing, access policy and fees, review speed) as hard filters, ranks with every cell sourced or marked unknown, and recommends a target plus two backups, leaving the choice to the author. Sourcing rules are the core discipline: no impact factor, acceptance rate, turnaround, or APC from memory; every metric carries a source URL and date or is marked unknown. Egress mirrors the other research skills: searches carry study topics, designs, and journal names only. The listing budget rises from 11,500 to 11,700 for the 68th skill (11,686 total, the same ~172-char average per skill), and every displayed count surface (READMEs, website, brand SVG) moves 67 -> 68. Eval sets follow the skill-creator contract (3 evals, 10/10 triggers). The audit doc records gap closures: #2 (PR #138), #4 (PR #140), and the journal-selection half of #5, with the ICMJE COI half left open as a decision. Description fingerprints regenerated from history.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes the reference-integrity gap from the MedSci Skills competitive audit (gap #2,
docs/audits/2026-10-19-medsci-skills-competitive-audit.md): the plugin could build citations but could not catch fabricated ones.method-citationsnow shipsscripts/verify_references.py, a deterministic audit that parses an existing bibliography (BibTeX, RIS, delimited, or a plain numbered list), resolves every DOI via CrossRef then OpenAlex and every PMID via PubMed, and classifies each entry:OKMISMATCHUNVERIFIEDFABRICATEDDesign decisions
fields_trustworthymarks the distinction, and the resolved record is attached for a human to compare.--lookup-cacherecords and reuses lookups;--offlineclassifies from a populated cache, so the test suite never reaches the network.classifyalso accepts--entries/--lookupsfiles directly.0.0similarity was skipped as a falsy value; the gate is now on both sides having tokens.Regression coverage
tests/verify-references.test.tspins the bugs found while smoke-testing: DOIs containing parentheses (legitimate in registered DOIs, e.g.10.1016/S0140-6736(19)30001-1), bracketed[1]/[2]numbering splits, and the zero-overlap title report. Verified live before committing: a real DOI resolves OK, a fabricated one is caught FABRICATED with exit 2.Repo surfaces
plugins/methodsmith/skills/method-citations/— the script; SKILL.md documents the audit as step 4 of the evidence workflow with the anchoredSKILL_DIRinvocation and a python-probe fallback row in failure handling.evals/methodsmith/skill-creator/method-citations/— eval 3 gains the integrity-gate expectation; one should-trigger query now covers fabricated-reference detection (eval and trigger counts stay at 3 and 10/10 per the skill-creator contract).plugins/methodsmith/README.md— the/method-citationsrow mentions the bibliography gate.Validation
bun test— 1729 tests, 0 fail (includes frontmatter, portability, displayed-counts, skill-creator-evals, description-fingerprints contracts)bunx tsc --noEmit— cleanbun run release:validate— in sync, 55 agents / 67 skillsverify_references.pyto parse and audit BibTeX, RIS, delimited, and numbered-list bibliographies.OK,MISMATCH,UNVERIFIED, orFABRICATED.--strict.