Select reversibility heuristics per thermodynamic data source (eQuilibrator gets its own cutoff) - #285
Merged
Merged
Conversation
The direction cascade had exactly one rule set, DEFAULT_HEURISTICS, derived
from the Group-Contribution era and applied to every source. Split it into a
registry keyed by source, with GC as the default for anything without a set of
its own.
GC unchanged Jankowski 2008 cascade -- the default
EQ eQuilibrator: Noor 2012 reversibility index, gated by Beber 2022
uncertainty (new)
EQ2 eQuilibrator 2.0: the same index as a bare point estimate
Three defects in the eQuilibrator path motivated this:
1. `Estimate_Reaction_Reversibility.py EQ` never read eQuilibrator energies.
It pulled the canonical `deltag`, merely gated on eQuilibrator eligibility.
Since the additive-thermodynamics refactor nothing overwrites `deltag`, so
only 1,797 of 25,028 reactions with an eQuilibrator record actually had
`deltag == thermodynamics['eQuilibrator'][0]`; the other 23,140 were scored
on the Group-Contribution number and labelled eQuilibrator. EQ runs now read
the eQuilibrator sublist's own dG and sigma.
2. eQuilibrator's ~1e5 kJ/mol "cannot decompose this reaction" marker was being
consumed as an error bar. 4,933 records carry it; the GC bounds rule cannot
fire that wide, so they fell through to a permissive "=". They now return
"?". Observed real sigma tops out at 65.35 kcal/mol against a marker of
23,900.57, so the cut at 1e4 kJ/mol sits in an empty gap.
3. `Retrieve_eQuilibrator_Reactions_Energies.py` keys its MetaNetX formula on
compound id and so discards compartment, collapsing any species present on
both sides; 1,102 transport reactions carry a dG for a different reaction.
Beber 2022 separately notes the transformed framework needs a
-N_H*RT*ln(10^dpH) - Q*F*dPhi term across a membrane that we never apply.
Transport is now decided structurally (ATPS/ABCT) or returns "?".
The reversibility index eQuilibrator has been computing for us since the table
was first generated -- column 4 of MetaNetX_Reaction_Energies.tbl -- was parsed
and discarded on every run, and `make_ln_reversibility_index_heuristic` was
never invoked. It now decides 8,944 reactions, and the bare no-evidence
`default` fallback drops from 7,120 to zero.
Add_Reaction_Thermodynamics_Operators, _thermo_helpers and Promote_* now pass
the source label through, so each method's stored operator is computed with its
own rule set instead of GC's.
Scripts/Tests/test_eq_heuristics.py verifies the ln(Gamma) implementation
reproduces eQuilibrator's own published ln_reversibility_index on 17,771
reactions; every residual is a MetaNetX-collapsed reaction.
test_reaction_direction.py now distinguishes invariant sources (GC,
dGPredictor -- must match exactly) from intentionally re-scored ones
(eQuilibrator), with --strict to require equality everywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Output of Scripts/Thermodynamics/Rerun_Thermodynamics.sh with the per-source
rule sets. Verified idempotent: a second run reproduces every file byte-for-byte
("Entries refreshed/added: 0").
Per-method operators, vs the previous dev state:
Group contribution 56,002 records 0 changed
eQuilibrator 25,028 records 7,103 changed
dGPredictor 27,715 records 0 changed
dGPredictor-ModelSEED 31,924 records 0 changed
Only eQuilibrator moves. The three columns scored with the untouched GC rule
set are byte-identical, which is what isolates the change to the new cutoff
rather than the refactor.
Canonical reversibility: > 12,670 -> 9,584 (-3,086)
< 2,323 -> 1,603 (-720)
= 15,059 -> 14,002 (-1,057)
? 25,960 -> 30,823 (+4,863)
The database gets less confident, and that is the point: the +4,863 unknowns
track the 4,933 undecomposable records almost exactly. Those had been receiving
a permissive "=" or an inherited direction on the strength of a dG carrying
+/-23,900 kcal/mol -- an unread sentinel, not evidence.
Deciding rule for the EQ run:
Incomplete (no eQuilibrator energy) 24,828
EQ:lnGamma 8,944
EQ:reversible 8,905
Incomplete (GCC, keeps GC answer) 6,146
EQ:undecomposable 4,933
ABCT 1,151
EQ:ambiguous 871
EQ:transport-uncorrected 209
ATPS 15
Empty 10
Canonical deltag is untouched and mostly remains the Group-Contribution value;
reconciling it is Promote_Reaction_Thermodynamics_to_Canonical.py, which is
deliberately not part of Rerun_Thermodynamics.sh and was not run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Write-up of the work behind PR ModelSEED#285, following the convention of the other Papers/NAR_Update_2026/data/ investigation notes. Records what dev was actually doing and how that was established (the eQuilibrator reversibility index was computed, stored in column 4 of MetaNetX_Reaction_Energies.tbl, and discarded unread on every run), the three defects in the eQuilibrator path, and the results of the rerun. Section 3 is the part worth keeping for a methods section: eQuilibrator ships no directionality classifier, Beber 2022 defines no reversibility index and defers directionality to downstream tools, and the table there separates the parts of EQ_HEURISTICS that are faithful reimplementation (the ln Gamma formula, verified against 17,771 published values) from the parts that are ours (the +/-1 sigma margin, the undecomposable and transport gates). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both corrections came from supervisor review of PR ModelSEED#285. The code is unchanged; the characterisation of why it was needed was wrong. 1. "The EQ run never read eQuilibrator energies" was wrong. The design was correct: from c263e23 (2019) through 17c9739 (2023-09-11), Update_Reaction_eQuilibrator_Energies.py overwrote canonical deltag/deltagerr with the eQuilibrator values and stamped the EQU note, so `Estimate_Reaction_Reversibility.py EQ` reading deltag gated on EQU was reading eQuilibrator energies exactly as intended. 3e50646 (2023-09-13) removed the write half -- moving the updater to thermodynamics['eQuilibrator'] only, dropping both the deltag overwrite and the EQU stamp -- and left the read half in place. Later GC rebuilds and the PR ModelSEED#265 promotion then rewrote deltag from other sources. Only 1,586 of the 17,094 reactions still carrying EQU have deltag == the eQuilibrator value. EQU is a fossil of the pre-2023 pipeline, and is distinct from EQC (which the retrieval script writes to mean "all reagents had structures"). 2. "The marker was read as an error bar" understated it. For the residual case GibbsEnergyPredictor.standard_dg returns Q_(0, "kJ/mol") outright -- the computed mean is discarded, not projected. standard_dg_prime then adds the Legendre/pH transform, so the nonzero dG'o stored for those reactions is the transform alone. Of 4,607 sigma-flagged reactions, 2,895 are this zeroed case (61 still exactly 0.000000, having no net proton change) and 1,712 have a real mean with an undetermined covariance direction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Standalone companion to the heuristics write-up, aimed at a reader who wants to follow the defect end to end rather than the design of the fix. Covers: the pipeline step by step against origin/dev's code with file:line references; what eQuilibrator returns when it cannot decompose a reaction (standard_dg returns Q_(0, "kJ/mol") and standard_dg_prime adds the Legendre transform on top, so the stored number is the transform alone); the 61 reactions still sitting at exactly 0.000000 because they have no net proton change; the three separate breakages introduced by 3e50646 on 2023-09-13 and why each later version preserved them; and why the 2026-07-05 eQuilibrator rerun turned a handful of such records into 4,934. Impact table contrasts dev (0 of 4,934 marked "?", 2,940 given a hard direction) against this branch (4,933 marked "?"), and attributes dev's answers to the specific GC rules that produced them -- 627 directional calls from low_energy_heuristic, which reads the sign of what is only a pH transform. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third companion document, focused on the defect where the EQ reversibility step
scores canonical deltag rather than the eQuilibrator estimate. Leads with
verification rather than assertion -- four independently reproducible checks:
1. No script in Rerun_Thermodynamics.sh writes canonical deltag. The only
writer is Promote_Reaction_Thermodynamics_to_Canonical.py, which is not in
the rerun. deltag is static data, not pipeline output.
2. Estimate_Reaction_Reversibility.py:74 nonetheless reads it as the energy
source; reversibility_heuristics.py:94 looks up the eQuilibrator pair,
discards it, and returns deltag tagged with eQuilibrator's label.
3. deltag matches no currently-stored source for 49.4% of reactions, and for
EQ-eligible reactions matches the eQuilibrator value only 7.2% of the time.
Canonical deltagerr is never the eQuilibrator sentinel (0 of 4,934).
4. Replaying both energies through the cascade gives different directions for
1,575 reactions where eQuilibrator is confident (sigma <= 5 kcal/mol), e.g.
rxn00003 flips '=' to '<' on a 1.13 kcal/mol input difference.
Traces the cause to 3e50646 (2023-09-13) deleting the writer's half of a
two-sided contract, explains why it failed silently (EQU is persisted data,
deltag stayed populated, the cascade cannot see provenance), and records that no
later commit repaired it.
Also states plainly what is NOT fixed: the GC path reads the wrong energy 79.3%
of the time for the identical structural reason, left alone deliberately so the
byte-identical GC column remains a control for PR ModelSEED#285. And that deltag is
currently neither pipeline output nor a curated field, which needs a decision.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Promotes the GC path from a "not fixed" footnote to a full section, with the impact actually measured rather than inferred from the energy drift. Simulating the full pipeline both ways (GC step then EQ step, in Rerun_Thermodynamics.sh order) and comparing final canonical reversibility: 1,156 reactions differ. But the failure mode is not eQuilibrator's -- only 5 are hard directional flips and 105 turn a confident call into "?", because the stale values are mostly older GC numbers sitting a median of 2.07 kcal/mol from the current ones, close enough that the cascade usually lands the same way. The dominant effect is lost coverage: 843 reactions move from "?" to a confident call. 892 reactions hold a stored Group-contribution energy that is unreachable because canonical deltag is the 10000000 sentinel, so _energy_for returns None and _incomplete_decision emits "?" despite the energy being present. Records why the GC fix cannot be the same one-line swap the EQ fix was: of those 892, three carry |dG| > 1000 kcal/mol and five carry error > 100, and they are currently invisible precisely because Promote_Reaction_Thermodynamics_to_Canonical.py applies MAX_ABS_DG/MAX_ERR before writing deltag. Reading deltag is wrong but protective -- it is accidentally acting as a quality filter. The EQ rule set carries its own gate (eq_undecomposable_heuristic) and so needed no equivalent; the GC cascade has none, since stored_bounds_heuristic abstains on a wide error bar rather than rejecting it. A follow-up PR should swap the GC source, port the plausibility guards into the GC rule set, and report the ~884 recovered reactions as an intended coverage gain. Kept out of PR ModelSEED#285 so the byte-identical GC column remains a control. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Restructured from 343 lines to 197, led by the mental model rather than the evidence. The core is now: every reaction stores energies in two places (a worked rxn01442 record where canonical deltag is visibly dGPredictor's number, not eQuilibrator's), the two-step handoff that used to keep them in sync, and what 3e50646 broke -- with a sticky-note/whiteboard analogy for a reader who does not know the schema. Verification is kept but compressed to the four numbers that carry it: no pipeline script writes deltag, deltag matches no current source for 49.4% of reactions, it is eQuilibrator's value only 7.2% of the time in the EQ path, and 1,575 directions differ. The long code excerpts and per-rule cascade tables are dropped -- they live in the zero-energy companion note. The GC section is retained in full since it carries the finding that changes the follow-up plan: deltag is accidentally acting as a quality filter, so that fix needs a plausibility gate ported from Promote_*, not just a source swap. Also keeps the correction on record: the EQ step read eQuilibrator energies correctly for four years; "never" was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…gh it
reversibility_index.py is the Noor 2012 / eQuilibrator 2.0 index
(Bioinformatics 28:2037; NAR 40:D770) as a dependency-free function, so it can
be called on a stoichiometry and a dG without standing up a cascade.
Context.ln_gamma now delegates to it, which is what keeps the two from drifting;
agreement with eQuilibrator's own published ln_reversibility_index column rises
from 17,771 to 17,776 reactions, the gain coming from merging repeated
(compound, compartment) pairs the way eQuilibrator's Reaction does.
Transport is deliberately ordinary chemistry throughout the module: every
(compound, compartment) pair is its own species, with no membrane term, no
compartment collapse and no structural shortcut. That is what lets a transport
call be compared with a cytosolic one on the same axis, and it is the opposite
of the compartment-collapsing bug in Retrieve_eQuilibrator_Reactions_Energies.py.
test_reversibility_index.py checks the formula against hand-computed identities,
then applies it to the two families we currently decide structurally. Both
results argue against trusting the index there:
* All 15 ATP synthase reactions score an identical |ln Gamma| = 11.97, because
the translocated protons cancelled in the collapsed MetaNetX formula and
eQuilibrator scored ADP + Pi <=> ATP + H2O every time. The index calls the
most reversible enzyme in the cell irreversible and splits 7 ">" / 8 "<"
on how each reaction happened to be written.
* ABC transporters agree with the structural ATP-sign rule 94.1% of the time,
but the substrate also cancels, leaving plain ATP hydrolysis at
ln Gamma = -7.18 against a cut of -6.91. 1,074 of the 1,111 hard calls sit
within 15% of the threshold.
Both dGPredictor sources now use DGP_HEURISTICS (index at z=1, no sigma gate)
instead of falling through to the GC concentration bounds, and DGP/DGPM read
thermodynamics[label] rather than the flat deltag -- nothing has rewritten that
field since the additive refactor, so scoring a dGPredictor level off it scores
the Group-Contribution number under dGPredictor's name.
Consequence for the regression test: Group contribution is now the only
byte-compare anchor. dGPredictor was one too, on the grounds that it was scored
with the untouched GC cascade; that is no longer true, so it moves to
EXPECTED_CHANGE_SOURCES (2,816 operators re-scored). GC still matches exactly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three scripts, all read-only unless explicitly told otherwise. Compare_Reversibility_Heuristics.py hands two or more rule sets the *same* (dG, sigma) pair, so any disagreement is attributable to the rules alone. Comparing the shipped GC directions against the shipped eQuilibrator directions cannot do this, because those runs differ in their energies as well as their rules. On eQuilibrator energies GC and EQ agree on 71.6% of 25,028 reactions, and the gap is dominated by one cell: 4,369 reactions GC calls "=" and EQ calls "?", the undecomposable population the GC bounds rule swallows. Check_eQuilibrator_Energy_Errors.py finds the records where eQuilibrator reported a failure as a number. It never raises: on a residual it returns Q_(0, "kJ/mol") +/- RMSE_inf, discards the computed mean, and standard_dg_prime then adds the pH/Mg transform unconditionally, so what we store is 0 + transform. Found: 4,934 UNDECOMPOSABLE, 632 ZERO_RETURN, and -- only with --table, since column 4 is discarded on ingest -- 1,178 COLLAPSED_FORMULA where the published ln_RI disagrees with the index recomputed from our stoichiometry. Normalize_eQuilibrator_Sentinels.py translates eQuilibrator's marker into ModelSEED's [10000000.0, 10000000.0, '?'], so a consumer that correctly skips one skips the other. Detection is check_record() imported from the checker, so there is one definition of "sentinel" and not two. Dry-run by default; only thermodynamics['eQuilibrator'] is touched, never the sibling triples or the flat deltag. zero_dg is off by default, and the reason is worth recording. It is tempting to read a bare dG == 0.0 as the residual branch showing through where the transform was nil, but that branch always stamps RMSE_inf on sigma, so by construction these are not it. 280 of the 288 carry sigma == 0.0 exactly and 232 of those are one-in/one-out isomerizations -- L-lysine to D-lysine, D-threo- to D-erythro-isocitrate, 16alpha- to 16beta-hydroxysteroid. eQuilibrator's decomposition is stereo-blind, so both sides decompose to identical groups and the difference is exactly zero with exactly zero propagated error. That is a real statement that the stereoisomers are isoenergetic, not a failure report. COLLAPSED_FORMULA is excluded from the translation for a different reason: it is a wrong-but-finite energy from our own retrieval step dropping the compartment, which needs the formula rebuilt, not blanked. All three self-test. The checker's asserts the two headline cases explicitly (the 0-return energy and the 1e5 sentinel); the normaliser's covers 14 translation cases plus idempotence, and 9 end-to-end assertions over a reactions dict. --apply was exercised against a sandbox copy of Biochemistry, where it changed only the eQuilibrator entries and left everything else byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scripts/Thermodynamics/README.md gains four sections covering the standalone index, what it says about ATP synthase and ABC transporters, applying it to dGPredictor, the same-energies comparison, and the two eQuilibrator scripts -- placed after the existing "Energy source for the EQ run" subsection so the heading nesting stays intact. Papers/NAR_Update_2026/data/reversibility_index_and_sentinels_2026-08-21.md is the plain-language companion to the two 2026-08-18 eQuilibrator notes, written for a reader who does not know the cascade. It is organised by finding rather than by task, and leads with the ATP synthase result, since "the index looks like it agrees with our shortcuts and the agreement is an artifact" is the part that changes what someone would do next. The section on the confidence margin explains why z matters on exactly one source: the median propagated sigma on ln Gamma is 0.30 for dGPredictor and 0.57 for decomposable eQuilibrator records, but 18.43 for dGPredictor-ModelSEED -- 2.67x the entire ln(1000) decision threshold, with 70% of reactions carrying a one-sigma error bar larger than the threshold itself. rxn00985 (propionate kinase) is worked through end to end as the example. It also records the finding that GC's middling hard-call rate on that source is not good calibration: of the 9,922 reactions it calls forward, 54% come from the concentration-bounds test that uses sigma, 33% from the low-energy-compound rule that never looks at it, and 13% from the ABC shortcut that never looks at the energy at all. Moving from the low-sigma source to the high-sigma one, the bounds test correctly backs off (49% of reactions to 20%) and the sigma-blind rule more than doubles its forward calls as the evidence gets weaker. Every figure was recomputed from the database rather than reused from a console log, which caught two errors during drafting: the dGPredictor point-estimate hard-call rate (60.5% -> 53.6%) and a claim that GC's rate "barely moves" between sources when it drops from 58.7% to 34.9%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comparisons so far hold the energy fixed and vary the rules. --matrix does the opposite: it holds the rule set fixed and varies which prediction supplies the energy, scoring each source from its own thermodynamics[source] entry. Each cell is the percentage of shared reactions where two sources give the same direction; overlaps run from 17,021 (eQuilibrator n dGPredictor) to 25,969 (GroupContrib n dGPredictor-ModelSEED). Average pairwise agreement across the four sources falls from 71.0% under the default GC cascade to 58.2% under the index. That reads as the index making things worse, and it is the opposite -- the default cascade was manufacturing the agreement. print_agreement_decomposition splits each pair's agreement into why they agree. The best-agreeing pair in the default matrix is Group contribution and dGPredictor-ModelSEED at 83.3%, and 54.1 of those 83.3 points are both sources saying "reversible". Only 29.2% is two sources committing to the same direction. Under the index the same pair drops to 63.9% overall while the both-committed share *rises* to 32.9%, and outright conflicts -- one source forward, the other backward -- go from 0.4% to 5.7%. Those conflicts were always there; the fallback to "reversible" was hiding them. The decomposition also reports agreement excluding "?", because under the index eQuilibrator returns "?" for the 4,934 reactions it declined and a "?" matches nothing, which drags its row down for a reason that is not disagreement. Restricted to reactions where both sources made a call, eQuilibrator's pairs rise from 68.6/51.6/55.0% to 80.4/63.3/68.4%. The three pairs involving no "?" still drop under the index, so the finding stands -- but eQuilibrator is a better-behaved source than its raw row suggests, once an honest refusal stops counting as a wrong answer. Written up as section 4 of the 2026-08-21 note; the sentinel section renumbers to 5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The matrix previously applied the index to all four sources, which is not a configuration anyone would ship. Group contribution is the byte-compare anchor and the cascade was designed around its concentration bounds; only the prediction sources are candidates for the index. score_all_sources now takes a per-source rule-set mapping rather than one name, and --index-sources controls which sources take the alternate set (default: the three prediction sources). The baseline matrix is unchanged -- default cascade everywhere -- so the comparison is now "what ModelSEED does today" against "what changes if only the predictions move". Group contribution's row shifts accordingly: 68.6/51.0/63.9% under the old all-index matrix, 62.2/54.3/61.6% now. The three prediction-source pairs are untouched, since they were already on the index in both. Average pairwise agreement falls 71.0% -> 57.3%, and every pair moves the same way: agreement down, mutual "=" down, direct conflicts up. Group contribution and dGPredictor-ModelSEED are the best-agreeing pair in the default matrix at 83.3%, and 54.1 of those points are both sources saying "reversible". Also corrects an overgeneralisation in the previous write-up, which said the index converts abstention into agreed-upon directions. It does not: agreed directions fall in five of six pairs. Abstention becomes a mix of consensus and conflict, because the two rule families commit on different reactions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
samseaver
pushed a commit
that referenced
this pull request
Sep 4, 2026
Every thermodynamics["eQuilibrator"] value is now computed from the structure
ModelSEED holds for that compound, with no fuzzy matching anywhere in the
id -> structure -> energy chain.
The values these replace came from a MetaNetX-mediated retrieval that matched
ModelSEED compounds down a progressively loosening InChIKey ladder: full key,
then key minus the protonation layer, then connectivity alone. Falling back to
connectivity means an energy could rest on a different stereoisomer or
protonation state than the structure we actually curate. Nothing failed when it
did -- you got a well-formed energy for the wrong molecule.
Provenance of the new numbers:
cache eQuilibrator's Zenodo cache with every seed: accession repointed
at our structure (Path A), then training kegg: accessions
repointed so kegg:X and seed:cpd##### resolve to the same
compound (Path B). 97.1% of the 28,075 compounds where we hold a
structure resolve to it; the rest lacked pKas and keep the prior
mapping, flagged unverified.
parameters component-contribution retrained on that cache with TECRDB
de-duplicated -- 78 measurement groups that eQuilibrator counted
2-6 times, 88 redundant rows removed, improving cross-validated
prediction at p=1.7e-05.
conditions pH 7.0, ionic strength 0.25 M, pMg 3.0, 298.15 K.
Counting real energies rather than keys -- the old compound script wrote the
10000000 sentinel for structureless compounds, so 8,765 of its 30,607 entries
were never energies:
before after lost gained net
reactions 25,028 21,853 6,238 3,063 -3,175
compounds 21,842 16,372 9,223 3,753 -5,470
The new ingests write no sentinels: an entry either carries an energy or
carries no eQuilibrator key.
Most of the reaction loss is deliberate. Of the ~6,200 reactions that had a
value and no longer do, ~4,000 were outside the component-contribution span and
~1,900 do not balance -- the old code reported a number for both without
checking the infinite-uncertainty term. Roughly 270 are a genuine cache gap.
Where old and new overlap they agree to a median 0.40 kcal/mol.
The ingest moves in-tree. Update_{Reaction,Compound}_eQuilibrator_Energies.py
previously read MetaNetX_*_Energies.tbl, so a Rerun_Thermodynamics.sh pass
would have silently reverted these values to the InChIKey-ladder numbers. They
now read ModelSEED_{Reaction,Compound}_Energies.tsv, committed alongside, and
the pipeline is reproducible from a clean checkout. The .tbl files are kept as
archived inputs -- Check_eQuilibrator_Energy_Errors.py, the two reversibility
tests, Preview_Structure_Update.py and Build_Compound_Field_Provenance.py still
read them -- but they no longer feed the database.
Two new helpers in _thermo_helpers.py: parse_modelseed_energy_table, and
run_{reaction,compound}_table_update, which REMOVE the key where a record no
longer earns an energy. run_reaction_lookup_update deliberately leaves absent
reactions untouched, which is right for an incremental refresh of one pipeline
and wrong for a regeneration -- a stale number under a current-looking key is
worse than no key.
Per-source operators are computed at write time through the eQuilibrator rule
set merged in #285, rather than deferred. Only 294 reactions land on "?", where
carrying the old flags forward would have left 3,063 of them unresolved.
Also lands, all previously outside the repo:
eQuilibrator-2020_{reactions,compounds}.tsv the pre-regeneration values,
25,028 / 30,607 entries, as the
paper's baseline
missing_compounds.tsv ~4,650 compounds absent from the
cache, ranked by reactions blocked
lost_reactions_review.tsv 15 reactions that stopped
balancing under our structures
Biochemistry/Thermodynamics/eQuilibrator/README.md records the provenance and
five open review items that are NOT settled by this commit: the 19,984
compounds / 10,693 reactions outside the CC span (unaudited -- nobody has
characterised what they are), the cache-absent compounds, the 15 unbalanced
reactions, the structure checklist, and the inherited pMg 3.0 convention.
Canonical deltag/deltagerr/reversibility are untouched here. Promotion weighs
reversibility and must run once, after every source has been regenerated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
samseaver
pushed a commit
that referenced
this pull request
Sep 4, 2026
…redictions Three changes, all to thermodynamics["dGPredictor"]. 1. RETRAINED, because the model was broken. The predecessor dGPredictor-ModelSEED record had a median uncertainty of 88.6 kJ/mol (259.5 for quinones), 44.5% of predictions past 100 kJ/mol, worst error 8,531. The cause was RDKit canonicalization drift: the Wang lab shipped a fragment vocabulary written by an older RDKit (CC(C)=O) and current RDKit emits CC(=O)C for the same fragment, so reindexing silently dropped most fragment counts. BayesianRidge then extrapolated from its prior, and the prior covariance is what those error bars were. The fix is a guarantee rather than a patch: the vocabulary is rebuilt from current-RDKit output, and training and prediction call the same count_substructures, so they can no longer disagree about a fragment's name. 2. FITTED TO THE SAME MEASUREMENTS AS eQUILIBRATOR. Training moves from the raw Zenodo TECRDB.tsv (4,544 rows) to TECRDB_dedup.csv (4,456) -- the table component-contribution's cc_params_dedup.npz was fitted to. The 88 removed rows are redundant copies of 78 measurement groups counted 2-6 times; one copy of each is kept, so no measurement is lost. Fitting the two methods to different row sets of the same measurements is not defensible when the paper compares them head to head. Training RMSE rises 5.4 -> 5.85 kJ/mol, which is what removing rows the fit got for free must do. The question needs held-out data, so scripts/08 runs the comparison eQuilibrator ran for component-contribution: 10 folds, both arms predicting an identical test set, every copy of a tested measurement withheld from both arms so the raw arm cannot win on leakage. Over 3,633 held-out measurements, median |error| 1.7125 -> 1.6952, closer on 53.3%, paired Wilcoxon p=1.5e-07. Highly significant and small -- the point is that consistency with eQuilibrator costs nothing measurable, not that de-duplication is a large win. Per-row errors in crossval_training_sets.tsv. 3. NO COVERAGE FLOOR. Every prediction ships with its own uncertainty. The earlier policy installed only coverage >= 0.9 (8,279 reactions), on the reasoning that below that a prediction is prior-dominated and is "we don't know" wearing a wide error bar. Measured against eQuilibrator on the 20,691 reactions both methods score, that does not hold: the fraction within one combined sigma is 67-79% in EVERY coverage stratum against 68% nominal. At coverage < 0.5 the model says +/-111 kJ/mol and is off by 87. The bars are honest, and mildly conservative if anything. Coverage also turned out to be the wrong axis to filter on. Spearman against |deviation from eQuilibrator| is -0.349 for coverage but +0.650 for sigma, and the two correlate only -0.498 with each other. The 0.9 floor was admitting 1,947 predictions with high sigma (median deviation 43.3 kJ/mol) while rejecting 5,038 with low sigma (median deviation 12.3). So all 29,617 reactions and 30,185 compounds are installed, and consumers filter on sigma. Coverage is stored as provenance -- it explains why sigma is large -- but the README says plainly that it is not the filter, and gives a sigma-threshold table instead. DGP_COVERAGE_FLOOR reinstates a floor if one is ever wanted; it defaults to 0. This does not smuggle false confidence into the direction field. DGP_HEURISTICS (the Noor 2012 reversibility index at one sigma, merged in #285) reads the wide bars and declines: of 29,617 records, 20,723 are "=", 6,870 ">" and 2,024 "<". Record shape gains coverage, appended so positional readers are unaffected: reaction [dg, err, operator, coverage] compound [dg, err, coverage] no operator; formation energy has no direction Both ingest scripts, and the swap, now pass source= to reversibility_from_energy. Without it a dGPredictor energy is silently scored with Group-Contribution rules, and the stored operator disagrees with what Estimate_Reaction_Reversibility.py computes for the same record. This was latent in the pre-#285 code and would have been wrong the moment #285 landed. Coverage against the other sources, on 56,012 reactions: Group contribution 56,002 eQuilibrator 21,853 dGPredictor 29,617 Compounds gain dGPredictor as a third source: 30,185 records. dGPredictor and eQuilibrator overlap on 7,247 reactions at the old floor and 20,691 without it -- that overlap is the honest denominator for any head-to-head comparison. dGPredictor reaches 1,032 reactions eQuilibrator cannot (518 unbalanced, 384 outside the CC span), so neither method dominates. Also lands: dGPredictor-2020.tsv 27,715 archived KEGG-trained predictions, the paper's baseline. Read with care: 17,271 of them (62%) hold an energy computed for a KEGG reaction that is not among that reaction's aliases -- the 2023 staging notebook's stale-variable carry-over (PR #286). Confirmed here independently: only 10,444 rows carry a matching KEGG id. retrained_dG*.json staged inputs, all predictions crossval_training_sets.tsv per-row held-out errors, both training sets README.md provenance, the calibration evidence, and what to filter on Removed: Update_Reaction_dGPredictor_ModelSEED_Energies.py, the 61 json_files and 61 pickle_files staging shards, and modelseed_retrained_dG.json. 07_atomic_swap.py now refuses to overwrite an existing dGPredictor-2020.tsv. Re-running it after a swap would otherwise capture the RETRAINED values as the "2020 baseline" and destroy the only copy of the pre-retrain record. Canonical deltag/deltagerr/reversibility are untouched and are now stale with respect to every source beneath them. Re-promotion is deliberately out of scope: the field is being retired. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
samseaver
pushed a commit
that referenced
this pull request
Sep 4, 2026
Rerun_Thermodynamics.sh had drifted into three disagreeing versions across
branches. This is the reconciled one:
* Update_Reaction_dGPredictor_ModelSEED_Energies.py is gone -- the script was
deleted with the retrain.
* Update_Compound_dGPredictor_Energies.py is added -- dGPredictor is now a
compound-level source too.
* The canonical-field steps (Estimate_Reaction_Reversibility.py GC / EQ and
Promote_Reaction_Thermodynamics_to_Canonical.py) are documented but NOT
run, because those fields are being retired. Leaving them silently in the
chain would keep rewriting a field we are about to delete, from sources
that have all moved.
* set -euo pipefail, so a failing step stops the chain instead of letting the
next one run against half-updated JSON.
Add_Reaction_Thermodynamics_Operators.py rebuilt each record as a fixed
[dg, dge, operator] triple, which silently truncated dGPredictor's fourth
element (coverage) on every backfill. It now preserves everything past the
operator. Verified: 107,472 entries seen, 0 refreshed, no JSON diff -- which
also confirms independently that the operators each updater wrote at write time
match what the cascade computes on its own.
Scripts/Thermodynamics/README.md documents the per-source record shapes (they
now differ by kind AND source), the source= requirement on
reversibility_from_energy, and that the canonical fields are stale and being
retired. The thirteen sections #285 added are untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The reaction-direction cascade had exactly one rule set —
DEFAULT_HEURISTICS, derived from the Group-Contribution era — applied to every thermodynamic source. This splits it into a registry keyed by source:GC(default)=EQ=EQ2GCis the default for any source without a set of its own, so both dGPredictor variants are unchanged. The GC cascade itself is untouched.Why
Three defects in the eQuilibrator path. (Points 1 and 2 were re-characterised after review — see the "Corrected after review" note at the bottom. The code did not change; the explanation of why it was needed was wrong.)
1. The
EQrun stopped reading eQuilibrator energies in 2023. The design was correct. Fromc263e233(2019) through17c9739b(2023-09-11),Update_Reaction_eQuilibrator_Energies.pyoverwrote the canonical energy and stamped the note that gates the reversibility step:So reading
deltaggated onEQUwas reading eQuilibrator energies, exactly as intended.3e50646b(2023-09-13) removed the write half and left the read half in place — the updater moved tothermodynamics['eQuilibrator']only, dropping both thedeltagoverwrite and theEQUstamp. Later GC rebuilds (Convention A,ad34d6ab) and the PR #265 promotion then rewrotedeltagfrom other sources. Today only 1,797 of 25,028 reactions with an eQuilibrator record havedeltag == thermodynamics['eQuilibrator'][0]; restricted to the 17,094 still carryingEQU, only 1,586 do.EQUis now a fossil of the pre-2023 pipeline. Pointing the EQ run at the eQuilibrator sublist restores the original intent rather than inventing a new one.2. For most σ-flagged reactions eQuilibrator returns no energy at all — we stored a transform term as if it were one.
GibbsEnergyPredictor.standard_dgshort-circuits when a component falls outside both the reactant- and group-contribution spans:Literally zero — the computed mean is discarded, not projected.
standard_dg_prime, which our retrieval script calls, then adds the Legendre/pH transform on top, so the nonzero ΔG′° we store is only the transform.rxn00017's −20.46 kcal/mol is the pH-7 transform of an energy eQuilibrator declined to estimate.Of the 4,607 σ-flagged reactions: 2,895 are this zeroed case (σ exactly
RMSE_inf; 61 still exactly0.000000, having no net proton change for the transform to act on — e.g.rxn02677,rxn04059), and 1,712 carryRMSE_inf·‖σ_inf‖and do have a real mean but with an undetermined covariance direction. Neither supports a directional call; the GC bounds rule cannot fire at that width, so all of them fell through to a permissive=.3. Transport energies are compartment-collapsed.
Retrieve_eQuilibrator_Reactions_Energies.pykeys its MetaNetX formula on compound id and discards compartment, so any species on both sides nets out — 1,102 transport reactions carry a ΔG′° for a different reaction (rxn12518at +524 kcal/mol). A further 76 non-transport reactions collapse via stereo-neutral InChIKey matching (rxn00816: D-glucose and galactose merge). Beber 2022 separately notes the transformed framework needs a−N_H·RT·ln(10^ΔpH) − Q·FΔΦterm we never apply.Separately: the reversibility index eQuilibrator has been computing for us — column 4 of
MetaNetX_Reaction_Energies.tbl— was parsed and discarded on every run, andmake_ln_reversibility_index_heuristicwas never invoked. It now decides 8,944 reactions, and the bare no-evidencedefaultfallback drops from 7,120 to zero.Results
Per-method operators vs current
dev:Only eQuilibrator moves. The three columns scored with the untouched GC rule set are byte-identical, which isolates the change to the new cutoff rather than the refactor.
Canonical reversibility:
>12,670→9,584 (−3,086) ·<2,323→1,603 (−720) ·=15,059→14,002 (−1,057) ·?25,960→30,823 (+4,863)The database gets less confident, and that is the point. The +4,863 unknowns track the σ-flagged records almost exactly — reactions that had been getting a permissive
=on the strength of a number that, for 2,895 of them, is eQuilibrator's literal zero plus a pH transform.Verification
Scripts/Tests/test_eq_heuristics.py— the ln Γ implementation reproduces eQuilibrator's own publishedln_reversibility_indexon 17,771 reactions; every residual is a MetaNetX-collapsed reaction, asserted rather than assumed.Scripts/Tests/test_reaction_direction.pynow separates invariant sources (GC, dGPredictor — must match exactly) from intentionally re-scored ones (eQuilibrator), with--strictto require equality everywhere. Passes.Rerun_Thermodynamics.shis idempotent: a second run reproduces every file byte-for-byte.rxn00001→>(ln Γ −9.18) ·rxn00017→?·rxn12518→?·rxn08173ATP synthase →=.Full write-up:
Papers/NAR_Update_2026/data/equilibrator_reversibility_heuristics_2026-08-18.md.Scope notes for review
deltagis untouched and mostly remains the Group-Contribution value. Reconciling it isPromote_Reaction_Thermodynamics_to_Canonical.py, deliberately not part ofRerun_Thermodynamics.shand not run here.>calls are the main review surface: some are the index genuinely disagreeing with the GC bounds rule, others are reactions whose only evidence was the zeroed value. Separable from the generated report on request.is_using_group_contribution()would replace the σ-threshold with eQuilibrator's own verdict;standard_dg_prime_multi()returns the covariance the current per-reaction loop cannot; and the MetaNetX collapse in the retrieval script is fixable at source.Corrected after review
Commit
a86f6adfrevises the framing of defects 1 and 2 following review. The original text claimed the EQ run "never" read eQuilibrator energies (it did, correctly, until 2023) and that the σ marker was "read as an error bar" (understated — the energy itself is a discarded mean replaced by zero). No code changed.🤖 Generated with Claude Code