Skip to content

Select reversibility heuristics per thermodynamic data source (eQuilibrator gets its own cutoff) - #285

Merged
samseaver merged 13 commits into
ModelSEED:devfrom
Cooper-Taylor:eQuilibrator-fix
Aug 29, 2026
Merged

Select reversibility heuristics per thermodynamic data source (eQuilibrator gets its own cutoff)#285
samseaver merged 13 commits into
ModelSEED:devfrom
Cooper-Taylor:eQuilibrator-fix

Conversation

@Cooper-Taylor

@Cooper-Taylor Cooper-Taylor commented Aug 18, 2026

Copy link
Copy Markdown

Summary

The reaction-direction cascade had exactly one rule set — DEFAULT_HEURISTICS, derived from the Group-Contribution era — applied to every thermodynamic source. This splits it into a registry keyed by source:

set cascade basis
GC (default) ATPS → ABCT → MdeltaG bounds → mMdeltaG band → lowE → = Jankowski 2008
EQ ATPS → ABCT → undecomposable → uncorrected transport → ln Γ ± 1σ → = Noor 2012 index, Beber 2022 uncertainty
EQ2 same, index as a bare point estimate Flamholz 2012

GC is the default for any source without a set of its own, so both dGPredictor variants are unchanged. The GC cascade itself is untouched.

Why

Three defects in the eQuilibrator path. (Points 1 and 2 were re-characterised after review — see the "Corrected after review" note at the bottom. The code did not change; the explanation of why it was needed was wrong.)

1. The EQ run stopped reading eQuilibrator energies in 2023. The design was correct. From c263e233 (2019) through 17c9739b (2023-09-11), Update_Reaction_eQuilibrator_Energies.py overwrote the canonical energy and stamped the note that gates the reversibility step:

reactions_dict[rxn]['deltag']    = float(eq_reactions[rxn]['dg'])
reactions_dict[rxn]['deltagerr'] = float(eq_reactions[rxn]['dge'])
if('EQU' not in notes_list): notes_list.append('EQU')

So reading deltag gated on EQU was reading eQuilibrator energies, exactly as intended.

3e50646b (2023-09-13) removed the write half and left the read half in place — the updater moved to thermodynamics['eQuilibrator'] only, dropping both the deltag overwrite and the EQU stamp. Later GC rebuilds (Convention A, ad34d6ab) and the PR #265 promotion then rewrote deltag from other sources. Today only 1,797 of 25,028 reactions with an eQuilibrator record have deltag == thermodynamics['eQuilibrator'][0]; restricted to the 17,094 still carrying EQU, only 1,586 do. EQU is now a fossil of the pre-2023 pipeline. Pointing the EQ run at the eQuilibrator sublist restores the original intent rather than inventing a new one.

2. For most σ-flagged reactions eQuilibrator returns no energy at all — we stored a transform term as if it were one. GibbsEnergyPredictor.standard_dg short-circuits when a component falls outside both the reactant- and group-contribution spans:

mu, sigma_fin, sigma_inf, residual = self.get_reaction_prediction(reaction)
if residual:
    return Q_(0, "kJ/mol").plus_minus(self.preprocess.RMSE_inf)

Literally zero — the computed mean is discarded, not projected. standard_dg_prime, which our retrieval script calls, then adds the Legendre/pH transform on top, so the nonzero ΔG′° we store is only the transform. rxn00017's −20.46 kcal/mol is the pH-7 transform of an energy eQuilibrator declined to estimate.

Of the 4,607 σ-flagged reactions: 2,895 are this zeroed case (σ exactly RMSE_inf; 61 still exactly 0.000000, having no net proton change for the transform to act on — e.g. rxn02677, rxn04059), and 1,712 carry RMSE_inf·‖σ_inf‖ and do have a real mean but with an undetermined covariance direction. Neither supports a directional call; the GC bounds rule cannot fire at that width, so all of them fell through to a permissive =.

3. Transport energies are compartment-collapsed. Retrieve_eQuilibrator_Reactions_Energies.py keys its MetaNetX formula on compound id and discards compartment, so any species on both sides nets out — 1,102 transport reactions carry a ΔG′° for a different reaction (rxn12518 at +524 kcal/mol). A further 76 non-transport reactions collapse via stereo-neutral InChIKey matching (rxn00816: D-glucose and galactose merge). Beber 2022 separately notes the transformed framework needs a −N_H·RT·ln(10^ΔpH) − Q·FΔΦ term we never apply.

Separately: the reversibility index eQuilibrator has been computing for us — column 4 of MetaNetX_Reaction_Energies.tbl — was parsed and discarded on every run, and make_ln_reversibility_index_heuristic was never invoked. It now decides 8,944 reactions, and the bare no-evidence default fallback drops from 7,120 to zero.

Results

Per-method operators vs current dev:

source records changed
Group contribution 56,002 0
eQuilibrator 25,028 7,103
dGPredictor 27,715 0
dGPredictor-ModelSEED 31,924 0

Only eQuilibrator moves. The three columns scored with the untouched GC rule set are byte-identical, which isolates the change to the new cutoff rather than the refactor.

Canonical reversibility: > 12,670→9,584 (−3,086) · < 2,323→1,603 (−720) · = 15,059→14,002 (−1,057) · ? 25,960→30,823 (+4,863)

The database gets less confident, and that is the point. The +4,863 unknowns track the σ-flagged records almost exactly — reactions that had been getting a permissive = on the strength of a number that, for 2,895 of them, is eQuilibrator's literal zero plus a pH transform.

Verification

  • New Scripts/Tests/test_eq_heuristics.py — the ln Γ implementation reproduces eQuilibrator's own published ln_reversibility_index on 17,771 reactions; every residual is a MetaNetX-collapsed reaction, asserted rather than assumed.
  • Scripts/Tests/test_reaction_direction.py now separates invariant sources (GC, dGPredictor — must match exactly) from intentionally re-scored ones (eQuilibrator), with --strict to require equality everywhere. Passes.
  • Rerun_Thermodynamics.sh is idempotent: a second run reproduces every file byte-for-byte.
  • Spot checks: rxn00001> (ln Γ −9.18) · rxn00017? · rxn12518? · rxn08173 ATP synthase → =.

Full write-up: Papers/NAR_Update_2026/data/equilibrator_reversibility_heuristics_2026-08-18.md.

Scope notes for review

  • Canonical deltag is untouched and mostly remains the Group-Contribution value. Reconciling it is Promote_Reaction_Thermodynamics_to_Canonical.py, deliberately not part of Rerun_Thermodynamics.sh and not run here.
  • The ~3,086 lost > calls are the main review surface: some are the index genuinely disagreeing with the GC bounds rule, others are reactions whose only evidence was the zeroed value. Separable from the generated report on request.
  • Follow-ups worth considering: is_using_group_contribution() would replace the σ-threshold with eQuilibrator's own verdict; standard_dg_prime_multi() returns the covariance the current per-reaction loop cannot; and the MetaNetX collapse in the retrieval script is fixable at source.

Corrected after review

Commit a86f6adf revises the framing of defects 1 and 2 following review. The original text claimed the EQ run "never" read eQuilibrator energies (it did, correctly, until 2023) and that the σ marker was "read as an error bar" (understated — the energy itself is a discarded mean replaced by zero). No code changed.

🤖 Generated with Claude Code

Cooper-Taylor and others added 13 commits August 18, 2026 07:45
The direction cascade had exactly one rule set, DEFAULT_HEURISTICS, derived
from the Group-Contribution era and applied to every source. Split it into a
registry keyed by source, with GC as the default for anything without a set of
its own.

  GC   unchanged Jankowski 2008 cascade -- the default
  EQ   eQuilibrator: Noor 2012 reversibility index, gated by Beber 2022
       uncertainty (new)
  EQ2  eQuilibrator 2.0: the same index as a bare point estimate

Three defects in the eQuilibrator path motivated this:

1. `Estimate_Reaction_Reversibility.py EQ` never read eQuilibrator energies.
   It pulled the canonical `deltag`, merely gated on eQuilibrator eligibility.
   Since the additive-thermodynamics refactor nothing overwrites `deltag`, so
   only 1,797 of 25,028 reactions with an eQuilibrator record actually had
   `deltag == thermodynamics['eQuilibrator'][0]`; the other 23,140 were scored
   on the Group-Contribution number and labelled eQuilibrator. EQ runs now read
   the eQuilibrator sublist's own dG and sigma.

2. eQuilibrator's ~1e5 kJ/mol "cannot decompose this reaction" marker was being
   consumed as an error bar. 4,933 records carry it; the GC bounds rule cannot
   fire that wide, so they fell through to a permissive "=". They now return
   "?". Observed real sigma tops out at 65.35 kcal/mol against a marker of
   23,900.57, so the cut at 1e4 kJ/mol sits in an empty gap.

3. `Retrieve_eQuilibrator_Reactions_Energies.py` keys its MetaNetX formula on
   compound id and so discards compartment, collapsing any species present on
   both sides; 1,102 transport reactions carry a dG for a different reaction.
   Beber 2022 separately notes the transformed framework needs a
   -N_H*RT*ln(10^dpH) - Q*F*dPhi term across a membrane that we never apply.
   Transport is now decided structurally (ATPS/ABCT) or returns "?".

The reversibility index eQuilibrator has been computing for us since the table
was first generated -- column 4 of MetaNetX_Reaction_Energies.tbl -- was parsed
and discarded on every run, and `make_ln_reversibility_index_heuristic` was
never invoked. It now decides 8,944 reactions, and the bare no-evidence
`default` fallback drops from 7,120 to zero.

Add_Reaction_Thermodynamics_Operators, _thermo_helpers and Promote_* now pass
the source label through, so each method's stored operator is computed with its
own rule set instead of GC's.

Scripts/Tests/test_eq_heuristics.py verifies the ln(Gamma) implementation
reproduces eQuilibrator's own published ln_reversibility_index on 17,771
reactions; every residual is a MetaNetX-collapsed reaction.
test_reaction_direction.py now distinguishes invariant sources (GC,
dGPredictor -- must match exactly) from intentionally re-scored ones
(eQuilibrator), with --strict to require equality everywhere.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Output of Scripts/Thermodynamics/Rerun_Thermodynamics.sh with the per-source
rule sets. Verified idempotent: a second run reproduces every file byte-for-byte
("Entries refreshed/added: 0").

Per-method operators, vs the previous dev state:

  Group contribution      56,002 records        0 changed
  eQuilibrator            25,028 records    7,103 changed
  dGPredictor             27,715 records        0 changed
  dGPredictor-ModelSEED   31,924 records        0 changed

Only eQuilibrator moves. The three columns scored with the untouched GC rule
set are byte-identical, which is what isolates the change to the new cutoff
rather than the refactor.

Canonical reversibility:  >  12,670 -> 9,584   (-3,086)
                          <   2,323 -> 1,603     (-720)
                          =  15,059 -> 14,002  (-1,057)
                          ?  25,960 -> 30,823  (+4,863)

The database gets less confident, and that is the point: the +4,863 unknowns
track the 4,933 undecomposable records almost exactly. Those had been receiving
a permissive "=" or an inherited direction on the strength of a dG carrying
+/-23,900 kcal/mol -- an unread sentinel, not evidence.

Deciding rule for the EQ run:

  Incomplete (no eQuilibrator energy)   24,828
  EQ:lnGamma                             8,944
  EQ:reversible                          8,905
  Incomplete (GCC, keeps GC answer)      6,146
  EQ:undecomposable                      4,933
  ABCT                                   1,151
  EQ:ambiguous                             871
  EQ:transport-uncorrected                 209
  ATPS                                      15
  Empty                                     10

Canonical deltag is untouched and mostly remains the Group-Contribution value;
reconciling it is Promote_Reaction_Thermodynamics_to_Canonical.py, which is
deliberately not part of Rerun_Thermodynamics.sh and was not run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Write-up of the work behind PR ModelSEED#285, following the convention of the other
Papers/NAR_Update_2026/data/ investigation notes.

Records what dev was actually doing and how that was established (the
eQuilibrator reversibility index was computed, stored in column 4 of
MetaNetX_Reaction_Energies.tbl, and discarded unread on every run), the three
defects in the eQuilibrator path, and the results of the rerun.

Section 3 is the part worth keeping for a methods section: eQuilibrator ships
no directionality classifier, Beber 2022 defines no reversibility index and
defers directionality to downstream tools, and the table there separates the
parts of EQ_HEURISTICS that are faithful reimplementation (the ln Gamma
formula, verified against 17,771 published values) from the parts that are
ours (the +/-1 sigma margin, the undecomposable and transport gates).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both corrections came from supervisor review of PR ModelSEED#285. The code is unchanged;
the characterisation of why it was needed was wrong.

1. "The EQ run never read eQuilibrator energies" was wrong. The design was
   correct: from c263e23 (2019) through 17c9739 (2023-09-11),
   Update_Reaction_eQuilibrator_Energies.py overwrote canonical deltag/deltagerr
   with the eQuilibrator values and stamped the EQU note, so
   `Estimate_Reaction_Reversibility.py EQ` reading deltag gated on EQU was
   reading eQuilibrator energies exactly as intended.

   3e50646 (2023-09-13) removed the write half -- moving the updater to
   thermodynamics['eQuilibrator'] only, dropping both the deltag overwrite and
   the EQU stamp -- and left the read half in place. Later GC rebuilds and the
   PR ModelSEED#265 promotion then rewrote deltag from other sources. Only 1,586 of the
   17,094 reactions still carrying EQU have deltag == the eQuilibrator value.
   EQU is a fossil of the pre-2023 pipeline, and is distinct from EQC (which
   the retrieval script writes to mean "all reagents had structures").

2. "The marker was read as an error bar" understated it. For the residual case
   GibbsEnergyPredictor.standard_dg returns Q_(0, "kJ/mol") outright -- the
   computed mean is discarded, not projected. standard_dg_prime then adds the
   Legendre/pH transform, so the nonzero dG'o stored for those reactions is the
   transform alone. Of 4,607 sigma-flagged reactions, 2,895 are this zeroed
   case (61 still exactly 0.000000, having no net proton change) and 1,712 have
   a real mean with an undetermined covariance direction.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Standalone companion to the heuristics write-up, aimed at a reader who wants to
follow the defect end to end rather than the design of the fix.

Covers: the pipeline step by step against origin/dev's code with file:line
references; what eQuilibrator returns when it cannot decompose a reaction
(standard_dg returns Q_(0, "kJ/mol") and standard_dg_prime adds the Legendre
transform on top, so the stored number is the transform alone); the 61
reactions still sitting at exactly 0.000000 because they have no net proton
change; the three separate breakages introduced by 3e50646 on 2023-09-13 and
why each later version preserved them; and why the 2026-07-05 eQuilibrator
rerun turned a handful of such records into 4,934.

Impact table contrasts dev (0 of 4,934 marked "?", 2,940 given a hard
direction) against this branch (4,933 marked "?"), and attributes dev's answers
to the specific GC rules that produced them -- 627 directional calls from
low_energy_heuristic, which reads the sign of what is only a pH transform.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third companion document, focused on the defect where the EQ reversibility step
scores canonical deltag rather than the eQuilibrator estimate. Leads with
verification rather than assertion -- four independently reproducible checks:

  1. No script in Rerun_Thermodynamics.sh writes canonical deltag. The only
     writer is Promote_Reaction_Thermodynamics_to_Canonical.py, which is not in
     the rerun. deltag is static data, not pipeline output.
  2. Estimate_Reaction_Reversibility.py:74 nonetheless reads it as the energy
     source; reversibility_heuristics.py:94 looks up the eQuilibrator pair,
     discards it, and returns deltag tagged with eQuilibrator's label.
  3. deltag matches no currently-stored source for 49.4% of reactions, and for
     EQ-eligible reactions matches the eQuilibrator value only 7.2% of the time.
     Canonical deltagerr is never the eQuilibrator sentinel (0 of 4,934).
  4. Replaying both energies through the cascade gives different directions for
     1,575 reactions where eQuilibrator is confident (sigma <= 5 kcal/mol), e.g.
     rxn00003 flips '=' to '<' on a 1.13 kcal/mol input difference.

Traces the cause to 3e50646 (2023-09-13) deleting the writer's half of a
two-sided contract, explains why it failed silently (EQU is persisted data,
deltag stayed populated, the cascade cannot see provenance), and records that no
later commit repaired it.

Also states plainly what is NOT fixed: the GC path reads the wrong energy 79.3%
of the time for the identical structural reason, left alone deliberately so the
byte-identical GC column remains a control for PR ModelSEED#285. And that deltag is
currently neither pipeline output nor a curated field, which needs a decision.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Promotes the GC path from a "not fixed" footnote to a full section, with the
impact actually measured rather than inferred from the energy drift.

Simulating the full pipeline both ways (GC step then EQ step, in
Rerun_Thermodynamics.sh order) and comparing final canonical reversibility:
1,156 reactions differ. But the failure mode is not eQuilibrator's -- only 5 are
hard directional flips and 105 turn a confident call into "?", because the stale
values are mostly older GC numbers sitting a median of 2.07 kcal/mol from the
current ones, close enough that the cascade usually lands the same way.

The dominant effect is lost coverage: 843 reactions move from "?" to a confident
call. 892 reactions hold a stored Group-contribution energy that is unreachable
because canonical deltag is the 10000000 sentinel, so _energy_for returns None
and _incomplete_decision emits "?" despite the energy being present.

Records why the GC fix cannot be the same one-line swap the EQ fix was: of those
892, three carry |dG| > 1000 kcal/mol and five carry error > 100, and they are
currently invisible precisely because Promote_Reaction_Thermodynamics_to_Canonical.py
applies MAX_ABS_DG/MAX_ERR before writing deltag. Reading deltag is wrong but
protective -- it is accidentally acting as a quality filter. The EQ rule set
carries its own gate (eq_undecomposable_heuristic) and so needed no equivalent;
the GC cascade has none, since stored_bounds_heuristic abstains on a wide error
bar rather than rejecting it.

A follow-up PR should swap the GC source, port the plausibility guards into the
GC rule set, and report the ~884 recovered reactions as an intended coverage
gain. Kept out of PR ModelSEED#285 so the byte-identical GC column remains a control.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Restructured from 343 lines to 197, led by the mental model rather than the
evidence. The core is now: every reaction stores energies in two places (a
worked rxn01442 record where canonical deltag is visibly dGPredictor's number,
not eQuilibrator's), the two-step handoff that used to keep them in sync, and
what 3e50646 broke -- with a sticky-note/whiteboard analogy for a reader who
does not know the schema.

Verification is kept but compressed to the four numbers that carry it: no
pipeline script writes deltag, deltag matches no current source for 49.4% of
reactions, it is eQuilibrator's value only 7.2% of the time in the EQ path, and
1,575 directions differ. The long code excerpts and per-rule cascade tables are
dropped -- they live in the zero-energy companion note.

The GC section is retained in full since it carries the finding that changes the
follow-up plan: deltag is accidentally acting as a quality filter, so that fix
needs a plausibility gate ported from Promote_*, not just a source swap.

Also keeps the correction on record: the EQ step read eQuilibrator energies
correctly for four years; "never" was wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…gh it

reversibility_index.py is the Noor 2012 / eQuilibrator 2.0 index
(Bioinformatics 28:2037; NAR 40:D770) as a dependency-free function, so it can
be called on a stoichiometry and a dG without standing up a cascade.
Context.ln_gamma now delegates to it, which is what keeps the two from drifting;
agreement with eQuilibrator's own published ln_reversibility_index column rises
from 17,771 to 17,776 reactions, the gain coming from merging repeated
(compound, compartment) pairs the way eQuilibrator's Reaction does.

Transport is deliberately ordinary chemistry throughout the module: every
(compound, compartment) pair is its own species, with no membrane term, no
compartment collapse and no structural shortcut. That is what lets a transport
call be compared with a cytosolic one on the same axis, and it is the opposite
of the compartment-collapsing bug in Retrieve_eQuilibrator_Reactions_Energies.py.

test_reversibility_index.py checks the formula against hand-computed identities,
then applies it to the two families we currently decide structurally. Both
results argue against trusting the index there:

  * All 15 ATP synthase reactions score an identical |ln Gamma| = 11.97, because
    the translocated protons cancelled in the collapsed MetaNetX formula and
    eQuilibrator scored ADP + Pi <=> ATP + H2O every time. The index calls the
    most reversible enzyme in the cell irreversible and splits 7 ">" / 8 "<"
    on how each reaction happened to be written.
  * ABC transporters agree with the structural ATP-sign rule 94.1% of the time,
    but the substrate also cancels, leaving plain ATP hydrolysis at
    ln Gamma = -7.18 against a cut of -6.91. 1,074 of the 1,111 hard calls sit
    within 15% of the threshold.

Both dGPredictor sources now use DGP_HEURISTICS (index at z=1, no sigma gate)
instead of falling through to the GC concentration bounds, and DGP/DGPM read
thermodynamics[label] rather than the flat deltag -- nothing has rewritten that
field since the additive refactor, so scoring a dGPredictor level off it scores
the Group-Contribution number under dGPredictor's name.

Consequence for the regression test: Group contribution is now the only
byte-compare anchor. dGPredictor was one too, on the grounds that it was scored
with the untouched GC cascade; that is no longer true, so it moves to
EXPECTED_CHANGE_SOURCES (2,816 operators re-scored). GC still matches exactly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three scripts, all read-only unless explicitly told otherwise.

Compare_Reversibility_Heuristics.py hands two or more rule sets the *same*
(dG, sigma) pair, so any disagreement is attributable to the rules alone.
Comparing the shipped GC directions against the shipped eQuilibrator directions
cannot do this, because those runs differ in their energies as well as their
rules. On eQuilibrator energies GC and EQ agree on 71.6% of 25,028 reactions,
and the gap is dominated by one cell: 4,369 reactions GC calls "=" and EQ calls
"?", the undecomposable population the GC bounds rule swallows.

Check_eQuilibrator_Energy_Errors.py finds the records where eQuilibrator
reported a failure as a number. It never raises: on a residual it returns
Q_(0, "kJ/mol") +/- RMSE_inf, discards the computed mean, and standard_dg_prime
then adds the pH/Mg transform unconditionally, so what we store is
0 + transform. Found: 4,934 UNDECOMPOSABLE, 632 ZERO_RETURN, and -- only with
--table, since column 4 is discarded on ingest -- 1,178 COLLAPSED_FORMULA where
the published ln_RI disagrees with the index recomputed from our stoichiometry.

Normalize_eQuilibrator_Sentinels.py translates eQuilibrator's marker into
ModelSEED's [10000000.0, 10000000.0, '?'], so a consumer that correctly skips
one skips the other. Detection is check_record() imported from the checker, so
there is one definition of "sentinel" and not two. Dry-run by default; only
thermodynamics['eQuilibrator'] is touched, never the sibling triples or the flat
deltag.

zero_dg is off by default, and the reason is worth recording. It is tempting to
read a bare dG == 0.0 as the residual branch showing through where the transform
was nil, but that branch always stamps RMSE_inf on sigma, so by construction
these are not it. 280 of the 288 carry sigma == 0.0 exactly and 232 of those are
one-in/one-out isomerizations -- L-lysine to D-lysine, D-threo- to
D-erythro-isocitrate, 16alpha- to 16beta-hydroxysteroid. eQuilibrator's
decomposition is stereo-blind, so both sides decompose to identical groups and
the difference is exactly zero with exactly zero propagated error. That is a
real statement that the stereoisomers are isoenergetic, not a failure report.

COLLAPSED_FORMULA is excluded from the translation for a different reason: it is
a wrong-but-finite energy from our own retrieval step dropping the compartment,
which needs the formula rebuilt, not blanked.

All three self-test. The checker's asserts the two headline cases explicitly
(the 0-return energy and the 1e5 sentinel); the normaliser's covers 14
translation cases plus idempotence, and 9 end-to-end assertions over a reactions
dict. --apply was exercised against a sandbox copy of Biochemistry, where it
changed only the eQuilibrator entries and left everything else byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scripts/Thermodynamics/README.md gains four sections covering the standalone
index, what it says about ATP synthase and ABC transporters, applying it to
dGPredictor, the same-energies comparison, and the two eQuilibrator scripts --
placed after the existing "Energy source for the EQ run" subsection so the
heading nesting stays intact.

Papers/NAR_Update_2026/data/reversibility_index_and_sentinels_2026-08-21.md is
the plain-language companion to the two 2026-08-18 eQuilibrator notes, written
for a reader who does not know the cascade. It is organised by finding rather
than by task, and leads with the ATP synthase result, since "the index looks
like it agrees with our shortcuts and the agreement is an artifact" is the part
that changes what someone would do next.

The section on the confidence margin explains why z matters on exactly one
source: the median propagated sigma on ln Gamma is 0.30 for dGPredictor and 0.57
for decomposable eQuilibrator records, but 18.43 for dGPredictor-ModelSEED --
2.67x the entire ln(1000) decision threshold, with 70% of reactions carrying a
one-sigma error bar larger than the threshold itself. rxn00985 (propionate
kinase) is worked through end to end as the example.

It also records the finding that GC's middling hard-call rate on that source is
not good calibration: of the 9,922 reactions it calls forward, 54% come from the
concentration-bounds test that uses sigma, 33% from the low-energy-compound rule
that never looks at it, and 13% from the ABC shortcut that never looks at the
energy at all. Moving from the low-sigma source to the high-sigma one, the
bounds test correctly backs off (49% of reactions to 20%) and the sigma-blind
rule more than doubles its forward calls as the evidence gets weaker.

Every figure was recomputed from the database rather than reused from a console
log, which caught two errors during drafting: the dGPredictor point-estimate
hard-call rate (60.5% -> 53.6%) and a claim that GC's rate "barely moves"
between sources when it drops from 58.7% to 34.9%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comparisons so far hold the energy fixed and vary the rules. --matrix does
the opposite: it holds the rule set fixed and varies which prediction supplies
the energy, scoring each source from its own thermodynamics[source] entry. Each
cell is the percentage of shared reactions where two sources give the same
direction; overlaps run from 17,021 (eQuilibrator n dGPredictor) to 25,969
(GroupContrib n dGPredictor-ModelSEED).

Average pairwise agreement across the four sources falls from 71.0% under the
default GC cascade to 58.2% under the index. That reads as the index making
things worse, and it is the opposite -- the default cascade was manufacturing
the agreement.

print_agreement_decomposition splits each pair's agreement into why they agree.
The best-agreeing pair in the default matrix is Group contribution and
dGPredictor-ModelSEED at 83.3%, and 54.1 of those 83.3 points are both sources
saying "reversible". Only 29.2% is two sources committing to the same direction.
Under the index the same pair drops to 63.9% overall while the both-committed
share *rises* to 32.9%, and outright conflicts -- one source forward, the other
backward -- go from 0.4% to 5.7%. Those conflicts were always there; the
fallback to "reversible" was hiding them.

The decomposition also reports agreement excluding "?", because under the index
eQuilibrator returns "?" for the 4,934 reactions it declined and a "?" matches
nothing, which drags its row down for a reason that is not disagreement.
Restricted to reactions where both sources made a call, eQuilibrator's pairs
rise from 68.6/51.6/55.0% to 80.4/63.3/68.4%. The three pairs involving no "?"
still drop under the index, so the finding stands -- but eQuilibrator is a
better-behaved source than its raw row suggests, once an honest refusal stops
counting as a wrong answer.

Written up as section 4 of the 2026-08-21 note; the sentinel section renumbers
to 5.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The matrix previously applied the index to all four sources, which is not a
configuration anyone would ship. Group contribution is the byte-compare anchor
and the cascade was designed around its concentration bounds; only the
prediction sources are candidates for the index.

score_all_sources now takes a per-source rule-set mapping rather than one name,
and --index-sources controls which sources take the alternate set (default: the
three prediction sources). The baseline matrix is unchanged -- default cascade
everywhere -- so the comparison is now "what ModelSEED does today" against "what
changes if only the predictions move".

Group contribution's row shifts accordingly: 68.6/51.0/63.9% under the old
all-index matrix, 62.2/54.3/61.6% now. The three prediction-source pairs are
untouched, since they were already on the index in both.

Average pairwise agreement falls 71.0% -> 57.3%, and every pair moves the same
way: agreement down, mutual "=" down, direct conflicts up. Group contribution
and dGPredictor-ModelSEED are the best-agreeing pair in the default matrix at
83.3%, and 54.1 of those points are both sources saying "reversible".

Also corrects an overgeneralisation in the previous write-up, which said the
index converts abstention into agreed-upon directions. It does not: agreed
directions fall in five of six pairs. Abstention becomes a mix of consensus and
conflict, because the two rule families commit on different reactions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@samseaver
samseaver merged commit c578290 into ModelSEED:dev Aug 29, 2026
samseaver pushed a commit that referenced this pull request Sep 4, 2026
Every thermodynamics["eQuilibrator"] value is now computed from the structure
ModelSEED holds for that compound, with no fuzzy matching anywhere in the
id -> structure -> energy chain.

The values these replace came from a MetaNetX-mediated retrieval that matched
ModelSEED compounds down a progressively loosening InChIKey ladder: full key,
then key minus the protonation layer, then connectivity alone. Falling back to
connectivity means an energy could rest on a different stereoisomer or
protonation state than the structure we actually curate. Nothing failed when it
did -- you got a well-formed energy for the wrong molecule.

Provenance of the new numbers:
  cache      eQuilibrator's Zenodo cache with every seed: accession repointed
             at our structure (Path A), then training kegg: accessions
             repointed so kegg:X and seed:cpd##### resolve to the same
             compound (Path B). 97.1% of the 28,075 compounds where we hold a
             structure resolve to it; the rest lacked pKas and keep the prior
             mapping, flagged unverified.
  parameters component-contribution retrained on that cache with TECRDB
             de-duplicated -- 78 measurement groups that eQuilibrator counted
             2-6 times, 88 redundant rows removed, improving cross-validated
             prediction at p=1.7e-05.
  conditions pH 7.0, ionic strength 0.25 M, pMg 3.0, 298.15 K.

Counting real energies rather than keys -- the old compound script wrote the
10000000 sentinel for structureless compounds, so 8,765 of its 30,607 entries
were never energies:

                before   after    lost   gained     net
  reactions     25,028  21,853   6,238    3,063  -3,175
  compounds     21,842  16,372   9,223    3,753  -5,470

The new ingests write no sentinels: an entry either carries an energy or
carries no eQuilibrator key.

Most of the reaction loss is deliberate. Of the ~6,200 reactions that had a
value and no longer do, ~4,000 were outside the component-contribution span and
~1,900 do not balance -- the old code reported a number for both without
checking the infinite-uncertainty term. Roughly 270 are a genuine cache gap.
Where old and new overlap they agree to a median 0.40 kcal/mol.

The ingest moves in-tree. Update_{Reaction,Compound}_eQuilibrator_Energies.py
previously read MetaNetX_*_Energies.tbl, so a Rerun_Thermodynamics.sh pass
would have silently reverted these values to the InChIKey-ladder numbers. They
now read ModelSEED_{Reaction,Compound}_Energies.tsv, committed alongside, and
the pipeline is reproducible from a clean checkout. The .tbl files are kept as
archived inputs -- Check_eQuilibrator_Energy_Errors.py, the two reversibility
tests, Preview_Structure_Update.py and Build_Compound_Field_Provenance.py still
read them -- but they no longer feed the database.

Two new helpers in _thermo_helpers.py: parse_modelseed_energy_table, and
run_{reaction,compound}_table_update, which REMOVE the key where a record no
longer earns an energy. run_reaction_lookup_update deliberately leaves absent
reactions untouched, which is right for an incremental refresh of one pipeline
and wrong for a regeneration -- a stale number under a current-looking key is
worse than no key.

Per-source operators are computed at write time through the eQuilibrator rule
set merged in #285, rather than deferred. Only 294 reactions land on "?", where
carrying the old flags forward would have left 3,063 of them unresolved.

Also lands, all previously outside the repo:
  eQuilibrator-2020_{reactions,compounds}.tsv  the pre-regeneration values,
                                               25,028 / 30,607 entries, as the
                                               paper's baseline
  missing_compounds.tsv                        ~4,650 compounds absent from the
                                               cache, ranked by reactions blocked
  lost_reactions_review.tsv                    15 reactions that stopped
                                               balancing under our structures

Biochemistry/Thermodynamics/eQuilibrator/README.md records the provenance and
five open review items that are NOT settled by this commit: the 19,984
compounds / 10,693 reactions outside the CC span (unaudited -- nobody has
characterised what they are), the cache-absent compounds, the 15 unbalanced
reactions, the structure checklist, and the inherited pMg 3.0 convention.

Canonical deltag/deltagerr/reversibility are untouched here. Promotion weighs
reversibility and must run once, after every source has been regenerated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
samseaver pushed a commit that referenced this pull request Sep 4, 2026
…redictions

Three changes, all to thermodynamics["dGPredictor"].

1. RETRAINED, because the model was broken.

The predecessor dGPredictor-ModelSEED record had a median uncertainty of 88.6
kJ/mol (259.5 for quinones), 44.5% of predictions past 100 kJ/mol, worst error
8,531. The cause was RDKit canonicalization drift: the Wang lab shipped a
fragment vocabulary written by an older RDKit (CC(C)=O) and current RDKit emits
CC(=O)C for the same fragment, so reindexing silently dropped most fragment
counts. BayesianRidge then extrapolated from its prior, and the prior
covariance is what those error bars were.

The fix is a guarantee rather than a patch: the vocabulary is rebuilt from
current-RDKit output, and training and prediction call the same
count_substructures, so they can no longer disagree about a fragment's name.

2. FITTED TO THE SAME MEASUREMENTS AS eQUILIBRATOR.

Training moves from the raw Zenodo TECRDB.tsv (4,544 rows) to TECRDB_dedup.csv
(4,456) -- the table component-contribution's cc_params_dedup.npz was fitted
to. The 88 removed rows are redundant copies of 78 measurement groups counted
2-6 times; one copy of each is kept, so no measurement is lost. Fitting the two
methods to different row sets of the same measurements is not defensible when
the paper compares them head to head.

Training RMSE rises 5.4 -> 5.85 kJ/mol, which is what removing rows the fit got
for free must do. The question needs held-out data, so scripts/08 runs the
comparison eQuilibrator ran for component-contribution: 10 folds, both arms
predicting an identical test set, every copy of a tested measurement withheld
from both arms so the raw arm cannot win on leakage. Over 3,633 held-out
measurements, median |error| 1.7125 -> 1.6952, closer on 53.3%, paired Wilcoxon
p=1.5e-07. Highly significant and small -- the point is that consistency with
eQuilibrator costs nothing measurable, not that de-duplication is a large win.
Per-row errors in crossval_training_sets.tsv.

3. NO COVERAGE FLOOR. Every prediction ships with its own uncertainty.

The earlier policy installed only coverage >= 0.9 (8,279 reactions), on the
reasoning that below that a prediction is prior-dominated and is "we don't
know" wearing a wide error bar. Measured against eQuilibrator on the 20,691
reactions both methods score, that does not hold: the fraction within one
combined sigma is 67-79% in EVERY coverage stratum against 68% nominal. At
coverage < 0.5 the model says +/-111 kJ/mol and is off by 87. The bars are
honest, and mildly conservative if anything.

Coverage also turned out to be the wrong axis to filter on. Spearman against
|deviation from eQuilibrator| is -0.349 for coverage but +0.650 for sigma, and
the two correlate only -0.498 with each other. The 0.9 floor was admitting
1,947 predictions with high sigma (median deviation 43.3 kJ/mol) while
rejecting 5,038 with low sigma (median deviation 12.3).

So all 29,617 reactions and 30,185 compounds are installed, and consumers
filter on sigma. Coverage is stored as provenance -- it explains why sigma is
large -- but the README says plainly that it is not the filter, and gives a
sigma-threshold table instead. DGP_COVERAGE_FLOOR reinstates a floor if one is
ever wanted; it defaults to 0.

This does not smuggle false confidence into the direction field. DGP_HEURISTICS
(the Noor 2012 reversibility index at one sigma, merged in #285) reads the wide
bars and declines: of 29,617 records, 20,723 are "=", 6,870 ">" and 2,024 "<".

Record shape gains coverage, appended so positional readers are unaffected:
  reaction   [dg, err, operator, coverage]
  compound   [dg, err, coverage]         no operator; formation energy has no direction

Both ingest scripts, and the swap, now pass source= to
reversibility_from_energy. Without it a dGPredictor energy is silently scored
with Group-Contribution rules, and the stored operator disagrees with what
Estimate_Reaction_Reversibility.py computes for the same record. This was
latent in the pre-#285 code and would have been wrong the moment #285 landed.

Coverage against the other sources, on 56,012 reactions:
  Group contribution  56,002      eQuilibrator  21,853      dGPredictor  29,617
Compounds gain dGPredictor as a third source: 30,185 records.

dGPredictor and eQuilibrator overlap on 7,247 reactions at the old floor and
20,691 without it -- that overlap is the honest denominator for any head-to-head
comparison. dGPredictor reaches 1,032 reactions eQuilibrator cannot (518
unbalanced, 384 outside the CC span), so neither method dominates.

Also lands:
  dGPredictor-2020.tsv       27,715 archived KEGG-trained predictions, the
                             paper's baseline. Read with care: 17,271 of them
                             (62%) hold an energy computed for a KEGG reaction
                             that is not among that reaction's aliases -- the
                             2023 staging notebook's stale-variable carry-over
                             (PR #286). Confirmed here independently: only
                             10,444 rows carry a matching KEGG id.
  retrained_dG*.json         staged inputs, all predictions
  crossval_training_sets.tsv per-row held-out errors, both training sets
  README.md                  provenance, the calibration evidence, and what to
                             filter on

Removed: Update_Reaction_dGPredictor_ModelSEED_Energies.py, the 61 json_files
and 61 pickle_files staging shards, and modelseed_retrained_dG.json.

07_atomic_swap.py now refuses to overwrite an existing dGPredictor-2020.tsv.
Re-running it after a swap would otherwise capture the RETRAINED values as the
"2020 baseline" and destroy the only copy of the pre-retrain record.

Canonical deltag/deltagerr/reversibility are untouched and are now stale with
respect to every source beneath them. Re-promotion is deliberately out of scope:
the field is being retired.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
samseaver pushed a commit that referenced this pull request Sep 4, 2026
Rerun_Thermodynamics.sh had drifted into three disagreeing versions across
branches. This is the reconciled one:

  * Update_Reaction_dGPredictor_ModelSEED_Energies.py is gone -- the script was
    deleted with the retrain.
  * Update_Compound_dGPredictor_Energies.py is added -- dGPredictor is now a
    compound-level source too.
  * The canonical-field steps (Estimate_Reaction_Reversibility.py GC / EQ and
    Promote_Reaction_Thermodynamics_to_Canonical.py) are documented but NOT
    run, because those fields are being retired. Leaving them silently in the
    chain would keep rewriting a field we are about to delete, from sources
    that have all moved.
  * set -euo pipefail, so a failing step stops the chain instead of letting the
    next one run against half-updated JSON.

Add_Reaction_Thermodynamics_Operators.py rebuilt each record as a fixed
[dg, dge, operator] triple, which silently truncated dGPredictor's fourth
element (coverage) on every backfill. It now preserves everything past the
operator. Verified: 107,472 entries seen, 0 refreshed, no JSON diff -- which
also confirms independently that the operators each updater wrote at write time
match what the cascade computes on its own.

Scripts/Thermodynamics/README.md documents the per-source record shapes (they
now differ by kind AND source), the source= requirement on
reversibility_from_energy, and that the canonical fields are stale and being
retired. The thirteen sections #285 added are untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants