Skip to content

Backtest correctness, deterministic tests and versioned JSON reports - #7

Draft
heykav wants to merge 4 commits into
claude/hardeningfrom
claude/enhance
Draft

heykav wants to merge 4 commits into
claude/hardeningfrom
claude/enhance

Conversation

@heykav

@heykav heykav commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Backtests can now export an auditable JSON report with --report PATH: parameters, strategy/CSV SHA-256 hashes, metrics, equity curve, fills and runtime versions. Non-finite values become null. Atomic writes protect strategy, CSV and database files (including aliases); changed inputs are detected. Yahoo data and imported dependencies are not snapshotted, so this is not a full reproducibility guarantee.

Validation for the current addition: 143 tests passed on Python 3.12; Ruff lint/format and mypy passed. Added tests cover deterministic exports, independent fill/equity reconciliation, changed hashes, protected files, atomic failure and optional CLI behavior. OmniRoute generated the initial report scaffold; it was completed and independently reviewed before commit.

Existing changes in this stacked PR:

Stacked on #6 (claude/hardening). This is still backtesting only. There is no live or paper-trading engine, and the docs still say so.

Defects fixed (each has a regression test that fails on the old code)

  • Float-noise oversell left a tiny short position. Shorting was disabled, but a position of 725781.9604 shares, sold as 722027.273 and then 3754.6874, ended at about -6e-11 shares. After that, position == 0 was never true again. Now the broker sells exactly the shares held, and rounding dust is set to 0.
  • A tiny sell with no position was filled. For example sell(1e-12) with no position went through. It is now rejected.
  • Phantom trades. In compute_trade_pnls, buying 0.3 and then selling 0.1 three times left about 3e-17 shares as an open lot. The next round trip then counted an extra "winning" trade, which inflated num_trades and win rate.
  • CAGR OverflowError on short samples. For example 100x in one daily bar. It now reports inf.
  • Bars with the wrong symbol. A feed that returned bars labelled with a different symbol left every order pending forever, with no error. This now raises ValueError.
  • Yahoo end date was exclusive, but the README says --end is the last day included. YFinanceFeed now asks for one extra day and trims. Both feeds treat start/end as inclusive, and in CSVDataFeed a date-only end now includes that day's intraday bars.
  • Bad CSV input gave unclear pandas errors (AttributeError/TypeError). It now raises CSVFormatError (a ValueError), naming the file, line and column. The CLI prints it and exits with code 1, with no traceback.
  • periods_per_year <= 0 or a non-finite risk_free_rate now raise ValueError.

Tests added

  • Look-ahead invariance: replace every bar after k with other prices; the orders, fills and equity up to k must stay the same. I checked that this test fails when the engine is changed to pass the next bar to on_bar.
  • Fills ignore their own bar's close: changing that bar's close, high and low does not change the fill.
  • Edge cases: gap fills, a short that takes equity below zero (treated as ruin), a long-to-short flip, and an open position at the end of the data (marked to market, not liquidated).
  • Hand-derived metrics with exact expected values: Sharpe 1, Sortino 3, volatility 15%.
  • Determinism: repeated engine runs, a CSV feed against an in-memory feed, and two CLI runs into the same SQLite file all give identical results.
  • The web demo's copy of the engine (docs/quantdeck_src) must match src/ and must run without pandas.

Docs and visuals

  • The equity chart now has a separate drawdown panel. Its maximum is checked against compute_metrics.
  • Fixed the y-axis tick labels: a tick at 102.5k used to read "$102k".
  • The README and tutorial now describe: shorts have no margin, positions are not liquidated at the end, date ranges are inclusive, CSV errors, and how metrics are annualised. The README also says plainly that past results do not predict future results.

Behaviour changes, all covered by tests: the fixes above, and the CSVDataFeed date-only end change. The pinned README/CLI numbers have not changed.

Earlier numerical-fix validation: pytest (130 passed), ruff check, ruff format --check and mypy --strict are all clean. This was run with Python 3.11 and pandas 3.0.6; CI covers 3.10 to 3.12.


claude and others added 4 commits September 30, 2026 03:50
…atch

Engine/broker:
- A sell that overshoots the position only by float rounding now sells
  exactly the held quantity instead of leaving a microscopic short with
  shorting disabled; rounding dust after a fill is settled to zero.
- A sell of float-noise size with no position is rejected instead of
  being filled.
- Bars labelled with a symbol other than the engine's now raise instead
  of leaving every order pending forever.

Metrics:
- compute_trade_pnls ignores float-residue lots, which used to book an
  extra (winning) round trip after fractional partial closes.
- CAGR that overflows a float on very short samples is reported as inf
  instead of raising OverflowError.
- periods_per_year <= 0 and non-finite risk_free_rate raise ValueError.

Tests: look-ahead invariance (rewrite the future, the past must not
change; verified to fail when on_bar is handed the next bar), gaps,
shorts past zero equity, long/short flip, end-of-data marking, hand-
derived Sharpe/Sortino/volatility, and a check that the web demo's copy
of the engine matches src/ and runs without pandas.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A42wbaQJVk97N5TsWqXP4G
- CSVDataFeed raises CSVFormatError (a ValueError) naming the file and
  line for missing columns, unparseable timestamps and blank or
  non-numeric values, instead of AttributeError/TypeError from pandas.
- Header names are matched case-insensitively; timezone-aware
  timestamps work with naive start/end; sorting is stable.
- start/end are inclusive in both feeds. YFinanceFeed used to pass end
  straight to Yahoo, where it is exclusive, so the last requested day
  was dropped; it now requests one more day and trims. A date-only end
  includes intraday bars on that day in CSVDataFeed too.
- The CLI prints bad-data errors without a traceback and exits 1.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A42wbaQJVk97N5TsWqXP4G
- Equity figure gets a separate underwater drawdown panel (asserted to
  agree with compute_metrics' max drawdown) and exact $k tick labels
  (ticks at 102.5k were labelled "$102k").
- tests/test_determinism.py: repeated engine runs, CSV vs in-memory
  feeds and two CLI runs into one SQLite file are identical.
- README and tutorial: margin-free shorts, no end-of-data liquidation,
  inclusive dates, CSV errors, annualisation conventions, and an
  explicit "past results do not predict future results".

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A42wbaQJVk97N5TsWqXP4G
@heykav heykav changed the title Engine and metrics correctness fixes, clear CSV errors, determinism tests Backtest correctness, deterministic tests and versioned JSON reports Oct 4, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants