Repository navigation
Refuse gapped input to feature extraction; register the flux extractors - #603
Draft
DhavalRepo18 wants to merge 10 commits into
Draft
DhavalRepo18 wants to merge 10 commits into
DhavalRepo18 wants to merge 10 commits into
Conversation
extract_features and select_features passed gapped series straight to the extractors and replaced whatever came back NaN with 0.0 (nan_to_num), so 38 extractors reported 0.0 for any series with a missing point. Both tools now take impute=interpolate|drop|zero, reusing the forecast path's _impute, and return ErrorResult on gapped data when impute is unset. A value an extractor still cannot compute is returned as null, never 0.0. FS.select_features refuses non-finite input directly. The six flux_percentile extractors called np.percentile(interpolation=), which NumPy 2 rejects. build_extractors drops any function that raises at import, so they vanished from the registry while the catalog still listed them. interpolation= -> method=. smoke_features.py now checks the gap guard on the tool path rather than failing every extractor that cannot take NaN itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ufJDmdznCvjWDB7PUk41W
extract_features and select_features passed gapped series straight to the extractors and replaced whatever came back NaN with 0.0 (nan_to_num), so 38 extractors reported 0.0 for any series with a missing point. Both tools now take impute=interpolate|drop|zero, reusing the forecast path's _impute, and return ErrorResult on gapped data when impute is unset. A value an extractor still cannot compute is returned as null, never 0.0. FS.select_features refuses non-finite input directly. The six flux_percentile extractors called np.percentile(interpolation=), which NumPy 2 rejects. build_extractors drops any function that raises at import, so they vanished from the registry while the catalog still listed them. interpolation= -> method=. smoke_features.py now checks the gap guard on the tool path rather than failing every extractor that cannot take NaN itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ufJDmdznCvjWDB7PUk41W
…nt impute
- gate_gaps refuses a channel with no values at all. _impute('interpolate')
ends in fillna(0.0), so an all-missing channel would have become zeros.
- smoke_features.py prints which code it ran: the commit baked into the
runtime image (/opt/aob/.aob-commit), or git HEAD and whether the tree
is dirty. Drops an except branch no current code can reach.
- docs/mcp-servers.md lists impute on extract_features and select_features.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011ufJDmdznCvjWDB7PUk41W
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
The merge in 5c35041 resolved three files toward an earlier copy of the gap-refusal commit (1b43b7d, from a local aafeedback_changes), which undid part of ea7a278: - gate_gaps lost its refusal of an all-missing channel, so impute='interpolate' could again turn such a channel into zeros - the test for that refusal was deleted - smoke_features.py got back an except branch no current code can reach This restores those three files to their ea7a278 content. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ufJDmdznCvjWDB7PUk41W
…eatures Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
smoke_features.py calls the extractors directly. An agent never does: it reads list_features and recipe_template, then sends JSON to extract_features, select_features, run_tabular_recipe and run_recipe. This script sends that JSON through FastMCP's call_tool, against an in-memory store seeded from --catalog, and holds every answer to one contract: do what was asked, or return an error that names the cause. A success that silently did something else is a FAIL. Exit 1 on any FAIL. On the current branch 5 cases fail: unknown extractor names dropped or reported only by sklearn, stored transform cards ignored by run_tabular_recipe and unusable in run_recipe, and gapped instances zero-filled on the tabular path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ufJDmdznCvjWDB7PUk41W
…ances
smoke_agent_features.py found five ways a recipe returned success, or an
error naming nothing, for a request it did not carry out. All five now get
an error that names the cause, checked before the engine runs:
- _check_recipe validates recipe['transforms'] against the shapes each path
applies: run_recipe and evaluate take {sktime_class}; run_tabular_recipe
takes {extractors}, {flops_select} or {sktime_class}. Unknown extractor
names are named (they were dropped silently, or surfaced as sklearn's
"Found array with 0 feature(s)"). A {feature_id} spec is refused by card
name: run_tabular_recipe ignored it and ran the whole library, run_recipe
failed with "card 'None' has no sktime_class". Applying stored cards would
exec their code, which needs a sandbox first.
- run_tabular_recipe no longer fills NaN with 0.0. Gapped instances are
refused unless recipe['impute'] is interpolate, drop or zero
(composition.gate_gap_rows, reusing gate_gaps per instance).
- recipe_template documents the accepted transform shapes and impute.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011ufJDmdznCvjWDB7PUk41W
…tools
smoke_forecast.py builds each estimator directly. An agent calls
list_models / find_models / describe_models / resolve_model, then
run_recipe with {"estimator": {"model_id": ...}, "fh": [...]}. This script
makes those calls through FastMCP's call_tool against an in-memory store
seeded from --model-catalog, and holds every answer to the contract the
feature smoke test uses: do what was asked, or return an error naming the
cause. For runs that succeed it also checks a finite score, skill against
naive on the same holdout, that a zero-shot card runs zero-shot, that
resolve_model's "resolvable" means the run works, gapped input, and the
card's own prediction_length.
Naive is computed on exactly the holdout run_recipe scores: zero-shot
recipes are scored on one final holdout, fitted recipes over many
expanding folds, so their backtest_score values are not comparable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011ufJDmdznCvjWDB7PUk41W
…imit The first run on the private catalog printed nothing after discovery and was killed: rows were printed only after every model finished, and one model could hold the sweep indefinitely. A card that is not pinned zero-shot backtests over about 60 expanding-window refits at fh=24, and over stdio Stirrup sets no tool-call timeout, so an agent's run_recipe would block the same way. Each model now runs in a subprocess with --timeout (default 180 s; twice that for the gaps/horizon stage). A model that does not answer in time is reported as FAIL, rows print as each model finishes, and zero-shot cards run first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ufJDmdznCvjWDB7PUk41W
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
extract_features and select_features passed gapped series straight to the
extractors and replaced whatever came back NaN with 0.0 (nan_to_num), so
38 extractors reported 0.0 for any series with a missing point. Both tools
now take impute=interpolate|drop|zero, reusing the forecast path's _impute,
and return ErrorResult on gapped data when impute is unset. A value an
extractor still cannot compute is returned as null, never 0.0.
FS.select_features refuses non-finite input directly.
The six flux_percentile extractors called np.percentile(interpolation=),
which NumPy 2 rejects. build_extractors drops any function that raises at
import, so they vanished from the registry while the catalog still listed
them. interpolation= -> method=.
smoke_features.py now checks the gap guard on the tool path rather than
failing every extractor that cannot take NaN itself.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_011ufJDmdznCvjWDB7PUk41W
Description
Fix Details
Impact on Benchmarking
Related Issues
Verification Steps
uv run pytest tests/integrationChecklist