Skip to content

fix(scenario): warn when selected attack techniques have no registered factory - #2466

Open
fei (feiiiiii5) wants to merge 1 commit into
microsoft:mainfrom
feiiiiii5:issue2461/silent-technique-drop
Open

fix(scenario): warn when selected attack techniques have no registered factory#2466
fei (feiiiiii5) wants to merge 1 commit into
microsoft:mainfrom
feiiiiii5:issue2461/silent-technique-drop

Conversation

@feiiiiii5

Copy link
Copy Markdown
Contributor

Description

resolve_technique_factories() in pyrit/scenario/core/matrix_atomic_attack_builder.py silently dropped selected techniques whose factory is not registered — the docstring even documented the behavior as "silently dropped". When a scenario (or a user of extra_factories) selects a technique that was never registered, e.g. due to a typo or a custom technique added under a different name, the run quietly proceeds with fewer attacks and there is no signal anywhere that selections were discarded.

This PR emits a single warning naming every missing technique, then proceeds with the remaining ones as before:

WARNING  pyrit.scenario.core.matrix_atomic_attack_builder:matrix_atomic_attack_builder.py:171 Skipping 2 selected attack technique(s) with no registered factory: missing_a, missing_b. Register the technique(s) (or pass them via extra_factories) to include them in the run.

Design notes:

  • Warning, not an exception: dropping is intentional fallback behavior (per the original docstring), and callers may legitimately probe which techniques are available; this only makes it observable. Happy to switch to raising if maintainers prefer strictness.
  • One aggregated warning per call, names deduplicated but kept in selection order, so repeated selections don't spam the log.

Fixes #2461

Tests

Added three regression tests to TestResolveTechniqueFactories:

  • warning is emitted when a selected technique has no factory, and names it
  • no warning when all selected techniques resolve
  • one warning listing each distinct missing name once, in selection order (deduplicated)

All 35 tests in tests/unit/scenario/core/test_matrix_atomic_attack_builder.py pass locally (73 failed / 569 passed elsewhere in tests/unit/scenario/ is identical to the pre-change baseline on this machine — those failures are environment-only, unrelated modules). Ruff check + format pass on both touched files.

…d factory

resolve_technique_factories silently dropped techniques whose factory was
not registered, so a typo or an unregistered custom technique shrank the
run without any signal. Emit one warning naming the missing technique(s)
in selection order (deduplicated), and update the docstring accordingly.

Fixes microsoft#2461
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

resolve_technique_factories silently drops selected techniques with no registered factory

1 participant