Repository navigation
Extend --magnitude to support specific metrics #2
Description
Users report wanting to filter different metrics at different magnitudes (e.g. p99 vs p50). However, the --magnitude argument filters across all metrics.
One option could be to allow setting multiple metric=min_magnitude pairs with the --magnitude flag, or exclusively a single float. This would be pretty verbose... Ex:
hunter ... --magnitude p50=0.05 --magnitude p99=0.2Another option would be setting the min_magnitude in the config for each test's metric, with support for inheritance so each test doesn't need to define it. This option should also support overriding the config.
templates:
common_metrics:
metrics:
p50:
scale: 1.0e-6
direction: -1
min_magnitude: 0.05
p99:
scale: 1.0e-6
direction: -1
min_magnitude: 0.2
tests:
local2:
type: csv
file: tests/resources/sample.csv
time_column: time
metrics:
m1:
column: metric1
direction: 1
m2:
column: metric2
direction: -1
min_magnitude: 0.99 # Be able to set min_magnitude for specific tests
attributes: [ commit ]
remote1:
inherit: [common_metrics] # min_magnitude set via inheritance
prefix: "performance_regressions.my_product.%{BRANCH}.test1"
Activity
henrikingo commentedon Feb 12, 2025
I just want to state for the record, that one of the Nyrkiö 2024 patches contains a fix that improved the accuracy of Hunter for finding small changes and in my experience enabled going more in the direction where the user can rather use a really small p-value (0.0001) and then set the min threshold to zero or really small too. If other users have a similar experience (once they actually upgrade to what is now the head of the master branch, then I would not invest in making the min threshold more complicated.