You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
usersim eval scores a repeated trajectory_id once per row, silently #40
One trajectory_id stands for one conversation. A writer that keeps its own ids, such as an extension
running a multi-session benchmark, can add a second row for a conversation, for example after
restarting a crashed session. Nothing flags it:
usersim eval evaluates every row of the run (cli/evaluate.py), so that conversation counts
twice in the scores.
usersim select keeps both rows of one run, each joined to the first row's evaluation
(selection/engine.py:354), and across several runs keeps the first row per id
(selection/io.py:104), which may be the crashed attempt.
What would you like to happen?
The smaller scope agreed when this was first raised:
usersim eval warns when a trajectory_id appears more than once in the run it evaluates, naming
the ids. Rows are still evaluated as they are.
The docs state the convention: an id is derived from what the conversation stands for (for a
multi-session benchmark: member, arc and session), not from the attempt, and a restarted
conversation's row replaces the earlier one.
Activity