Skip to content

usersim eval scores a repeated trajectory_id once per row, silently #40

Closed

Description

@isarasua

What problem are you trying to solve?

One trajectory_id stands for one conversation. A writer that keeps its own ids, such as an extension
running a multi-session benchmark, can add a second row for a conversation, for example after
restarting a crashed session. Nothing flags it:

  • usersim eval evaluates every row of the run (cli/evaluate.py), so that conversation counts
    twice in the scores.
  • usersim select keeps both rows of one run, each joined to the first row's evaluation
    (selection/engine.py:354), and across several runs keeps the first row per id
    (selection/io.py:104), which may be the crashed attempt.

What would you like to happen?

The smaller scope agreed when this was first raised:

  • usersim eval warns when a trajectory_id appears more than once in the run it evaluates, naming
    the ids. Rows are still evaluated as they are.
  • The docs state the convention: an id is derived from what the conversation stands for (for a
    multi-session benchmark: member, arc and session), not from the attempt, and a restarted
    conversation's row replaces the earlier one.

Which part of the project?

The evaluator CLI and the output docs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions