how to interpret results for Auto Code Rover SWE-bench? #47

Activity

1. ramsey-coding commented on May 14, 2024

   

   ramsey-coding

   on May 14, 2024

   Author

   More actions

   what's the difference between the following fields?

           "generated": 249,
           "with_logs": 249,
           "applied": 245,
           "resolved": 48
   
2. zhiyufan commented on May 16, 2024

   

   zhiyufan

   on May 16, 2024

   Contributor

   More actions

   cost_X_Y: X is the budget cost of running swe-agent in our experiment, and Y is the trail of repetition.
   In this case, we used a budget of 2 USD, and repeated the experiment 3 times.
   Inside the cost_X_Y directory, *.traj files are the conversation log files for each task instance in swe-bench.
   all_pred.jsonl includes all the generated patches.

   For AutoCodeRover acr-run-1, acr-run-2, and acr-run-3 results align with Table-3, In our environment, the ACR column.

   • generated: there is an agent-generated patch for this issue
   • with_logs: a log file is produced when executing the passing/failing test-cases of this issue
   • applied: the patch can be applied successfully to the original program.
   • resolved: the patch made the passing/failing test-cases of this issue pass

   The details on how the stats are generated can be found here: https://github.com/yuntongzhang/SWE-bench/blob/main/metrics/report.py#L264C5-L264C21