Repository navigation
Possible underestimated pass@3 results #45
Description
I have evaluated your predictions using my Docker based swe-bench evaluator. I achieve 26% on pass@3 compared to the 22% you reported. It might be worthwhile to review the logs for the failed benchmarks to see if your agent can actually achieve even better results :D
You find the logs and report here
And here's a sheet I use to compare the results.
Activity
[-]Possible Underestimated pass@3 Results[/-][+]Possible underestimated pass@3 results[/+]zhiyufan commentedon May 13, 2024
Thank you for this detailed report!
I quickly went through the comparison sheet you shared and noticed that there are many opposite acr-2/3 results in the two evaluation environments. Maybe you mistakenly mixed up the acr-2/3 results?
aorwall commentedon May 13, 2024
You're right. Good catch. That explains why it looked like some of the benchmark instances failed in my benchmark. Now it covers all the cases you resolved.
yuntongzhang commentedon May 13, 2024
This looks great! The dockerized evaluation environment would be very useful for obtaining consistent evaluation results. We will also try out your docker-based evaluation on our machines :)