Skip to content

Multi-turn runs should not report success until the coding agent has completed #679

Open

Description

Original author: Garry Trinder (@garrytrinder)

Image

Source: http://scope-int.westus3.cloudapp.azure.com/runs/29ffc362-f6db-47c7-bf00-74928ef84c65/turns

This is a multi-turn run, set to the default 10 iterations, however the run is stopped after iteration 1 because all judging criteria has passed, unfortunately this is premature as the coding agent has not finished, as we can see by the agent response which is asking the user whether they want to continue or not.

{"type":"confirmation","kind":"confirmation","title":"Continue to iterate?","data":{"copilotRequestedRoundLimit":75}}  

After inspecting the iteration archive we can see that the workspace is incomplete. Running dotnet build results in an error due to a hallucinated dependency version for Microsoft.AspNetCore.RateLimiting package.

❯ dotnet build
    /Users/garrytrinder/Downloads/29ffc362-f6db-47c7-bf00-74928ef84c65/iteration-1/src/ECommerce.Api/ECommerce.Api.csproj : error NU1102: 
      Unable to find package Microsoft.AspNetCore.RateLimiting with version (>= 8.0.14)
        - Found 6 version(s) in nuget.org [ Nearest version: 7.0.0-rc.2.22476.2 ]

Restore failed with 1 error(s) in 4.9s

We also see that the API project contains no code

Image

The run should not be stopped on the basis that the criteria has been passed, but should continue until the agent completes. In this case, we report a false positive, yes the criteria has passed but the solution is incomplete, given a new follow up prompt the coding agent could make changes that invalidates the results of the first iteration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: workerCoding agent worker runtime, task execution, and lifecycle.type: bugIncorrect behavior or a regression that needs fixing.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions