Skip to content

Trace: end-to-end test matrix for agent harnesses #3635

Closed

Description

@jiparis

Chainloop Trace supports several coding agents (Claude Code, Cursor, OpenCode, and Pi soon). Each agent supports a different set of Trace features. The trace README lists them in a feature table.

Nothing checks this table against the real agents. An agent upgrade or a model change can break a feature, for example the welcome message or spec capture, and nobody sees it until a user reports it.

Add an end-to-end test framework that runs each supported agent headless with real models, checks each feature the table marks as supported, and compares the result matrix with the expected one. A periodic job runs the suite and reports when a cell changes from passing to failing. Features the table marks as not supported are not tested.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions