Skip to content

refactor(add): swap harness to accept project schema directly instead of api shape. - #2034

Merged
notgitika merged 6 commits into
aws:refactorfrom
Hweinstock:refactor-add-harness-flags
Aug 20, 2026
Merged

refactor(add): swap harness to accept project schema directly instead of api shape. #2034
notgitika merged 6 commits into
aws:refactorfrom
Hweinstock:refactor-add-harness-flags

Conversation

@Hweinstock

@Hweinstock Hweinstock commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Problem

The existing convention of mirroring API shapes has some issues.

  • evaluators experience diverges significantly from the APIs. (ex. this single API is responsible for multiple resources in the project).
  • for customers looking at their agentcore.json, its not obvious what flags to pass.
  • complexity in rejecting unsupported API fields.
  • does not extend well to supporting update/edit in the future.

Solution

swap to mirroring the schemas directly. This approach is simpler (removes ~300 lines of mapping code from harness) and will extend better to agentcore project resources that have diverged from the API.

mirroring the schemas with json blobs also reduces the amount of flags significantly.

The PR also ports the harness tests to their own file.

Testing

Existing Unit Tests ported over.

Helping screen becomes:

Usage: agentcore project add harness [options]

adds a harness to the current project

Options:
  --name <name>                                          the name of the harness
  --execution-role-arn <execution-role-arn>              IAM role the harness assumes; a default role is created when omitted
  --system-prompt <system-prompt>                        the agent's system prompt
  --model <model>                                        model configuration (JSON)
  --tools <tools>                                        tools available to the agent (JSON)
  --skills <skills>                                      skills available to the agent (JSON)
  --allowed-tools <allowed-tools...>                     tool allowlist patterns (e.g. * or @serverName/toolName)
  --memory <memory>                                      memory configuration (JSON)
  --truncation <truncation>                              context truncation configuration (JSON)
  --network-mode <network-mode>                          network mode for the harness environment (PUBLIC or VPC)
  --network-config <network-config>                      VPC network configuration (JSON)
  --lifecycle-config <lifecycle-config>                  lifecycle configuration (JSON)
  --session-storage-path <session-storage-path>          mount path for session storage
  --efs-access-points <efs-access-points>                EFS access point configurations (JSON)
  --s3-access-points <s3-access-points>                  S3 access point configurations (JSON)
  --environment-variables <environment-variables>        environment variables (JSON object of key/value strings)
  --container-uri <container-uri>                        ECR container image URI
  --authorizer-type <authorizer-type>                    inbound authorizer type (AWS_IAM or CUSTOM_JWT)
  --authorizer-configuration <authorizer-configuration>  inbound authorizer configuration (JSON)
  --max-iterations <max-iterations>                      max agent loop iterations per invocation
  --max-tokens <max-tokens>                              max total output tokens per invocation
  --timeout-seconds <timeout-seconds>                    max duration in seconds per invocation
  --tags <tags>                                          tags to apply (JSON object of key/value strings)
  --dockerfile <dockerfile>                              path to local dockerfile to use as the container image for the harness
  -h, --help                                             display help for command

Notes

This does introduce a discoverability gap where customers need to know our project schemas, which means we'll need solid documentation here.

@github-actions github-actions Bot added the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 19, 2026
@codecov-commenter

codecov-commenter commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.09%. Comparing base (33e2d2f) to head (2af765c).

Additional details and impacted files
@@             Coverage Diff              @@
##           refactor    #2034      +/-   ##
============================================
- Coverage     97.14%   97.09%   -0.05%     
============================================
  Files           382      382              
  Lines         22884    22581     -303     
============================================
- Hits          22231    21926     -305     
- Misses          653      655       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 19, 2026
@Hweinstock
Hweinstock marked this pull request as ready for review August 19, 2026 13:50
jariy17
jariy17 previously approved these changes Aug 19, 2026
? toModelConfig(inputModelConfig)
: { provider: "bedrock" as const, modelId: "global.anthropic.claude-sonnet-4-6" },
model: parseJsonFlag("model", flags["model"]) ?? {
provider: "bedrock" as const,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Make it into a CONST

efsAccessPoints: env?.efsAccessPoints,
s3AccessPoints: env?.s3AccessPoints,
containerUri: artifact?.containerUri,
tags: parseJsonFlag("tags", flags["tags"]),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Tags differ from the shared convention

This command currently accepts only:

  --tags '{"team":"ml","env":"prod"}'

Other refactor-branch commands use parseTags, allowing both:

  --tags team=ml env=prod
  --tags '{"team":"ml","env":"prod"}'

They declare the flag as z.array(z.string()).optional() and convert the values into Record<string, string> with parseTags.

Should we support both? Even though the flags don't match in the imperative one, we should keep the tag conventions the same between both.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think to be consistent, we should swap it.

notgitika
notgitika previously approved these changes Aug 19, 2026

@notgitika notgitika left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OOS for here but when I was playing around with it I noticed it didn't have a parameterized help like we see in harness create --help. The JSON-valued flags are otherwise difficult to construct.

the help should document the harness spec shapes specifically, since they differ from the API shapes used by harness create.

can we discuss having that as a future improvement?

flag("max-tokens", "max total output tokens per invocation", z.number().optional()),
flag("timeout-seconds", "max duration in seconds per invocation", z.number().optional()),
flag("tags", "tags to apply (JSON object of key/value strings)", z.string().optional()),
flag("tags", "tags as key=value (repeatable) or JSON object", z.array(z.string()).optional()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add --connections here? HarnessSpecSchema supports it but this handler currently doesn't

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICT this isn't used in the old CLI so I left it out.

@notgitika
notgitika merged commit eaf92f0 into aws:refactor Aug 20, 2026
8 of 11 checks passed
notgitika added a commit to notgitika/agentcore-cli that referenced this pull request Aug 21, 2026
Upstream moved the per-resource `project add` tests out of the monolithic
project.test.ts into colocated add/<resource>/index.test.ts suites (harness
in aws#2034, online-eval in aws#2048). Move the memory tests to match, with the
same locally-duplicated run/inProject helpers those suites use.

project.test.ts is now identical to upstream/refactor again, so this PR no
longer touches it. Also drops the DeserializationError, FsReadWriteJson and
ReadWriteJson imports, left dead there once the harness tests that used them
moved to add/harness/index.test.ts.

No test content changed: 187 project tests still pass, now across 10 files
instead of 9.
notgitika added a commit to notgitika/agentcore-cli that referenced this pull request Aug 21, 2026
Upstream moved the per-resource `project add` tests out of the monolithic
project.test.ts into colocated add/<resource>/index.test.ts suites (harness
in aws#2034, online-eval in aws#2048). Move the memory tests to match, with the
same locally-duplicated run/inProject helpers those suites use.

project.test.ts is now identical to upstream/refactor again, so this PR no
longer touches it. Also drops the DeserializationError, FsReadWriteJson and
ReadWriteJson imports, left dead there once the harness tests that used them
moved to add/harness/index.test.ts.

No test content changed: 187 project tests still pass, now across 10 files
instead of 9.
jariy17 pushed a commit that referenced this pull request Aug 24, 2026
* feat(project): add `project add memory`

Registers a `memory` leaf under `project add`, following the same
SDK-union -> flat project-schema conversion pattern as `project add
harness`. A memory scaffolds no files, so the command only appends an
entry to `spec.memories` in agentcore.json; the L3 CDK turns that into
an `AWS::BedrockAgentCore::Memory` at deploy time.

Flags: --name, --event-expiry-duration, --strategies, --indexed-keys,
--stream-delivery-resources, --encryption-key-arn, --execution-role-arn,
--tags.

--strategies accepts two forms: a comma-separated list of strategy types
expanded with the CLI's default namespace templates, or a JSON
MemoryStrategyInput[] mirroring the CreateMemory API for strategies that
need explicit names, descriptions, or namespaces.

clientToken is excluded (it is CreateMemory idempotency and this command
makes no API call), and description is excluded until the L3 CDK schema
supports it.

* feat: add --description to 'project add memory'

Stores an optional memory description in agentcore.json, matching the
CreateMemory API's description field (max 4096 characters).

The generated CDK app pins @aws/agentcore-cdk 0.1.0-alpha.45, whose
MemorySchema is a non-strict z.object with no description field, so the key
is stripped at synth rather than rejected until
aws/agentcore-l3-cdk-constructs#325 ships and that pin is bumped. The flag
help text says so.

* feat: accept a CUSTOM memory strategy in 'project add memory'

The CDK's memory schema already models CUSTOM (@aws/agentcore-cdk
0.1.0-alpha.45 maps it to CFN customMemoryStrategy), so the CLI's four-type
enum was the outlier. A customMemoryStrategy in the --strategies JSON now
converts to { type: 'CUSTOM', name, description, namespaceTemplates }.

The shorthand form still takes managed types only: CUSTOM has no default
namespaces to expand. An extraction configuration or memoryRecordSchema is
rejected rather than dropped, since the CDK schema carries neither.

Also names the offending field in the memory validation error.

* revert: drop CUSTOM memory strategy from "project add memory"

Reverts 87be86e. I added CUSTOM because the CDK schema already had it in
MemoryStrategyTypeSchema, which turns out to be the argument PR #694 made --
and #713 reverted a day later.

The CLI has removed CUSTOM twice on purpose. Offering the type without
somewhere to put its extraction configuration is #241 ("select custom memory
strategy, note there is no option to add prompts"); #266 removed it as a P0 to
stop users picking an unsupported option, #694/#696 added it back with
semanticOverride, and #713 reverted both as premature. #676 tracks doing it
properly. The CDK keeping CUSTOM in its enum without a configuration field is
the same hole, not a licence.

So both forms are rejected again, now with an error that says why and points
at #676. The one thing kept from the reverted commit: memory validation errors
name the offending field, since issue.path was being dropped.

* refactor: drop the long-form help for --description

The one-line flag description is enough; the deploy-time caveat lives in the
PR discussion rather than in help output.

* docs: comment change

* fix: change function name and add comment for clarity

* test: add uncovered unsupported stream content type test

* style: make json example concrete, remove comments

* refactor(project): reuse shared spec validation for memory

* fix(project): validate memory JSON inputs

* fix(project): harden memory input validation

* test(project): colocate memory tests with the add/memory handler

Upstream moved the per-resource `project add` tests out of the monolithic
project.test.ts into colocated add/<resource>/index.test.ts suites (harness
in #2034, online-eval in #2048). Move the memory tests to match, with the
same locally-duplicated run/inProject helpers those suites use.

project.test.ts is now identical to upstream/refactor again, so this PR no
longer touches it. Also drops the DeserializationError, FsReadWriteJson and
ReadWriteJson imports, left dead there once the harness tests that used them
moved to add/harness/index.test.ts.

No test content changed: 187 project tests still pass, now across 10 files
instead of 9.

* refactor(project): match --strategies JSON to the agentcore.json schema

The --strategies flag re-declared its own strategy input schema, modelled
on the CreateMemory API's tagged union (semanticMemoryStrategy et al.) and
requiring a name. agentcore.json stores strategies flat with an optional
name, so the flag accepted a shape the project file never holds and
rejected one it does.

Parse the JSON form with MemoryStrategySchema itself, wrapped only for the
unsupported-field diagnostics, so the flag cannot drift from the schema.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants