[PMerlet]

fixes PRD-1670

Adds @forestadmin/datasource-dynamodb: a datasource for Amazon DynamoDB with the capabilities of the Cosmos DB one (introspection, manual schema, composite keys, relations, search, aggregation, charts, arrays as collections, native queries) plus single-table design. DynamoDB cannot sort, count or aggregate server-side, so the datasource reads by key whenever a filter allows it and does the rest on the items it reads; scanPolicy (warn by default) and maxScannedItems keep a request that no key serves from reading a whole table unnoticed.

What changes

1. Before: a DynamoDB table could not be used as a datasource. After: createDynamoDBDataSource({ region }) introspects the tables (keys and indexes from DescribeTable, attributes from a sample), or takes a manual schema; the partition and sort keys are the composite primary key.
2. Before: n/a. After: each request takes the cheapest path: Query on the table or an index when an equality pins a partition key, BatchGetItem when the filter names whole keys (what relation emulation sends), parallel Scan otherwise. Operators DynamoDB lacks (ILike, EndsWith, date intervals), sorts on non-key attributes, counts, aggregations and charts are computed on the items read, streamed so memory stays flat.
3. Before: one collection per table. After: several collections can share a table by sort key prefix or discriminator attribute, and an array of maps can be exposed as a collection keyed by its parent and its index.
4. Before: the CI knows no DynamoDB. After: datasource-dynamodb joins the test matrix, whose job starts DynamoDB Local (docker-compose.yml in the package) and sets DYNAMODB_ENDPOINT for that package only.

Tests

Unit tests use a fake client and assert the requests sent. An integration suite (22 tests, skipped without DYNAMODB_ENDPOINT) runs against DynamoDB Local, since a fake cannot tell whether DynamoDB accepts an expression: it is what validates the key-attribute and sort-key rules of Query filters, the nested array writes and the fan-out of an In on a partition key. A real agent over seeded DynamoDB Local data was also exercised by hand (list, count, paging, search, relations, chart, writes).

Not delivered, per the ticket's own list of additions: collections backed by a secondary index, and optimistic updates.

Decisions

• PRD-1670: 8 functional, 2 architectural · 7 unticked

Definition of Done

General

• Write an explicit title for the Pull Request, following Conventional Commits specification
• Test manually the implemented changes
• Validate the code quality (indentation, syntax, style, simplicity, readability)

Security

• Consider the security impact of the changes made

Squash message

feat(datasource-dynamodb): add a DynamoDB datasource

DynamoDB reads fast by key and scans otherwise, and cannot sort, count or aggregate on the server. The datasource reads by key whenever a filter allows it (Query, a Query per partition of an In, BatchGetItem, parallel Scan otherwise) and does the rest on the items it reads. It introspects tables or takes a manual schema, exposes the keys as a composite primary key, shares a table between collections, exposes arrays of maps (nested too) as collections, and runs PartiQL SELECT native queries. scanPolicy and maxScannedItems keep a request that no key serves from reading a table unnoticed.

🤖 Generated with Claude Code

Note

Add a DynamoDB datasource package with introspection and virtual array collections

• Adds a new datasource-dynamodb package with createDynamoDBDataSource, supporting manual schemas, builder-based table registration, and live introspection of tables and indexes (index.ts).
• Reads choose between Query, Scan, secondary indexes, and BatchGetItem based on filter shape, with key-type coercion and partition-key In fan-out across parallel queries (access-path.ts).
• Includes a StreamingAggregator for count/sum/avg/min/max with timezone-aware date grouping, a pagination-token cache with bounded memory and TTL, and scan limits via maxScannedItems and maxOffset (executor.ts).
• Supports single-table entity discriminators, virtual array collections exposed as nested reads, and read-only PartiQL native queries with parameterization and page/item limits (datasource.ts).
• Ships DynamoDB Local docker-compose setup, a CI matrix entry, and extensive unit plus integration test suites.

^{Macroscope summarized 519b61f.}

[linear-code]

PRD-1670

[qltysh]

70 new issues

Tool	Category	Rule	Count
qlty	Structure	Function with high complexity (count = 28): readRecords	24
qlty	Structure	Function with many parameters (count = 4): aggregate	14
qlty	Structure	Function with many returns (count = 5): computeAggregation	14
qlty	Structure	Deeply nested control flow (level = 4)	8
qlty	Structure	Complex binary expression	6
qlty	Structure	High total complexity (count = 129)	4

[qltysh]

Coverage Impact

This PR will not change total coverage.

Modified Files with Diff Coverage (23)

Rating	File	% Diff	Uncovered Line #s
	packages/datasource-dynamodb/src/schema/manual-schema.ts	97.4%	39, 79
	packages/datasource-dynamodb/src/utils/sort.ts	100.0%
	packages/datasource-dynamodb/src/utils/concurrency.ts	100.0%
	packages/datasource-dynamodb/src/collection.ts	98.4%	101-102, 478-479, 709
	packages/datasource-dynamodb/src/utils/aggregator.ts	97.4%	29, 35
	packages/datasource-dynamodb/src/datasource.ts	97.1%	89
	packages/datasource-dynamodb/src/introspection/introspector.ts	100.0%
	packages/datasource-dynamodb/src/errors.ts	100.0%
	packages/datasource-dynamodb/src/utils/type-converter.ts	97.6%	72
	packages/datasource-dynamodb/src/virtual-collections.ts	100.0%
	packages/datasource-dynamodb/src/utils/executor.ts	100.0%
	packages/datasource-dynamodb/src/client.ts	100.0%
	packages/datasource-dynamodb/src/utils/partiql.ts	100.0%
	packages/datasource-dynamodb/src/settings.ts	100.0%
	packages/datasource-dynamodb/src/utils/pagination-cache.ts	96.6%	67
	packages/datasource-dynamodb/src/index.ts	100.0%
	packages/datasource-dynamodb/src/utils/access-path.ts	98.3%	136, 162
	packages/datasource-dynamodb/src/array-collection.ts	95.6%	374, 397, 429-431...
	packages/datasource-dynamodb/src/utils/serializer.ts	96.6%	135-136
	packages/datasource-dynamodb/src/utils/compare.ts	100.0%
	packages/datasource-dynamodb/src/utils/expression-builder.ts	100.0%
	packages/datasource-dynamodb/src/introspection/builder.ts	100.0%
	packages/datasource-dynamodb/src/schema/schema-converter.ts	100.0%
	Total	98.2%

🤖 Increase coverage with AI coding...

In the `feature/prd-1670-dynamodb-datasource-for-the-nodejs-agent-parity-with-cosmos` branch, add test coverage for this new code:

- `packages/datasource-dynamodb/src/array-collection.ts` -- Lines 374, 397, 429-431, 452-454, 480, 535, 544, and 566
- `packages/datasource-dynamodb/src/collection.ts` -- Lines 101-102, 478-479, and 709
- `packages/datasource-dynamodb/src/datasource.ts` -- Line 89
- `packages/datasource-dynamodb/src/schema/manual-schema.ts` -- Lines 39 and 79
- `packages/datasource-dynamodb/src/utils/access-path.ts` -- Lines 136 and 162
- `packages/datasource-dynamodb/src/utils/aggregator.ts` -- Lines 29 and 35
- `packages/datasource-dynamodb/src/utils/pagination-cache.ts` -- Line 67
- `packages/datasource-dynamodb/src/utils/serializer.ts` -- Line 135-136
- `packages/datasource-dynamodb/src/utils/type-converter.ts` -- Line 72

🚦 See full report on Qlty Cloud »

🛟 Help

• Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

• Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

• File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

  • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

[macroscopeapp]

🟠 High src/array-collection.ts:557

The rewrite fallbacks can erase successful concurrent writes, including appends and updates to unrelated parent attributes. replaceElements and appendByRewriting put the entire item snapshot read by get, while their conditions only check item existence and entity membership; because PutItem replaces the whole item, changes made after the read are lost. Restrict writes to the affected path or condition the replacement on the snapshot and retry on conflict.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/datasource-dynamodb/src/array-collection.ts around line 557:

The rewrite fallbacks can erase successful concurrent writes, including appends and updates to unrelated parent attributes. `replaceElements` and `appendByRewriting` put the entire item snapshot read by `get`, while their conditions only check item existence and entity membership; because `PutItem` replaces the whole item, changes made after the read are lost. Restrict writes to the affected path or condition the replacement on the snapshot and retry on conflict.

[PMerlet]

Claude Sonnet 5.5 (claude-sonnet-5-5): Not fixed here: a rewrite fallback can overwrite a write that lands between its read and its put, and closing that needs optimistic locking on a version attribute, which is a feature of its own.

Why

The read is strongly consistent and the put is conditioned on the item existing, so only a concurrent write inside that window is lost. Tracked in PRD-1673, which covers these fallbacks and replaceOne.

[PMerlet]

Spec (PRD-1670): 7 unvalidated decisions: not final. One option name differs from the ticket, see the finding on index.ts.