feat(ent): implement metering based on clickhouse direct

[julianpoy]

Because

• We want to use clickhouse directly instead of openmeter

This pull request

• Uses clickhouse directly instead of openmeter
• Revamps our ingestion pipeline
• Adds support for rolling windows and session meter types

Issue that this pull request solves

Closes: ENT-50

[xlisachan]

LGTM! Thanks Julian!

[Copilot]

🟡 Changes recommended

Dedupe durability, session boundaries, sliding idempotency, and window-expiry behavior contain correctness issues.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Replaces OpenMeter-based entitlements metering with a Pub/Sub, Redis, and direct ClickHouse pipeline.

Changes:

• Adds durable event publishing, deduplication, ClickHouse storage, and querying.
• Adds calendar, sliding, and session windows with scheduled threshold sweeps.
• Adds ClickHouse migrations, configuration, tests, and local infrastructure.

File summaries

File	Description
package.json	Replaces OpenMeter dependencies with Redis.
yarn.lock	Updates resolved dependencies.
tsconfig.base.json	Adds the metering testing export.
libs/shared/cms/codegen.config.ts	Uses the imported GraphQL client preset.
libs/shared/cms/src/lib/cms.error.ts	Adds invalid-window errors.
libs/shared/cms/src/lib/metering-configuration.manager.spec.ts	Tests normalized meters.
libs/shared/cms/src/lib/queries/meter/factories.ts	Adds raw meter factories.
libs/shared/cms/src/lib/queries/meter/query.ts	Queries structured window fields.
libs/shared/cms/src/lib/queries/meter/types.ts	Defines calendar, sliding, and session windows.
libs/shared/cms/src/lib/queries/meter/util.ts	Normalizes and validates meter windows.
libs/entitlements/metering/README.md	Documents threshold sweeps.
libs/entitlements/metering/jest.config.ts	Updates Jest transformation configuration.
libs/entitlements/metering/project.json	Adds migrations and integration-test targets.
libs/entitlements/metering/src/index.ts	Exports the new metering API.
libs/entitlements/metering/src/testing.ts	Adds integration-test utilities.
libs/entitlements/metering/src/lib/clickhouse.client.ts	Implements the ClickHouse HTTP client.
libs/entitlements/metering/src/lib/clickhouse.config.ts	Defines ClickHouse configuration.
libs/entitlements/metering/src/lib/metering-auth.guard.spec.ts	Updates authentication fixtures.
libs/entitlements/metering/src/lib/metering-cloud-tasks.controller.ts	Removes the Cloud Tasks endpoint.
libs/entitlements/metering/src/lib/metering-cloud-tasks.controller.spec.ts	Removes its tests.
libs/entitlements/metering/src/lib/metering-cloud-tasks.guard.ts	Removes Cloud Tasks authentication.
libs/entitlements/metering/src/lib/metering-consumer.service.ts	Consumes and stores Pub/Sub events.
libs/entitlements/metering/src/lib/metering-dedupe.manager.ts	Implements Redis deduplication.
libs/entitlements/metering/src/lib/metering-dedupe.manager.in.spec.ts	Tests Redis integration.
libs/entitlements/metering/src/lib/metering-dedupe.provider.ts	Provides the Redis client.
libs/entitlements/metering/src/lib/metering-events.manager.ts	Coordinates event persistence and queries.
libs/entitlements/metering/src/lib/metering-events.manager.spec.ts	Tests event-manager delegation.
libs/entitlements/metering/src/lib/metering-events.repository.ts	Implements event storage and usage sums.
libs/entitlements/metering/src/lib/metering-events.repository.spec.ts	Tests ClickHouse event queries.
libs/entitlements/metering/src/lib/metering-exception.filter.ts	Maps new metering errors to HTTP responses.
libs/entitlements/metering/src/lib/metering-exception.filter.spec.ts	Tests the new error mappings.
libs/entitlements/metering/src/lib/metering-ingest.manager.ts	Removes in-memory OpenMeter ingestion.
libs/entitlements/metering/src/lib/metering-publisher.manager.ts	Publishes usage events to Pub/Sub.
libs/entitlements/metering/src/lib/metering-publisher.manager.spec.ts	Tests publishing behavior.
libs/entitlements/metering/src/lib/metering-pubsub.config.ts	Defines Pub/Sub configuration.
libs/entitlements/metering/src/lib/metering-pubsub.provider.ts	Provides Pub/Sub resources.
libs/entitlements/metering/src/lib/metering-query.manager.ts	Removes OpenMeter querying.
libs/entitlements/metering/src/lib/metering-query.manager.spec.ts	Removes OpenMeter query tests.
libs/entitlements/metering/src/lib/metering-redis.config.ts	Defines Redis dedupe configuration.
libs/entitlements/metering/src/lib/metering-sweep.config.ts	Defines threshold-sweep settings.
libs/entitlements/metering/src/lib/metering-sweep.manager.ts	Coordinates sweep persistence operations.
libs/entitlements/metering/src/lib/metering-sweep.repository.ts	Implements sweep-related ClickHouse queries.
libs/entitlements/metering/src/lib/metering-sweep.service.ts	Detects and dispatches threshold crossings.
libs/entitlements/metering/src/lib/metering-threshold.service.ts	Removes task-driven threshold handling.
libs/entitlements/metering/src/lib/metering-threshold-tasks.manager.ts	Removes Cloud Tasks scheduling.
libs/entitlements/metering/src/lib/metering-threshold-tasks.provider.ts	Removes the Cloud Tasks provider.
libs/entitlements/metering/src/lib/metering-webhook.manager.ts	Updates webhook dispatch and error handling.
libs/entitlements/metering/src/lib/metering-webhook.manager.spec.ts	Tests updated webhook dispatches.
libs/entitlements/metering/src/lib/metering.constants.ts	Defines tables and pipeline limits.
libs/entitlements/metering/src/lib/metering.error.ts	Adds pipeline-specific errors.
libs/entitlements/metering/src/lib/metering.factories.ts	Adds metering pipeline fixtures.
libs/entitlements/metering/src/lib/metering.schema.ts	Defines wire events and integer usage.
libs/entitlements/metering/src/lib/metering.types.ts	Defines persistence and sweep contracts.
libs/entitlements/metering/src/lib/openmeter-client.d.ts	Removes the OpenMeter type shim.
libs/entitlements/metering/src/lib/openmeter.client.ts	Removes the OpenMeter client.
libs/entitlements/metering/src/lib/usage-grants.controller.ts	Reuses centralized request parsing.
libs/entitlements/metering/src/lib/usage-grants.controller.spec.ts	Updates grant-controller fixtures.
libs/entitlements/metering/src/lib/usage-grants.manager.spec.ts	Updates factory imports.
libs/entitlements/metering/src/lib/usage-grants.repository.ts	Documents a future repository refactor.
libs/entitlements/metering/src/lib/usage-grants.service.ts	Adapts grants to structured windows.
libs/entitlements/metering/src/lib/usage-grants.service.spec.ts	Tests window-based grant expiry.
libs/entitlements/metering/src/lib/usage.controller.ts	Passes authenticated client scope.
libs/entitlements/metering/src/lib/usage.controller.spec.ts	Tests client-scoped calls.
libs/entitlements/metering/src/lib/usage.service.ts	Publishes and queries direct metering data.
libs/entitlements/metering/src/lib/utils/batchBuffer.ts	Adds asynchronous batch buffering.
libs/entitlements/metering/src/lib/utils/batchBuffer.spec.ts	Tests buffering and draining.
libs/entitlements/metering/src/lib/utils/buildIdempotencyKey.ts	Removes the old webhook key helper.
libs/entitlements/metering/src/lib/utils/buildIdempotencyKey.spec.ts	Removes its tests.
libs/entitlements/metering/src/lib/utils/buildThresholdTaskId.ts	Removes Cloud Tasks identifiers.
libs/entitlements/metering/src/lib/utils/buildThresholdTaskId.spec.ts	Removes their tests.
libs/entitlements/metering/src/lib/utils/chunk.ts	Adds array chunking.
libs/entitlements/metering/src/lib/utils/chunk.spec.ts	Tests array chunking.
libs/entitlements/metering/src/lib/utils/classifyEnqueueError.ts	Removes Cloud Tasks error classification.
libs/entitlements/metering/src/lib/utils/classifyEnqueueError.spec.ts	Removes its tests.
libs/entitlements/metering/src/lib/utils/computeThresholdsMet.ts	Revises threshold arithmetic.
libs/entitlements/metering/src/lib/utils/computeWindow.ts	Replaced by structured window resolution.
libs/entitlements/metering/src/lib/utils/computeWindow.spec.ts	Removes legacy window tests.
libs/entitlements/metering/src/lib/utils/computeWindowId.ts	Adds stable window identities.
libs/entitlements/metering/src/lib/utils/computeWindowId.spec.ts	Tests window identities.
libs/entitlements/metering/src/lib/utils/ensurePubSubEmulatorResources.ts	Creates emulator topics and subscriptions.
libs/entitlements/metering/src/lib/utils/ensurePubSubEmulatorResources.spec.ts	Tests emulator setup.
libs/entitlements/metering/src/lib/utils/hashEventId.ts	Maps event IDs to UInt64 hashes.
libs/entitlements/metering/src/lib/utils/hashEventId.spec.ts	Tests event hashing.
libs/entitlements/metering/src/lib/utils/isTimestampInRange.ts	Validates event timestamp bounds.
libs/entitlements/metering/src/lib/utils/isTimestampInRange.spec.ts	Tests timestamp boundaries.
libs/entitlements/metering/src/lib/utils/notificationKey.ts	Builds notification lookup keys.
libs/entitlements/metering/src/lib/utils/notificationKey.spec.ts	Tests notification keys.
libs/entitlements/metering/src/lib/utils/parseRequest.ts	Centralizes Zod request parsing.
libs/entitlements/metering/src/lib/utils/parseRequest.spec.ts	Tests request errors.
libs/entitlements/metering/src/lib/utils/parseWireEvent.ts	Parses Pub/Sub event payloads.
libs/entitlements/metering/src/lib/utils/parseWireEvent.spec.ts	Tests wire-event validation.
libs/entitlements/metering/src/lib/utils/resolveWindow.ts	Resolves all supported window kinds.
libs/entitlements/metering/src/lib/utils/shouldNotify.ts	Applies notification suppression rules.
libs/entitlements/metering/src/lib/utils/shouldNotify.spec.ts	Tests suppression and cooldowns.
libs/entitlements/metering/src/lib/utils/stringToBoolean.ts	Transforms boolean configuration values.
libs/entitlements/metering/src/lib/utils/stringToBoolean.spec.ts	Tests boolean transformation.
libs/entitlements/metering/src/lib/utils/toClickHouseDateTime.ts	Formats ClickHouse timestamps.
libs/entitlements/metering/src/lib/utils/toClickHouseDateTime.spec.ts	Tests timestamp formatting.
libs/entitlements/metering/src/lib/utils/toClickHouseStringArray.ts	Serializes ClickHouse string arrays.
libs/entitlements/metering/src/lib/utils/toClickHouseStringArray.spec.ts	Tests array escaping.
libs/entitlements/metering/src/lib/utils/toError.ts	Normalizes unknown errors.
libs/entitlements/metering/src/lib/utils/toError.spec.ts	Tests error normalization.
libs/entitlements/metering/src/lib/utils/toMeteringCloudEvent.ts	Removes OpenMeter event conversion.
libs/entitlements/metering/src/lib/utils/toMeteringCloudEvent.spec.ts	Removes conversion tests.
libs/entitlements/metering/migrations/clickhouse/001_create_events.sql	Creates event storage.
libs/entitlements/metering/migrations/clickhouse/002_create_notifications_sent.sql	Creates notification history.
libs/entitlements/metering/migrations/clickhouse/003_create_sessions.sql	Creates session state.
libs/entitlements/metering/migrations/clickhouse/004_create_sweep_watermarks.sql	Creates sweep watermarks.
libs/entitlements/metering/migrations/clickhouse/README.md	Documents migration constraints.
libs/entitlements/metering/migrations/clickhouse/migrate.mjs	Implements the migration runner.
libs/entitlements/metering/migrations/clickhouse/target-patch.json	Selects migration level four.
apps/payments/api/src/app/app.module.ts	Registers the new metering pipeline.
apps/payments/api/src/app/metering-sweep.module.ts	Defines the standalone sweep module.
apps/payments/api/src/app/metering-sweep.providers.ts	Groups sweep providers.
apps/payments/api/src/config/config.module.ts	Extracts reusable typed configuration.
apps/payments/api/src/main.ts	Removes obsolete Swagger filtering.
apps/payments/api/src/metering-sweep.ts	Adds the sweep executable.
apps/payments/api/src/swagger.utils.ts	Removes internal-route stripping.
apps/payments/api/src/swagger.utils.spec.ts	Removes obsolete filtering tests.
apps/payments/api/project.json	Adds subscriber and sweep targets.
apps/payments/api/pm2.config.js	Starts local subscriber and sweep processes.
apps/payments/api/webpack.config.js	Builds the sweep entry point.
apps/payments/api/.env	Restricted configuration file; content not reviewed.
_scripts/clickhouse.sh	Starts and migrates local ClickHouse.
_dev/pm2/infrastructure.config.js	Adds ClickHouse to local infrastructure.

Review details

Files not reviewed (2)

• libs/shared/cms/src/generated/gql.ts: Generated file
• libs/shared/cms/src/generated/graphql.ts: Generated file

Suppressed comments (1)

libs/entitlements/metering/src/lib/metering-consumer.service.ts:211

• A Redis confirmation failure occurs after the rows are inserted, but this path still ACKs every Pub/Sub message. The only dedupe marker can therefore expire after the short claim TTL; a later delivery of the same event is then inserted again, and the usage queries sum both rows without FINAL. Keep the message retryable until the durable stored marker is confirmed.

• Files reviewed: 141/145 changed files
• Comments generated: 6
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.