Conversation
![postil-dev[bot]](https://avatars.githubusercontent.com/in/3455612?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 finding posted inline
1 suppressed
- Manual alert tests no longer trigger notifications at
.github/workflows/production-monitor.yml:452: below the configured confidence threshold; severity error, confidence 0.04. The notification job now runs only when the smoke job fails, so manual runs withtest_alert=trueskip the external alert. Preserve the operator-requested test-alert branch and its delivery verification.
There was a problem hiding this comment.
1 finding posted inline
3 suppressed
- Allow the manual canary to run when smoke fails at
.github/workflows/production-monitor.yml:540: cites a line the named construct does not sit on; severity warn, confidence 0.94. Because this job needssmokeand its condition is not wrapped inalways(), GitHub Actions skips the job whensmokefails or is cancelled, even wheninputs.test_alertis true. This regresses the prior manual test-alert path, which could validate delivery during monitor failures. Remove the dependency or use an always-evaluated condition that explicitly permits the manual canary to run. - Preserve external alerts for manual test runs at
.github/workflows/production-monitor.yml:598: cites a line the named construct does not sit on; severity warn, confidence 0.98. This condition removed theinputs.test_alert == truebranch, so a manually requested test run no longer reaches the external alert action even when the test is intended to verify alert delivery. Retain the test-alert condition or otherwise route test runs to the alert action without affecting production failure handling. - Restore test-alert notification delivery at
.github/workflows/production-monitor.yml:598: below the configured confidence threshold; severity error, confidence 0.04. The notification job no longer runs for operator-requested test alerts, so successful smoke and recovery checks cannot exercise the external alert path. Restore the test-alert condition or invoke an equivalent explicit alert action.
|
@postil re-review |
There was a problem hiding this comment.
1 finding posted inline
1 suppressed
- Preserve operator-requested test alerts at
.github/workflows/production-monitor.yml:598: cites a line the named construct does not sit on; severity error, confidence 0.94. This condition removed theinputs.test_alert == truebranch. When an operator requests a test alert and both prerequisite jobs pass,notifyis skipped, so the alerting path cannot be tested and the requested external alert is never sent. Include|| inputs.test_alert == truein the condition, while preserving the existing failure and cancellation cases.
| // final reads before the request deadline is recomputed and the mutation is | ||
| // dispatched. A concurrent action change during receiver preflight therefore | ||
| // cannot be overwritten from a stale plan. | ||
| const beforeMutationAttempt: BeforeMutationAttempt = async () => { |
There was a problem hiding this comment.
warn · confidence 0.86 · kind: risk
The source and action are re-read in beforeMutationAttempt, but that callback completes before management issues the PUT or POST. Another reconciliation can change the action or source in the gap, allowing this request to overwrite newer state despite the guard. Use an API-supported conditional version or ETag check, or an external per-integration lock held through mutation; otherwise verify and retry after a conflict.
There was a problem hiding this comment.
The residual race with arbitrary out-of-band writers is real. The iLert alert-action API and OpenAPI document expose no ETag, If-Match, version precondition, conflict response, idempotency key, or transaction, while incidents explicitly support optimistic concurrency; repository runs are serialized and the source and action state is revalidated before each mutation.
The only local elimination is verification-only reconciliation, which would remove automatic action creation, drift repair, and secret rotation.
There was a problem hiding this comment.
2 findings posted inline
1 suppressed
- Use the intended current-attempt handoff state at
scripts/reconcile-ilert-alert-stream.ts:1069: below the configured confidence threshold; severity warn, confidence 0.30. This branch checks for the literal handoff value "true", while the surrounding finalizer logic handles the named states "cleaned" and "unknown". If the current-attempt handoff uses the corresponding named state rather than the string "true", this discovery never loads or accounts for the current alert; cleanup can then submit a resolve event without verifying the persisted alert and eventually fail or leave it unmanaged. Replace this comparison with the actual handoff enum/state used by the producer, and verify each handoff path accounts for the current alert.
Before the next push
Run postil review --staged.
| if [[ "${EVENT_NAME}" != "workflow_run" ]]; then | ||
| exit 0 | ||
| fi | ||
| if [[ "${DEPLOY_EVENT}" != "workflow_run" && "${DEPLOY_EVENT}" != "workflow_dispatch" ]]; then |
There was a problem hiding this comment.
warn · confidence 0.72 · kind: risk
The monitor rejects deployments whose originating event is a push, even though the deploy workflow authorizes successful push-triggered runs on main. Accept the workflow's actual provenance and validate the originating main branch before running recovery or smoke jobs.
There was a problem hiding this comment.
These are nested workflow_run events: deploy runs after completed CI, so production-monitor runs after completed deploy and github.event.workflow_run.event is workflow_run, not the CI run origin push; this real deploy run records event: workflow_run. The monitor validates the triggering deploy attempt jobs, and the focused trust cases report 13 pass and 151 assertions (8 skipped).
|
@postil re-review |
There was a problem hiding this comment.
3 findings posted inline
1 suppressed
- Remove the committed webhook credential at
tests/reconcile-ilert-alert-stream.test.ts:1016: deterministically non-actionable; severity error, confidence 0.99. Rotate the credential, purge it from git history because the commit is permanent otherwise, and move it to an environment variable or secrets store. Even as a test fixture, embedding credential material in a webhook URL risks accidental reuse or disclosure.
Before the next push
Run postil review --staged.
| any(.jobs[]; .name == "Authorize deployment source" and .conclusion == "success") | ||
| ' <<<"${jobs}" >/dev/null; then |
There was a problem hiding this comment.
warn · confidence 0.91 · kind: risk
The deployment verifier aggregates authorization across all inspected attempts, allowing an earlier successful attempt to establish trust for a later rerun. Require successful authorization from the current deployment attempt before allowing recovery or monitoring decisions to proceed.
| needs: release-recovery | ||
| if: ${{ always() && (needs.release-recovery.outputs.clear == 'true' || (github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success')) }} | ||
| needs: [deployment-trust, release-recovery] | ||
| if: ${{ always() && needs.deployment-trust.outputs.trusted == 'true' && (needs.release-recovery.outputs.clear == 'true' || (github.event_name == 'workflow_run' && needs.deployment-trust.outputs.conclusion == 'success')) && (github.event_name != 'workflow_dispatch' || github.ref == 'refs/heads/main') }} |
There was a problem hiding this comment.
warn · confidence 0.76 · kind: risk
This condition replaces the prior github.event.workflow_run.conclusion check with needs.deployment-trust.outputs.conclusion. The deployment-trust dependency is only used here for its trusted output, so its conclusion output may be unset; on successful workflow_run events the resolver can therefore be skipped and leave postil-release-recovery open. Restore the event conclusion check or explicitly expose and use a verified equivalent output from deployment-trust.
|
@postil re-review |
| // dispatched. The provider offers no conditional write, so an out-of-band | ||
| // change after these reads remains a detected postcondition risk. |
There was a problem hiding this comment.
warn · confidence 0.94 · kind: risk
The PUT or POST can overwrite an out-of-band alert-action change after the final source and inventory reads complete; the code explicitly acknowledges that this remains an undetected postcondition risk. This can replace another operator's webhook, trigger set, or source binding. Use a provider-supported conditional write or an external distributed lock covering the read/mutation/verification sequence, and fail closed when the precondition cannot be enforced.
There was a problem hiding this comment.
1 finding posted inline
1 suppressed
Remove the committed credential-like webhook value
tests/reconcile-ilert-alert-stream.test.ts:1016
Suppressed: deterministically non-actionable.
Severity error, confidence 0.98.
This added fixture embeds a credential-shaped value directly in a webhook URL. Even if intended only for tests, it can trigger secret exposure and may be copied into logs or tooling. Rotate the value if it has ever been used, purge it from git history, and replace credential material with an environment variable or secrets-store reference; otherwise use a clearly non-credential fixture that cannot be mistaken for a secret.
| any(.jobs[]; .name == "Authorize deployment source" and .conclusion == "success") | ||
| ' <<<"${jobs}" >/dev/null; then | ||
| authorization_trusted=true | ||
| fi |
There was a problem hiding this comment.
error · confidence 0.90 · kind: risk
The authorization loop retains authorization_trusted after an earlier attempt succeeds, while deployment_conclusion is taken from only the current attempt. A rerun can therefore pass trust checks without authorization on its current attempt. Reset and evaluate authorization exclusively for the current attempt.
This branch has not been deployed
1 participant
The manual
production-monitorcanary verifies iLert delivery with an attempt-specific high-priority alert; GitHub OIDC loads scoped Infisical credentials; reconciliation validates its configured source, maintains one fail-closed webhook action, proves exact persisted create and resolve events, and performs deterministic cleanup.Deployment and recovery accept only trusted
mainor manual-main provenance. Private monitoring probes make three bounded attempts for transport and retryable status failures, release response bodies after use, and fail semantic checks immediately; self-hosted iLert monitoring requiresILERT_INTEGRATION_KEY.Running the manual canary sends and resolves a high-priority production test alert and may invoke the configured escalation policy.