Repository navigation
Retry failed change-feed batches and skip orchestrations that already exist - #55
Merged
Aayush Kataria (aayush3011) merged 3 commits intoNov 18, 2026
Conversation
… exist The change-feed trigger dropped a batch that failed, so the turns in it were never counted. The host now retries it up to five times with exponential backoff, then drops it as before. A retry repeats only what failed. Counters already skip turns they have counted. _safe_start now reads an instance status first and leaves an instance that exists alone, because Durable Functions starts a finished instance again when its id is reused. The highest change-feed LSN of the batch joins the instance id hash, so a count reached again after a counter expired is not mistaken for the earlier run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The instance id digest now also covers the change-feed batch, so the entry that describes it says so. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Aayush Kataria (aayush3011)
marked this pull request as ready for review
November 18, 2026 17:12
Aayush Kataria (aayush3011)
requested review from
Theo van Kraay (TheovanKraay) and
James Codella (jcodella)
as code owners
November 18, 2026 17:12
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The non-atomic status check and start allow overlapping workers to launch a duplicate run after an instance completes, bypassing the 409 handler.
1 open issue
What changed in this PR
This PR adds bounded retries to the Function App’s change-feed trigger, completing the retry-policy follow-up from #53. It also adds replay guards intended to preserve successful work without starting existing orchestrations again.
Changes:
- Retry failed batches up to five times with exponential backoff.
- Check orchestration status before starting and include the batch LSN in threshold-based instance IDs.
- Extend tests and documentation for retries, replay handling, and counter expiration.
| File | Description |
|---|---|
tests/unit/function_app/test_instance_ids.py |
Tests LSN-based identity and ID length limits. |
tests/unit/function_app/test_change_feed.py |
Adds replay, partial-failure, expiration, and retry-policy coverage. |
tests/unit/conftest.py |
Records retry-policy metadata in the Durable Functions stub. |
function_app/triggers/change_feed.py |
Adds bounded retries, status checks, and LSN-aware IDs. |
function_app/shared/instance_ids.py |
Supports an optional LSN in the instance-ID digest. |
Docs/troubleshooting.md |
Explains retry behavior and operational symptoms. |
Docs/concepts.md |
Describes retries and replay guards in the processing flow. |
Docs/architecture/orchestrators.md |
Documents existing-instance checks and LSN-based IDs. |
CHANGELOG.md |
Records retry behavior and instance-ID changes. |
Review effort: Balanced
Aayush Kataria (aayush3011)
merged commit Nov 18, 2026
c98dec7
into
AzureCosmosDB:main
8 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
The change-feed trigger dropped a batch that failed: the host moved on, so the turns in it were never counted and the orchestrations they would have started never ran. This makes the host retry the batch, and makes a retry safe: it counts no turn twice and runs no orchestration twice. It is the retry policy that #53 left for a follow-up.
What changed
on_memory_changeretries a failed batch up to 5 times with exponential backoff (5 s minimum, 5 min maximum). If every retry fails the batch is dropped, as before. The count is bounded because a failure that does not go away, such as a container with the wrong partition key, must be given up on and not retried forever. The values are constants, not settings._safe_startnow reads the instance's status first and leaves any instance that exists, in any status, alone. A failed status read fails the batch, so it is retried. The 409 handling stays for two deliveries racing._lsnof the batch that crossed the threshold is hashed into the digest, so that crossing gets an id of its own, and a replay of the same batch (same LSN) gets the same id. The id keeps its<kind>:<digest>[:<count>]shape, and a turn without an_lsngets the id it had before. Theproceduralid, built inside the extraction orchestrator, is unchanged.concepts.md,troubleshooting.md,architecture/orchestrators.mdand the CHANGELOG describe the retries and the replay guard. The test starter now knows the instances it starts, so a test can run a batch twice, and theazure.durable_functionsstub records a function's retry policy.Counters were already safe to replay: a replay of a batch (same LSN) returns the counts it returned the first time, so a retry counts no turn twice. A turn with an invalid id is skipped, and a failure for one thread or owner does not stop the rest of the batch (both from #53), so a bad turn cannot fail every attempt.
Behavior to know about
change-feed: instance <id> of <orchestration> already exists, skipping. That is expected, not a duplicate run.function_app/requirements.txtare unchanged.Validation
ruff check,ruff format --checkand the doc checks are clean.azure-functionsregisters the policy on the Cosmos DB trigger: indexingfunction_appand reading the function'sretry_policysetting givesexponential_backoff,5,00:00:05,00:05:00.