Skip to content

Retry failed change-feed batches and skip orchestrations that already exist - #55

Merged
Aayush Kataria (aayush3011) merged 3 commits into
AzureCosmosDB:mainfrom
aayush3011:users/akataria/changefeed-retry-policy
Nov 18, 2026
Merged

Aayush Kataria (aayush3011) merged 3 commits into
AzureCosmosDB:mainfrom
aayush3011:users/akataria/changefeed-retry-policy

Conversation

@aayush3011

Copy link
Copy Markdown
Collaborator

Summary

The change-feed trigger dropped a batch that failed: the host moved on, so the turns in it were never counted and the orchestrations they would have started never ran. This makes the host retry the batch, and makes a retry safe: it counts no turn twice and runs no orchestration twice. It is the retry policy that #53 left for a follow-up.

What changed

  • Retry policy. on_memory_change retries a failed batch up to 5 times with exponential backoff (5 s minimum, 5 min maximum). If every retry fails the batch is dropped, as before. The count is bounded because a failure that does not go away, such as a container with the wrong partition key, must be given up on and not retried forever. The values are constants, not settings.
  • An orchestration that already exists is not started again. Durable Functions refuses an instance id only while its instance is Pending or Running, and starts a finished one again. A retried batch derives the ids of the orchestrations its earlier attempt started, so _safe_start now reads the instance's status first and leaves any instance that exists, in any status, alone. A failed status read fails the batch, so it is retried. The 409 handling stays for two deliveries racing.
  • The batch LSN is part of the instance id. A counter expires 90 days after its last update, so a thread idle that long counts from 1 again and can reach a count it reached before, while the old instance still exists. The highest _lsn of the batch that crossed the threshold is hashed into the digest, so that crossing gets an id of its own, and a replay of the same batch (same LSN) gets the same id. The id keeps its <kind>:<digest>[:<count>] shape, and a turn without an _lsn gets the id it had before. The procedural id, built inside the extraction orchestrator, is unchanged.
  • Docs and tests. concepts.md, troubleshooting.md, architecture/orchestrators.md and the CHANGELOG describe the retries and the replay guard. The test starter now knows the instances it starts, so a test can run a batch twice, and the azure.durable_functions stub records a function's retry policy.

Counters were already safe to replay: a replay of a batch (same LSN) returns the counts it returned the first time, so a retry counts no turn twice. A turn with an invalid id is skipped, and a failure for one thread or owner does not stop the rest of the batch (both from #53), so a bad turn cannot fail every attempt.

Behavior to know about

  • A batch that is being retried keeps its lease. The changes behind it wait until it succeeds or the retries are used up: about a minute and a half when every retry fails (attempts at 0, 5, 15, 31, 56 and 94 s in a live run). A failure that clears itself costs a few seconds. This is the cost of retrying inside the host.
  • The host's retry count is best effort. Azure keeps it in the instance's memory, so a batch can be retried more than 5 times if an instance fails between retries, and a batch can also be delivered again after a function timeout or when leases move between instances. The same guards cover both: nothing is counted or started twice.
  • A retry that finds an orchestration its earlier attempt started logs change-feed: instance <id> of <orchestration> already exists, skipping. That is expected, not a duplicate run.
  • Upgrade: a batch that is in flight while the new Function App deploys, and delivered again afterwards, derives ids that include the LSN, so it can start an orchestration under a new id that the old id already started. Record ids are deterministic, so the extra run writes nothing twice; it costs one repeated model call.
  • This changes the Function App only. The SDK package and the pins in function_app/requirements.txt are unchanged.

Validation

  • 3,486 unit tests pass (8 new). ruff check, ruff format --check and the doc checks are clean.
  • 9 in-process mutants, each removing one behavior (no status read, the LSN left out of the id, a swallowed status failure, a 409 treated as an error, no / unbounded / fixed-delay retry policy), each make at least one test fail.
  • The real azure-functions registers the policy on the Cosmos DB trigger: indexing function_app and reading the function's retry_policy setting gives exponential_backoff, 5, 00:00:05, 00:05:00.
  • Live, against a local Functions host and a Cosmos DB database, with a fault injected into the worker's counter reads:
    • Three 503s on one thread's counter, in a batch that also holds a healthy thread. The host ran the batch 4 times; both counters read 4, not 8; the healthy thread's orchestration, started on the first attempt and finished before the last, was started once and found and skipped on all three retries. A counter deleted as its TTL would, then reaching the same count at a later LSN, started a new instance (19 checks).
    • Every read failing. The host ran the batch 6 times (1 + 5 retries) and then dropped it, and did not deliver it again in the next 90 s. The healthy thread was counted once and the failing one never, and a second batch written during the retries was processed only after the last attempt (12 checks).
  • Not checked: a deployed (not local) Function App.

… exist

The change-feed trigger dropped a batch that failed, so the turns in it were never counted. The host now retries it up to five times with exponential backoff, then drops it as before.

A retry repeats only what failed. Counters already skip turns they have counted. _safe_start now reads an instance status first and leaves an instance that exists alone, because Durable Functions starts a finished instance again when its id is reused. The highest change-feed LSN of the batch joins the instance id hash, so a count reached again after a counter expired is not mistaken for the earlier run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The instance id digest now also covers the change-feed batch, so the entry that describes it says so.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The non-atomic status check and start allow overlapping workers to launch a duplicate run after an instance completes, bypassing the 409 handler.

1 open issue
What changed in this PR

This PR adds bounded retries to the Function App’s change-feed trigger, completing the retry-policy follow-up from #53. It also adds replay guards intended to preserve successful work without starting existing orchestrations again.

Changes:

  • Retry failed batches up to five times with exponential backoff.
  • Check orchestration status before starting and include the batch LSN in threshold-based instance IDs.
  • Extend tests and documentation for retries, replay handling, and counter expiration.
File Description
tests/​unit/​function_app/​test_instance_ids.py Tests LSN-based identity and ID length limits.
tests/​unit/​function_app/​test_change_feed.py Adds replay, partial-failure, expiration, and retry-policy coverage.
tests/​unit/​conftest.py Records retry-policy metadata in the Durable Functions stub.
function_app/​triggers/​change_feed.py Adds bounded retries, status checks, and LSN-aware IDs.
function_app/​shared/​instance_ids.py Supports an optional LSN in the instance-ID digest.
Docs/​troubleshooting.md Explains retry behavior and operational symptoms.
Docs/​concepts.md Describes retries and replay guards in the processing flow.
Docs/​architecture/​orchestrators.md Documents existing-instance checks and LSN-based IDs.
CHANGELOG.md Records retry behavior and instance-ID changes.

Review effort: Balanced

Comment thread function_app/triggers/change_feed.py Resolved
@aayush3011
Aayush Kataria (aayush3011) merged commit c98dec7 into AzureCosmosDB:main Nov 18, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants