Add df.wait_for_condition() for condition-based triggering - #353
Draft
Todd J. Green (tjgreen42) wants to merge 5 commits into
Draft
Add df.wait_for_condition() for condition-based triggering#353Todd J. Green (tjgreen42) wants to merge 5 commits into
Todd J. Green (tjgreen42) wants to merge 5 commits into
Conversation
Durable functions could only be triggered on a wall-clock schedule via df.wait_for_schedule(). Workflows that need to react to data changes had to poll on a cron cadence, trading latency against wasted wakeups. Add df.wait_for_condition(condition, max_check_interval, notify_key), which blocks a durable instance until a SQL predicate becomes true. The predicate is evaluated when the instance starts waiting, whenever a matching pg_notify arrives, and at least once per max_check_interval as a backstop. notify_key is optional: omit it and the node degenerates to pure interval polling. Implementation: - WAIT_CONDITION node type, registered in VALID_NODE_TYPES, both nodes CHECK constraints, and df.explain(). Missing any of these makes Durofut::ensure() silently treat the node JSON as raw SQL. - df.condition_waiters registry, indexed on notify_key, populated by the register/unregister activities that bracket each wait. - A listener task in the background worker LISTENs on the pg_durable_condition channel and enqueues a duroxide event to each waiter registered for the payload key, with per-key suppression to collapse notification storms and a resync on reconnect. - Predicates run with default_transaction_read_only, and are wrapped as SELECT (<condition>) IS TRUE so PostgreSQL's own boolean rules decide the outcome: no rows or NULL is false, and a multi-row or non-boolean predicate is an error rather than a silent false. - ExecuteSqlInput.read_only is skipped when false so pre-existing duroxide histories serialize byte-identically on replay. - The reconcile tick sweeps waiter rows whose instance is no longer running. Includes the design doc, an E2E test covering five scenarios, and the 0.2.5 to 0.2.6 upgrade script DDL.
Three fixes from reviewing the wait-for-condition implementation: - Build the dequeue queue name with types::condition_queue_name() instead of repeating its format string in the orchestration. The listener already used the helper, so the two could have drifted apart and silently stopped waking waiters. - Reject a max_check_interval_secs below 1. df.wait_for_condition() already rejects sub-second intervals, but a graph can be hand-written as raw node JSON, where a negative value widened through `as u64` into an effectively infinite timer. - Record why dropping an un-awaited dequeue future is safe. DurableFuture::drop marks the token cancelled and duroxide skips cancelled subscriptions when matching arrivals in FIFO order, so a buffered notification passes to the next iteration rather than being consumed and lost. Non-obvious enough to state in both the code and the design doc. Also extend the PGRX_HOME/PG_PORT overrides to test-shutdown.sh and test-epoch-race.sh so the whole test-script family can target a non-default cluster.
The interval guard added in the previous commit was inline and therefore untestable. Extract it as check_interval() and test it alongside condition_met(), which had no direct coverage either: it is only exercised end-to-end, where a wrong answer looks like a hang rather than a failure.
The condition wait loop created a fresh dequeue_event subscription on every iteration. That was correct -- a dropped DurableFuture is marked cancelled and duroxide skips cancelled subscriptions when matching arrivals in FIFO order, so no notification was lost -- but it wrote a subscribe/cancel pair to history per check, and duroxide computes an arrival index by scanning every prior subscription for the queue name, making a long wait quadratic in the number of checks. Hold the subscription across iterations instead, recreating it only once it has actually delivered. DurableFuture is Unpin, so &mut fut is itself a future and the subscription outlives the select2 that borrows it. Over a 34-check probe this drops history from 6 events per check to 4, and from 35 subscriptions to 1. Add an E2E case that sends a second notification after the first was consumed with the predicate still false, which is the case that fails if the subscription is consumed without being recreated.
Other specs in this repo live at docs/spec-<topic>.md, not under docs/superpowers/specs/ with a date prefix. Point the cross-reference at spec-failure-policy.md, which has moved the same way.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the wait-for-condition design, included in this PR.
Problem
Durable functions could only be triggered on a wall-clock schedule via
df.wait_for_schedule(). A workflow that needs to react to data changes — the motivating case is background compaction of apg_textsearchBM25 index — had to poll on a cron cadence, trading latency against wasted wakeups.What this adds
Blocks a durable instance until a SQL predicate becomes true. The predicate is evaluated:
pg_notifyarrives, andmax_check_intervalas a backstop.notify_keyis optional — omit it and the node degenerates to pure interval polling, which is still correct, just higher-latency.The producer side is just
pg_notify('pg_durable_condition', 'ts_segments').Implementation notes
WAIT_CONDITIONnode type. Registered inVALID_NODE_TYPES, in bothnodesCHECK constraints, and indf.explain(). Worth calling out: missing any one of these fails silently —Durofut::ensure()falls back to treating the node's JSON as a raw SQL string, so the graph builds,df.start()succeeds, and the instance just sitspending. Cost me a debugging cycle; it's now written down in the spec.df.condition_waitersregistry, indexed onnotify_key, populated by register/unregister activities that bracket each wait.LISTENs on thepg_durable_conditionchannel and enqueues a duroxide event to each waiter registered for the payload key. Per-key suppression (1s) collapses notification storms; the listener resyncs on reconnect so nothing is missed across a dropped connection.default_transaction_read_only, wrapped asSELECT (<condition>) IS TRUE. This delegates to PostgreSQL's own boolean rules rather than inventing new ones: no rows orNULLis false, and a multi-row or non-boolean predicate is an error rather than a silent false.ExecuteSqlInput.read_onlyisskip_serializing_if = "is_false", so pre-existing duroxide histories serialize byte-identically.dequeue_event/enqueue_event(notraise_event) is the right duroxide primitive here: the action is emitted at future-creation time, so the subscription is durable before the predicate first runs, and it behaves as a mailbox — events that arrive while the instance is unparked are buffered rather than lost.dequeue_eventper iteration. That is correct — a droppedDurableFutureis marked cancelled and duroxide skips cancelled subscriptions when matching arrivals in FIFO order — but it writes a subscribe/cancel pair to history per check, and duroxide derives an arrival index by scanning every prior subscription for that queue name, so a long wait goes quadratic. The node now holds one subscription across iterations and recreates it only once it has actually delivered (DurableFutureisUnpin, so&mut futis a future and survives theselect2that borrows it). Measured over a 34-check probe: 6 history events per check → 4, and 35 subscriptions → 1.Testing
tests/e2e/sql/67_wait_for_condition.sql— 6 cases: already-true, becomes-true on the interval backstop, NOTIFY beating a 5-minute backstop (plus waiter-row lifecycle), a second NOTIFY after the first was consumed with the predicate still false — the case that fails if the subscription is not recreated — a writing predicate, and a non-boolean predicate.#[pg_test]cases insrc/lib.rs(DSL shape, table shape, index), plus unit tests forwrap_condition_sql,condition_queue_name, theread_onlyserialization contract, and the listener's suppression logic.cargo fmtclean. Clippy adds zero new findings (verified against themainbaseline: 27 pre-existinguninlined_format_argsbefore and after).Deferred
Bounded history — unwinding the orchestration after ~100 condition checks — is intentionally left out. It depends on the unwind mechanism from the sibling failure policy spec, which is being implemented separately. The
on_failureinteraction cases are deferred for the same reason.Flag for review
scripts/test-e2e-local.shandscripts/test-upgrade.shnow honorPGRX_HOME/PG_PORTenv overrides (2 lines each). I needed this to run an isolated cluster while another workstream shared~/.pgrxand port 28817. It's defensible on its own as a parallel-work enabler, but it is scope creep — happy to drop it if you'd rather keep this PR focused.