fix(dash-spv): resume on the invariant start_download asserts - #955
fix(dash-spv): resume on the invariant start_download asserts#955romchornyi wants to merge 2 commits into
Conversation
`FiltersManager::start_sync` decided whether in-flight work survived a
disconnect by asking `!self.active_batches.is_empty()`. `start_download`,
which the other branch falls into, asserts something stricter:
debug_assert!(self.is_idle(), "manager should have no in-flight state on start");
and `is_idle()` is four conditions — `active_batches`, `tracker`,
`pending_batches` and `filter_pipeline` all empty. The guard checked one
of them.
They come apart routinely. `on_disconnect` deliberately preserves the
first three and calls `requeue_in_flight()`, which moves the pipeline's
in-flight slots to *pending* — making the pipeline non-idle rather than
restoring it. And a batch that finished downloading leaves
`active_batches` while its verified output waits in `pending_batches` and
its blocks wait in the tracker. Any of those outlasting the last active
batch sent a reconnect through the guard and into the assert.
In a build with debug assertions on — which `dev-ios` and every other
`dev`-profile build is — that aborts the host process. It has: a
TestFlight build of the Dash iOS wallet (9.0.0/25, iOS 26.6) died with
SIGABRT in exactly this frame after 33 minutes in the background, where
network teardown and restore make the disconnect/reconnect cycle routine.
The guard now asks `is_idle()`. Resuming is safe for all four cases: the
path sets `Syncing`, and `tick` in that state runs `send_pending`,
`store_and_match_batches` and `try_process_batch` unconditionally, with
`send_pending` a no-op when the pipeline has nothing queued.
`is_idle` becomes `pub(super)` so both the decision and the assertion
read the same predicate.
The regression test fails without the guard change, panicking at the
assert — the production crash reproduced in a unit test.
cargo test -p dash-spv --lib # 538 passed
cargo clippy --all-targets + cargo fmt --check # clean
📝 WalkthroughWalkthroughHeader synchronization now drains buffered segments and completes during ticks. Filter synchronization now resumes any preserved in-flight work after reconnects, including pending verified batches, without starting duplicate downloads. ChangesSynchronization pipeline updates
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant SyncManager
participant BlockHeadersManager
participant RequestSender
participant Storage
SyncManager->>BlockHeadersManager: tick drains ready segments
BlockHeadersManager->>Storage: store promoted headers
BlockHeadersManager-->>SyncManager: return storage events
SyncManager->>BlockHeadersManager: finalize synchronization
BlockHeadersManager->>RequestSender: re-request tip segment if announcements remain
BlockHeadersManager-->>SyncManager: return completion event
sequenceDiagram
participant SyncManager
participant FiltersManager
participant RequestSender
SyncManager->>FiltersManager: start_sync checks is_idle
FiltersManager-->>SyncManager: preserved work remains active
SyncManager->>RequestSender: send pending requests
SyncManager-->>FiltersManager: enter Syncing without fresh initialization
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #955 +/- ##
==========================================
+ Coverage 75.62% 76.05% +0.43%
==========================================
Files 329 329
Lines 79258 79312 +54
==========================================
+ Hits 59939 60322 +383
+ Misses 19319 18990 -329
|
… on a message
A wallet restore froze with the whole testnet chain downloaded and none
of the tail of it stored:
Headers: Syncing 2520289/2520288 (100.0%) processed: 1274000, buffered: 1046289
Filter Headers: Syncing 1474000/2520288 (58.5%)
Blocks: WaitForEvents last_relevant: 1472978
Peers stayed connected and chain locks kept arriving for the sixteen
minutes the app was left running after that. Nothing advanced again.
`processed`/`buffered` decode as: 1,474,000 headers in storage, and
1,046,289 more downloaded, validated and sitting in memory. The
top-line percentage reads 100% because `current_height()` is
`tip + buffered` — it counts what was downloaded, not what was kept.
The only thing that promotes a finished segment into storage is
`take_ready_to_store`, and its single production caller was
`handle_headers_pipeline` — reached only when a `Headers` message
arrives. All 47 checkpoint segments had finished downloading by
22:23:12 (the last being segment 25), so no further `Headers` would
ever arrive, and the promotion had nothing left to trigger it. Filter
headers, filters and blocks then coasted to a stop over the following
four minutes as they consumed the backlog they had been racing ahead
on, which is what made the stall visible at 22:27:20.
`tick` runs every 100ms and called only `handle_timeouts` and
`send_pending`. It now also drains and, if that was the last of the
work, finalizes — so a promotion that gets missed self-heals on the next
tick rather than wedging the client until it is restarted.
`drain_ready_segments` and `finalize_sync_if_complete` are lifted
verbatim out of `handle_headers_pipeline`, which now calls both. No
behaviour change on the message-driven path.
What is NOT established: why the first drain opportunity — the message
that completed segment 25 — did not promote anything. No error is logged
anywhere near it and no early return in the current code fits the
evidence. This makes the pipeline able to recover from that miss; it
does not explain the miss. Worth a `RUST_LOG=dash_spv::sync::block_headers=trace`
reproduction.
The regression test fails without the change, at the assertion that the
tip advanced.
cargo test -p dash-spv --lib # 539 passed
cargo clippy --all-targets + cargo fmt --check # clean
|
Pushed a second fix onto this branch ( The stallA testnet wallet restore froze with the whole chain downloaded and none of the tail of it stored: Peers stayed connected and
Why it cannot recover
All 47 checkpoint segments finished downloading by 22:23:12 (segment 25 last). From that instant no further Filter headers, filters and blocks then coasted to a stop over the next four minutes as they consumed the backlog they had been racing ahead on — which is why the symptom looks like it starts at 22:27:20 rather than 22:23:12. The change
What this does not explainWhy the first drain opportunity — the message that completed segment 25 — promoted nothing. No error is logged anywhere near it, and no early return in the current code fits the evidence. This makes the pipeline able to recover from that miss; it does not explain the miss. A Test
Note for reviewersTwo subsystems in one PR is not ideal and I would normally split them. They are here because they are the same defect shape — state that only a network event can advance, with no periodic path to retry it — and because the second was found while validating the first in a real restore. Happy to split if you would rather review them separately. Also worth someone's eye, found while tracing this and not addressed here: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@dash-spv/src/sync/block_headers/manager.rs`:
- Around line 376-385: Extend the test around manager.tick to assert that the
manager reaches SyncState::Synced and that the returned events include
BlockHeaderSyncComplete, in addition to the existing BlockHeadersStored
assertion. Use the existing manager state accessor and SyncEvent variants,
preserving the current promotion checks.
In `@dash-spv/src/sync/filters/sync_manager.rs`:
- Around line 62-77: Update the reconnect guard in handle_new_filter_headers to
use the complete !self.is_idle() predicate instead of checking only
active_batches before calling start_download. Add a regression test covering
WaitForEvents with only a pending batch, verifying the route does not invoke
start_download and avoids the idle assertion.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 2b845b2e-8fbb-49e9-9d19-268d9c4ad7ca
📒 Files selected for processing (4)
dash-spv/src/sync/block_headers/manager.rsdash-spv/src/sync/block_headers/sync_manager.rsdash-spv/src/sync/filters/manager.rsdash-spv/src/sync/filters/sync_manager.rs
| let events = manager.tick(&sender).await.unwrap(); | ||
|
|
||
| assert!( | ||
| manager.tip().await.unwrap().height() > start_height, | ||
| "tick must promote buffered headers into storage" | ||
| ); | ||
| assert!( | ||
| events.iter().any(|e| matches!(e, SyncEvent::BlockHeadersStored { .. })), | ||
| "the promotion must be reported, not done silently" | ||
| ); |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Assert tick-driven synchronization completion.
Line 376 invokes tick, which now calls finalize_sync_if_complete. The test only verifies BlockHeadersStored. Assert SyncState::Synced and BlockHeaderSyncComplete so a regression that stores the last segment but leaves the manager in Syncing fails.
As per coding guidelines, "Write unit tests for new functionality".
Proposed test additions
assert!(
events.iter().any(|e| matches!(e, SyncEvent::BlockHeadersStored { .. })),
"the promotion must be reported, not done silently"
);
+ assert_eq!(manager.state(), SyncState::Synced);
+ assert!(
+ events
+ .iter()
+ .any(|e| matches!(e, SyncEvent::BlockHeaderSyncComplete { .. })),
+ "tick must report completion after storing the final segment"
+ );📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| let events = manager.tick(&sender).await.unwrap(); | |
| assert!( | |
| manager.tip().await.unwrap().height() > start_height, | |
| "tick must promote buffered headers into storage" | |
| ); | |
| assert!( | |
| events.iter().any(|e| matches!(e, SyncEvent::BlockHeadersStored { .. })), | |
| "the promotion must be reported, not done silently" | |
| ); | |
| let events = manager.tick(&sender).await.unwrap(); | |
| assert!( | |
| manager.tip().await.unwrap().height() > start_height, | |
| "tick must promote buffered headers into storage" | |
| ); | |
| assert!( | |
| events.iter().any(|e| matches!(e, SyncEvent::BlockHeadersStored { .. })), | |
| "the promotion must be reported, not done silently" | |
| ); | |
| assert_eq!(manager.state(), SyncState::Synced); | |
| assert!( | |
| events | |
| .iter() | |
| .any(|e| matches!(e, SyncEvent::BlockHeaderSyncComplete { .. })), | |
| "tick must report completion after storing the final segment" | |
| ); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@dash-spv/src/sync/block_headers/manager.rs` around lines 376 - 385, Extend
the test around manager.tick to assert that the manager reaches
SyncState::Synced and that the returned events include BlockHeaderSyncComplete,
in addition to the existing BlockHeadersStored assertion. Use the existing
manager state accessor and SyncEvent variants, preserving the current promotion
checks.
Source: Coding guidelines
| // | ||
| // The condition is `is_idle()` — the whole invariant `start_download` | ||
| // asserts — and not `active_batches` alone. They are not the same | ||
| // question: `on_disconnect` calls `requeue_in_flight`, which moves the | ||
| // pipeline's in-flight slots to *pending*, and a batch that finished | ||
| // downloading leaves `active_batches` while its verified output waits | ||
| // in `pending_batches` and its blocks wait in the tracker. Any of those | ||
| // three outlasting the last active batch used to fall through to | ||
| // `start_download` and trip its `debug_assert!(is_idle())` — an abort | ||
| // in any build with debug assertions on. | ||
| // | ||
| // Resuming is safe for all of them: this path sets `Syncing`, and | ||
| // `tick` in that state runs `send_pending`, `store_and_match_batches` | ||
| // and `try_process_batch` unconditionally. `send_pending` is a no-op | ||
| // when the pipeline has nothing queued. | ||
| if !self.is_idle() { |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Use the complete idle predicate in the other reconnect route.
dash-spv/src/sync/filters/manager.rs Lines 1078-1087 still tests only active_batches. If pending_batches, tracker state, or filter-pipeline state survives without an active batch, that route calls start_download. start_download then fails debug_assert!(is_idle()).
Replace that guard with !self.is_idle(). Add a regression test that calls handle_new_filter_headers from WaitForEvents with only a pending batch.
Proposed fix
- if !self.active_batches.is_empty() {
+ if !self.is_idle() {
self.filter_pipeline.extend_target(tip_height);
return Ok(vec![]);
}As per coding guidelines, “Write unit tests for new functionality.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@dash-spv/src/sync/filters/sync_manager.rs` around lines 62 - 77, Update the
reconnect guard in handle_new_filter_headers to use the complete !self.is_idle()
predicate instead of checking only active_batches before calling start_download.
Add a regression test covering WaitForEvents with only a pending batch,
verifying the route does not invoke start_download and avoids the idle
assertion.
Source: Coding guidelines
|
This PR has merge conflicts with the base branch. Please rebase or merge the base branch into your branch to resolve them. |
github-actions
Bot
added
the
merge-conflict
2 participants
The bug
FiltersManager::start_syncdecides whether in-flight work survived a disconnect by asking one question:start_download, which the other branch falls into, asserts a stricter one:and
is_idle()is four conditions —active_batches,tracker,pending_batchesandfilter_pipelineall empty. The guard checked one of them.They come apart routinely:
on_disconnectdeliberately preserves the first three (its own doc says so) and callsrequeue_in_flight(), which moves the pipeline's in-flight slots to pending — making the pipeline non-idle rather than restoring it.active_batcheswhile its verified output waits inpending_batchesand its blocks wait in the tracker.Any of those outlasting the last active batch sends a reconnect through the guard and straight into the assert.
Why it matters
In a build with debug assertions on this aborts the host process.
debug_assert!is not a no-op there, and the iOS profile pairs it withpanic = "abort", so there is no unwinding either.It has already happened in the field. A TestFlight build of the Dash iOS wallet (9.0.0/25, iPhone17,2, iOS 26.6) died with
SIGABRTin exactly this frame:The app had been running 33 minutes with
Role: Non UI— backgrounded, where iOS tearing down and restoring networking makes the disconnect/reconnect cycle routine rather than exotic.The fix
The guard now asks
is_idle()— the same predicate the assert states.is_idlebecomespub(super)so the decision and the assertion cannot drift apart again.Resuming is safe for all four cases, not just the one the guard covered: the resume path sets
Syncing, andtickin that state runssend_pending,store_and_match_batchesandtry_process_batchunconditionally.send_pendingreturnsOk(0)when the pipeline has nothing queued, so a manager whose only surviving state is a pending batch or a tracked block still gets drained by the ticker instead of being wedged.Deliberately not
reset_for_rescan()beforestart_download: that would also clear the stateon_disconnectdocuments itself as preserving, throwing away verified batches and forcing a re-download of work already done.Test
test_start_sync_resumes_when_only_pending_batches_surviveputs a verified batch inpending_batcheswithactive_batchesempty, asserts the two predicates disagree at that moment, and drivesstart_sync.It fails without the guard change, panicking at the assert — the production crash reproduced as a unit test.
Note for reviewers
tickhas the same narrow predicate inlet has_pending_work = !self.active_batches.is_empty();, used to decide whether to tick whileSynced. It cannot abort — the worst case is that a surviving pending batch waits for a state change instead of being drained — so I left it alone rather than widen the diff. Worth a look by someone who knows whether that path can strand work.Also: #921 and #902 both change
start_download's signature (requests→network). Neither touches this guard, but whichever lands first will make the other a trivial conflict here.Summary by CodeRabbit