SnodePool: don't let an undersized refresh wipe a good cache - #127
Open
mpretty-cyro wants to merge 5 commits into
Open
SnodePool: don't let an undersized refresh wipe a good cache#127mpretty-cyro wants to merge 5 commits into
mpretty-cyro wants to merge 5 commits into
Conversation
`_on_refresh_complete` handed whatever it computed straight to `_update_cache` with no size check, so a refresh that succeeded but produced too few nodes replaced a perfectly good cache. The multi-request path is the easy way in: it keeps only nodes present in at least `cache_min_num_refresh_presence_to_include_node` of the responses, and that intersection can come back near-empty or empty. The result is a client with nothing to route through until the next refresh lands. This is also what made the recently-fixed post-refresh-callback use-after-free reachable — an empty cache is what sent the deferred `get_swarm` back round to re-register itself mid-iteration. Gate on `cache_min_size` (and reject empty outright, so a client that opts out with 0 still can't be left cacheless) and treat a shortfall as a failed refresh: keep the existing cache and retry with the same backoff as an unparseable response. That threshold is already the codebase's "too small to be usable" line — the fallback-pool loader rejects below it, and `refresh_if_needed` triggers a refresh below it. Note the trade-off: on a network with genuinely fewer than `cache_min_size` nodes we now retry indefinitely rather than serving from an undersized pool. `refresh_if_needed` already spins in that situation, so such a deployment needs `cache_min_size` lowered either way. The retry block is hoisted into a `discard_and_retry` lambda rather than duplicated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Mirrors the captures of the `call_later` it schedules, so both lambdas in `_on_refresh_complete` have the same lifetime story rather than one holding a raw `this` next to one holding a weak ref. The inner lambda now copies that weak ref instead of recomputing `weak_from_this()`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mpretty-cyro
enabled auto-merge
August 10, 2026 05:29
jagerman
reviewed
Aug 10, 2026
Member
|
One possible change here (although I'm not fully convinced either way) would be for this to be a ratio than than an absolute number. For example, 0.5, so that you don't accept an update if it drops by more than 50%. One downside of that approach is that if some service node maliciously fed you a huge (> double the size) fake list, would you move to it and then stop accepting the legitimate list? |
Co-authored-by: Jason Rhinelander <jason@imaginary.ca>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
[pro_live] seeding died with `AttributeError: module 'base' has no attribute
'round_datetime_to_next_day'`, failing every [pro_live] case on any branch.
Session-Pro-Backend a6ff556 ("base: day-rounding is test scaffolding, so it
lives in tests") moved the helper out of `base` and into the backend's own
tests, because the backend rounds nothing to a day — proof expiry lands on a
per-account grid and every other instant is stored as it happened. So there is
no stable backend function to switch to; `round_datetime_up_onto_offset_grid`
would compute the same thing with a one-day period, but it is pendulum-based
where this script is stdlib datetime, and it keeps the coupling that broke.
Inline it instead. Third time this script has been broken by a backend internal
moving (see ec84015 for the obfuscated-account-id one), so it should stop
reaching for them where it can compute the value itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #105. That PR fixed the use-after-free in the post-refresh callbacks; this fixes the thing that made it reachable.
The bug
_on_refresh_completehanded whatever it computed straight to_update_cachewith no size check:So a refresh that succeeded but produced too few nodes replaced a perfectly good cache. The multi-request path is the easy way in — it keeps only nodes present in at least
cache_min_num_refresh_presence_to_include_nodeof the responses, and that intersection can come back near-empty or empty. The client is then left with nothing to route through until the next refresh completes.This is also what made the #105 use-after-free reachable: an empty
_snode_cacheis what sent the deferredget_swarmback round to re-register itself mid-iteration, reallocating the vector being walked.The fix
Gate on
cache_min_sizebefore updating, and treat a shortfall as a failed refresh — keep the existing cache and retry with the same backoff already used for an unparseable response.cache_min_sizeis already this codebase's "too small to be usable" line, so the gate is consistent rather than a new policy:snode_pool.cpp:510)refresh_if_neededtriggers a refresh below it (snode_pool.cpp:1064)Empty is rejected outright as well, so a client that opts out with
cache_min_size = 0still can't be left cacheless.The retry block is hoisted into a
discard_and_retrylambda rather than duplicated — it's the same clear-results / bump-failure-count / exponential-backoff / relaunch sequence in both places.Trade-off worth a look
On a network with genuinely fewer than
cache_min_sizenodes we now retry indefinitely instead of serving from an undersized pool. I think that's acceptable becauserefresh_if_neededalready spins in that situation, so such a deployment needscache_min_sizelowered either way — but it is a behaviour change and worth a second opinion, particularly for small local devnets.Testing
New
[network][refresh_min_cache_size]case driving_on_refresh_completewith encoded responses. Verified it fails without the gate:and passes with it. It also asserts the test's own encoder round-trips through
process_snode_cache_binfirst, so a fixture bug can't masquerade as a pass.Full suite green: 130 cases, 25,755,856 assertions.