fix: thread-safe cache writes and feature update handling

[vazarkevych]

Problem

Several race conditions existed in cache and feature-update handling:

• InMemoryFeatureCache had no locking — concurrent reads/writes could corrupt cache entries.
• FeatureRepository.load_features / load_features_async had no fetch coalescing — on a cold cache, many threads/coroutines
  requesting the same SDK payload could all hit the GrowthBook API/CDN at once (cache-miss stampede). Note: Python's GIL does not help
  here, as it is released during the blocking HTTP fetch.
• _feature_update_callbacks was mutated and iterated without a lock — concurrent add/remove/notify could raise RuntimeError: list changed size during iteration.
• FeatureCache.get_current_state returned mutable references to savedGroups and contextualBandits instead of copies.

Changes

This branch was rebuilt on top of current main to avoid the stale conflicts from the original PR:

• InMemoryFeatureCache: threading.Lock around get / set / clear.
• Per-key fetch coalescing in load_features / load_features_async: on a miss, only the first caller fetches for a given
  cache key; others wait and read the freshly-cached value (double-checked under the lock). Cache hits return before acquiring the
  coalescing lock, so the hot path only pays the cache's own short lock. Async locks are keyed by (event-loop id, cache key) to avoid
  reusing a lock bound to a finished loop (the cross-loop asyncio.Lock hang fixed earlier in main).
• Callback delivery: add/remove guarded by a dedicated _callbacks_lock; _notify copies the list under the lock and
  iterates outside it, preventing both the iteration error and deadlocks from slow callbacks.
• FeatureCache.get_current_state: returns dict() copies of savedGroups and contextualBandits.
• The sticky-bucket lock from the original branch is dropped — main has since replaced that cache wholesale with per-attributes
  coalescing, a TTL cache and a per-key save pipeline.

Tests

tests/test_thread_safety.py — sync stampede, async stampede, and callback-list mutation during notification. Each fails on main
without this change. Full suite: 974 passing.

Preserved from current main

• Remote-eval cache keys (_compute_cache_key) are untouched; the remote-eval path keeps its existing _remote_eval_inflight
  coalescing and is intentionally left out of the new lock.
• force_refresh semantics (the re-check under the lock also honors force_refresh, so SSE invalidation still triggers a refetch).
  Concurrent force_refresh callers each refetch, by design.
• SSE invalidation and async client behavior unchanged.

[madhuchavva]

@vazarkevych - thanks for identifying these tricky issues.

I guess the most pressing issue here is: the cache-miss stampede in FeatureRepository.load_features / load_features_async: if many threads or coroutines ask for the same uncached SDK payload at once, they can all hit the GrowthBook API/CDN simultaneously.

and, P2 list includes callback list mutation during notification, sticky bucket boolean lock. but the blast radius is limited. so, I’d salvage this by porting these ideas onto current main: thread-safe InMemoryFeatureCache, snapshot callback delivery, per-key fetch coalescing for sync/async loads, savedGroups copy semantics, and real sticky-bucket lock. There are many changes that went in and we'll need to preserve current remote-eval cache keys, force_refresh, SSE invalidation, and async client behavior.

[madhuchavva]

resolve the conflicts and address the review comments please

[vazarkevych]

resolve the conflicts and address the review comments please

Thanks for the advice — agreed on the priorities. I've rebuilt this branch on top of current main (post remote-eval) rather than merging, so the stale conflicts are gone

[vazarkevych]

@vazarkevych - thanks for identifying these tricky issues.

I guess the most pressing issue here is: the cache-miss stampede in FeatureRepository.load_features / load_features_async: if many threads or coroutines ask for the same uncached SDK payload at once, they can all hit the GrowthBook API/CDN simultaneously.

and, P2 list includes callback list mutation during notification, sticky bucket boolean lock. but the blast radius is limited. so, I’d salvage this by porting these ideas onto current main: thread-safe InMemoryFeatureCache, snapshot callback delivery, per-key fetch coalescing for sync/async loads, savedGroups copy semantics, and real sticky-bucket lock. There are many changes that went in and we'll need to preserve current remote-eval cache keys, force_refresh, SSE invalidation, and async client behavior.

All the items from your list are in: thread-safe InMemoryFeatureCache, snapshot callback delivery, per-key fetch coalescing for both sync and async loads, savedGroups copy, and a real sticky-bucket lock. Remote-eval cache keys, force_refresh, SSE invalidation and async behavior are preserved (the remote-eval path is left out of the new lock since it already coalesces via _remote_eval_inflight).

The only non-obvious bit is the async coalescing lock — it's keyed by (event-loop id, cache key) to avoid reusing a lock bound to a finished loop.

[vazarkevych]

Rebased again - main moved ~130 commits since my last push (now 3.1.1), so the
conflicts were back. Resolved, branch is green.

Two changes from what I described above:

• The sticky-bucket lock is dropped. Main has since replaced that
  boolean-flag cache wholesale with per-attributes inflight coalescing, an
  opt-in TTL cache and a per-key save pipeline, so that conflict is resolved
  in main's favour rather than merged.
• contextualBandits is now copied in get_current_state alongside
  savedGroups - it landed after my original branch and was being handed out
  by reference too.

Also added tests/test_thread_safety.py: sync stampede, async stampede, and
callback-list mutation during notification. Each fails on main without this
change. Full suite 974 passing.

Everything else stands as described - remote-eval cache keys, force_refresh,
SSE invalidation and async behavior unchanged.