If the flowmachine query locker (redis) has somehow got out of sync with the state of the flowmachine query cache in FlowDB, so that a cached query is in known state, then query.invalidate_db_cache() will silently fail to remove the cache table (because query_state_machine.reset() has no effect on a query in known state).
A side effect of this is that flowmachine.core.cache.shrink_below_size reports (in log messages) that the cache is shrinking even if it isn't, because it assumes a successful invalidate_query_by_id run means that the cache table has been dropped and subtracts the table size from its internal record of the cache size, and does not check for cache table existence or re-fetch the cache size from FlowDB between query drops.
I think there are a few issues here:
query.is_stored (which does not touch redis, and directly checks the DB for the corresponding cache table) should always be the source of truth for checking whether a cache record existsquery.invalidate_db_cache() should raise an error if query.is_stored is still True at the end (if drop=True)- I can't think of a good reason not to allow
QueryStateMachine.reset() to reset from known state as well, which would mean an existing cache entry could be cleared even if there's no corresponding query state in redis
flowmachine.core.cache.shrink_below_size shouldn't report the cache size as having shrunk if the cache table wasn't actually dropped - I think this should be covered by the above points (i.e. as long as query.invalidate_db_cache() raises an error if the cache table still exists at the end, then shrink_below_size can trust that the table has been removed).
If the flowmachine query locker (redis) has somehow got out of sync with the state of the flowmachine query cache in FlowDB, so that a cached query is in
knownstate, thenquery.invalidate_db_cache()will silently fail to remove the cache table (becausequery_state_machine.reset()has no effect on a query inknownstate).A side effect of this is that
flowmachine.core.cache.shrink_below_sizereports (in log messages) that the cache is shrinking even if it isn't, because it assumes a successfulinvalidate_query_by_idrun means that the cache table has been dropped and subtracts the table size from its internal record of the cache size, and does not check for cache table existence or re-fetch the cache size from FlowDB between query drops.I think there are a few issues here:
query.is_stored(which does not touch redis, and directly checks the DB for the corresponding cache table) should always be the source of truth for checking whether a cache record existsquery.invalidate_db_cache()should raise an error ifquery.is_storedis stillTrueat the end (ifdrop=True)QueryStateMachine.reset()to reset fromknownstate as well, which would mean an existing cache entry could be cleared even if there's no corresponding query state in redisflowmachine.core.cache.shrink_below_sizeshouldn't report the cache size as having shrunk if the cache table wasn't actually dropped - I think this should be covered by the above points (i.e. as long asquery.invalidate_db_cache()raises an error if the cache table still exists at the end, thenshrink_below_sizecan trust that the table has been removed).