cleanup: remove datafusion + integrate SketchIndex into sketch_db

[zzylol]

Summary

Removes the datafusion-backed query path (engines/physical, engines/logical, the datafusion_summary_library crate, all DF-specific tests, and the controller-side query_language adapter) and folds the new sid-keyed SketchIndex files into stores/sketch_db/.

Why: the controller crate already owns logical and physical plan layers (controller/src/algebra/{lower,physical,plan}.rs + intent_algebra/lower.rs + language_logical_plan/). Keeping a parallel datafusion-LogicalPlan stack alive cost an extra translation hop, ~5800 lines of code, and a test surface that papered over the divergence. The warm-tier sketch path landed in #122 doesn't need datafusion at all — it reads SketchIndex.query_range directly.

Scope: 53 files changed, 96 insertions, 11,436 deletions.

What was deleted

Path	What
crates/datafusion_summary_library/	8 files — sketch-aware datafusion ExecutionPlan operators
asap-query-engine/src/engines/physical/	7 files — datafusion ExecutionPlan adapters
asap-query-engine/src/engines/logical/	2 files — DF logical plan_builder
asap-query-engine/src/tests/datafusion/	11 test files exercising the deleted engines
controller/src/query_language/datafusion/	small adapter
datafusion = "43" + datafusion_summary_library	from asap-query-engine/Cargo.toml + workspace Cargo.toml

What was moved

• asap-query-engine/src/stores/sketch_index.rs → asap-query-engine/src/stores/sketch_db/sketch_index.rs
• asap-query-engine/src/stores/epoch_columnar.rs → asap-query-engine/src/stores/sketch_db/epoch_columnar.rs
• All cross-file imports rewritten to the new sketch_db:: paths.

In-source surgery

• engines/simple/engine.rs: deleted methods that imported SessionContext, physical_plan::collect, record_batch_to_result_map, engines::logical::plan_builder::build_*. The Phase 5 SketchIndex warm-tier hook (with_sketch_index + the classify branch in execute()) is preserved verbatim — only its imports were rewritten to sketch_db::sketch_index::*.

• stores/sketch_db/simple_map_store/per_key.rs: query_disk_parts and EpochSource::snapshot_sealed_epoch now panic!("datafusion-dependent path removed; ingest/persistence still under refactor") — both fire only on persistence-enabled runs (the agreed-upon breakage). Marked // TODO: replace with non-datafusion path.

• 8 controller files (query_language/{mod,language,language_ast,tests}.rs, language_logical_plan/{plan,tests}.rs, algebra/physical.rs, types_v2.rs): deleted DF-referencing variants, match arms, trait impls.

Verification

• cargo build --release -p query_engine_rust — clean (3 dead-code warnings)
• cargo build --release -p controller — clean (5 unused-import warnings on pre-existing items)
• cargo test --release -p query_engine_rust --lib — 792 passed, 2 failed. The 2 failures are tests::schema_timeline_dispatch_tests::*, both pre-existing flakes in the warm-tier dispatch path. 8 persistence integration tests are filtered out (they fire the TODO panics).
• No datafusion strings remain except in TODO comments + panic messages (per_key.rs:670/678/999/1005, simple/engine.rs:2180-2181).
• ls asap-query-engine/src/stores/sketch_db/{sketch_index,epoch_columnar}.rs → both exist; top-level stores/ no longer has them.

Follow-ups

• Replace persistence TODO panics with a non-datafusion warm-tier serializer (likely the asap_sketchlib proto codec).
• The per-Capability sketch reducer (warm-tier query evaluator) is in flight in a parallel branch — replaces the all-Hit CapabilityMiss returned by SimpleEngine::execute with real sketch evaluation from SketchIndex.query_range.

🤖 Generated with Claude Code