Skip to content

docs(optimizer): reconcile optimization and sketch-bench formulations #652

Description

@milindsrivastava1997

Summary

Reconcile the overlapping optimization/sketch-bench design documents so there is one clearly identified canonical formulation and an explicit record of what is conceptual, proposed, implemented, or blocked.

Scope

Review and update:

  • .design_docs/optimizer-simplified-formulation.md
  • .design_docs/sketch-config-optimization-formulation.md
  • .design_docs/optimizer-mip-formulation.md
  • .design_docs/optimizer-v1-implementation-plan.md
  • .design_docs/sliding-window-multi-window-merge-blocker.md where needed

Reconciliation points

  • Explain the progression from abstract summarization strategies to AQE/config-level facility-location MIP.
  • Establish the canonical definitions of RQE, QE, AQE, candidate config, deployment cost, and assignment cost.
  • Reconcile the two sliding-window models: current single-window/exact-range engine behavior versus the proposed multi-window sliding extension.
  • Reconcile ingestion-memory versus retained/query-working-memory accounting, including sliding retention overhead.
  • State the assumptions and limits of O(1) subtraction, especially when the repeat interval exceeds the window size.
  • Clarify the role and current data gaps of sketch-bench: atomic cost measurements, accuracy lookup, cardinality/distribution profiling, and batch-versus-streaming validity.
  • Align the documented MIP with the implementation status: current per-AQE greedy selection, missing cross-AQE sharing, placeholder label-group multiplier, and stub costs.
  • Mark superseded or historical formulations clearly, or consolidate them into a single canonical design document with links from the others.

Acceptance criteria

  • A reader can identify which document is authoritative for the mathematical formulation and which describes implementation status.
  • The sliding-window feasibility rules are unambiguous and distinguish supported behavior from future extensions.
  • Cost and accuracy inputs are defined consistently across documents.
  • Known unresolved assumptions remain explicitly tracked rather than appearing as settled behavior.
  • All affected internal links and terminology are updated.

References

  • .design_docs/CAPABILITY_MATCHING_DESIGN.md
  • .design_docs/query-engine-fetch-merge-audit.md
  • .design_docs/sliding-window-multi-window-merge-blocker.md

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions