Skip to content

[Sad Catapella B4][Performance] Design reconciliation and duplicate checks repeatedly scan all rows #85

Open

Description

@guijacquemet

Baseline and classification

Sad Catapella Batch 4, commit d7a538e56d41ac1093ba79adcad397c21b3dfd32. Confirmed O(N²)-class algorithmic work, not a measured bottleneck.

Evidence

Impact

Slow design loading, reconciliation and confirmation as biological-repeat count grows. This is separate from N-image histogram memory (#77) and mask-filename matching (#84 if numbered accordingly).

Fix/acceptance

Build key→row dictionary once per reconciliation; precompute group→condition counters for conflicts; use collections.Counter/a single seen-key pass for duplicates, preserving error order and strict/lenient semantics. Benchmark 100/1k/10k biological repeats with adversarial group sizes; assert exact issue messages and deterministic order. Numba not appropriate.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions