Skip to content

perf(neo4j): no index on _module, so every incremental purge scans the node store #217

Closed

Description

@rahlk

Problem

BoltWriter's per-module purge matches on _module:

MATCH (x:JCompilationUnit|JType|JCallable|...|JBodyNode) WHERE x._module = $m

There is no index on _module. Schema.INDEXES carries four entries — JCallable.name, JType.name, JAnnotation.name, and the j_code_fts fulltext — and none of them helps here.

So each of those 15 labels is scanned in full, per changed module, twice (once for the edge purge, once for the vanished-node purge). On an incremental push touching N modules that is 30N label scans over the whole node store.

codeanalyzer-typescript indexes exactly this and says why:

CREATE INDEX cannode_module IF NOT EXISTS FOR (n:CanNode) ON (n._module)
"— without this they would scan the whole node store."

It can do so with one statement because every project-owned node there carries a single shared :CanNode label. Java's module-owned nodes merge under six different labels, so it needs one index per label.

Not a correctness problem — the anchoring fixed in #213 is right, and this is purely how fast it executes. It matters at ThingsBoard scale (4131 modules) and not at all on a fixture, which is why no test surfaces it.

Scope boundary

Index declarations in Schema.INDEXES, and whatever derives them. Not in scope: the anchoring itself (#213, already merged), the snapshot path (CypherWriter emits DDL but does not run the purge), or the label vocabulary.

Goals

  • An index on _module for every label in CypherWriter.MODULE_OWNED, derived from the catalogs rather than hand-listed so it cannot drift from the anchor it serves
  • Confirm the planner actually uses them — PROFILE the purge and check for NodeIndexSeek rather than NodeByLabelScan
  • Measure an incremental push before and after on a large repository

Caveats and known risks

  • 15 labels means 15 index entries, which cost write throughput on the initial full push. That trade is probably right for a writer whose whole purpose is repeated incremental updates, but it should be measured rather than assumed.
  • A composite index will not help: the two statements filter on _module alone.
  • Neo4j will not use an index for a label-disjunction pattern in all versions. If MATCH (x:A|B) WHERE x._module = $m does not seek, the statements may need to be issued per label — worth checking before assuming the index is enough.

Definition of done

  • PROFILE output showing an index seek on the purge path
  • Before/after timing of an incremental push on a repository large enough for the difference to be visible

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions