Skip to content

fix(artifacts): every XML data file becomes config keys — 42,768 keys on ThingsBoard, most of them not configuration #210

Closed

Description

@rahlk

Problem

On ThingsBoard v4.0 (4131 Java files), a plain -a 1 run emits a 503 MB analysis.json and takes 7m39s. The symbol table is not the problem — it is 4131 modules / 4131 types / 28,763 callables, and only ~20 MB of module source text. The artifact layer is:

count
artifacts 4772
config keys 42,768
dependencies 626

The top config-key sources are not configuration files at all:

1332  pom.xml
 805  application/src/main/resources/thingsboard.yml
 615  application/src/main/data/lwm2m-registry/10326.xml
 525  application/src/main/data/lwm2m-registry/3421.xml
 471  application/src/main/data/lwm2m-registry/10242.xml
 453  application/src/main/data/lwm2m-registry/12.xml

The LwM2M registry files are static data resources, one per object type. ArtifactDiscovery's catch-all rule new Rule("*.xml", "xml", List.of("unknown")) classifies every XML in the repository as an artifact, and ConfigKeys.extractXml then flattens every element path into a JConfigKey. A data file with hundreds of elements becomes hundreds of "configuration keys".

453 XML artifacts are discovered this way; a handful of them account for thousands of keys.

Why this is a design question, not a bug

The XML flattener is documented in ConfigKeys as "deliberately the simplest scheme that could work", and for a genuine web.xml or Spring bean file it is right. The gap is upstream, in discovery: there is no notion of an XML file that is data rather than configuration, so the flattener is handed files it should never see.

Deciding which files count as configuration is a judgement call and should not be made by widening or narrowing a regex in passing.

Scope boundary

Constrain which artifacts reach config-key extraction, and/or bound what a single artifact may contribute. Does not change the JSON or graph shape, the id grammar, or any existing key that is genuinely configuration.

Explicitly out of scope: the --emit neo4j OOM (#209). That has its own cause and its own fix; this issue makes its input smaller but is not a prerequisite.

Options worth weighing (not a decision)

  • A per-artifact key cap, with the artifact's extraction marked partial when it trips — bounds worst-case output without deciding what configuration means.
  • Restrict the *.xml config path to recognized basenames/roots (web.xml, *-context.xml, pom.xml, …) rather than every XML found.
  • Treat a src/main/data/-style path as data and skip extraction while keeping the artifact in the inventory, so the file is still recorded but not flattened.
  • Leave discovery alone and let a consumer filter on namespace — cheapest, but ships the 503 MB.

Caveats and known risks

  • Whatever rule is chosen will exclude something a user considers configuration. Prefer a rule that keeps the artifact in the inventory and only skips flattening, so nothing disappears from the file listing.
  • pom.xml contributing 1332 keys is arguably correct — it really is configuration — so a fix that targets "big XML" rather than "data XML" would throw away real signal.
  • --no-artifact-text already exists and reduces payload, but it empties source and nulls value; it does not reduce the key count.

Definition of done

  • A fixture with a data-shaped XML file demonstrates the chosen bound, failing before and passing after
  • The daytrader8 config-key count is unchanged, proving genuine configuration is untouched
  • ThingsBoard v4.0 -a 1 output size and key count recorded before/after in the PR
  • Full suite green apart from the pre-existing Docker-dependent CodeAnalyzerIntegrationTest

Activity

  1. added a commit that references this issue on Sep 1, 2026
    7312929
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions