Repository navigation
fix(artifacts): every XML data file becomes config keys — 42,768 keys on ThingsBoard, most of them not configuration #210
Description
Problem
On ThingsBoard v4.0 (4131 Java files), a plain -a 1 run emits a 503 MB analysis.json and takes 7m39s. The symbol table is not the problem — it is 4131 modules / 4131 types / 28,763 callables, and only ~20 MB of module source text. The artifact layer is:
| count | |
|---|---|
| artifacts | 4772 |
| config keys | 42,768 |
| dependencies | 626 |
The top config-key sources are not configuration files at all:
1332 pom.xml
805 application/src/main/resources/thingsboard.yml
615 application/src/main/data/lwm2m-registry/10326.xml
525 application/src/main/data/lwm2m-registry/3421.xml
471 application/src/main/data/lwm2m-registry/10242.xml
453 application/src/main/data/lwm2m-registry/12.xml
The LwM2M registry files are static data resources, one per object type. ArtifactDiscovery's catch-all rule new Rule("*.xml", "xml", List.of("unknown")) classifies every XML in the repository as an artifact, and ConfigKeys.extractXml then flattens every element path into a JConfigKey. A data file with hundreds of elements becomes hundreds of "configuration keys".
453 XML artifacts are discovered this way; a handful of them account for thousands of keys.
Why this is a design question, not a bug
The XML flattener is documented in ConfigKeys as "deliberately the simplest scheme that could work", and for a genuine web.xml or Spring bean file it is right. The gap is upstream, in discovery: there is no notion of an XML file that is data rather than configuration, so the flattener is handed files it should never see.
Deciding which files count as configuration is a judgement call and should not be made by widening or narrowing a regex in passing.
Scope boundary
Constrain which artifacts reach config-key extraction, and/or bound what a single artifact may contribute. Does not change the JSON or graph shape, the id grammar, or any existing key that is genuinely configuration.
Explicitly out of scope: the --emit neo4j OOM (#209). That has its own cause and its own fix; this issue makes its input smaller but is not a prerequisite.
Options worth weighing (not a decision)
- A per-artifact key cap, with the artifact's
extractionmarkedpartialwhen it trips — bounds worst-case output without deciding what configuration means. - Restrict the
*.xmlconfig path to recognized basenames/roots (web.xml,*-context.xml,pom.xml, …) rather than every XML found. - Treat a
src/main/data/-style path as data and skip extraction while keeping the artifact in the inventory, so the file is still recorded but not flattened. - Leave discovery alone and let a consumer filter on
namespace— cheapest, but ships the 503 MB.
Caveats and known risks
- Whatever rule is chosen will exclude something a user considers configuration. Prefer a rule that keeps the artifact in the inventory and only skips flattening, so nothing disappears from the file listing.
pom.xmlcontributing 1332 keys is arguably correct — it really is configuration — so a fix that targets "big XML" rather than "data XML" would throw away real signal.--no-artifact-textalready exists and reduces payload, but it emptiessourceand nullsvalue; it does not reduce the key count.
Definition of done
- A fixture with a data-shaped XML file demonstrates the chosen bound, failing before and passing after
- The
daytrader8config-key count is unchanged, proving genuine configuration is untouched - ThingsBoard v4.0
-a 1output size and key count recorded before/after in the PR - Full suite green apart from the pre-existing Docker-dependent
CodeAnalyzerIntegrationTest
Activity
fix(artifacts): flatten only configuration XML, not every XML in the …