Keep leaf nodes spread across files when pruning the candidate set

[StefanZoerner]

Refs #75

Problem

With functions included as leaf candidates (840e1f4), umoria produces 770
candidates, which crosses LEAF_REDUCTION_THRESHOLD (400). The pruning in
topo_sort.py then discards everything that is a dependency of anything
else, leaving only components nothing calls — 27 of 770, covering 11 of 46
source files. The node set is then small enough to skip LLM clustering
entirely and fall back to whole-repository mode.

The two mechanisms work against each other: the better function inclusion
gets, the more reliably the threshold is crossed and the harder the pruning
cuts.

Change

Compare file coverage before and after the pruning. If it drops below half,
discard the pruned set and cap the full candidate list instead, picking
entries round-robin across files so no file is dropped while another
contributes dozens.

Also raises the pruning log from DEBUG to INFO — previously there was no
indication at normal verbosity that a reduction had happened — and stops
rejecting identifiers that merely contain a word like invalid.

Effect on umoria

	before	after
leaf nodes	27	400
files covered by entry points	11	46
modules created	0 (fallback)	46
source files mentioned in the generated docs	12 of 77	31 of 77

Tests

pytest tests/ --ignore=tests/smoke_test_mcp.py: 48 passed, 2 failed. Both
failures are in test_gitignore_filtering.py and occur on main as well.

[anhnh2002]

Thanks for looking into this, and sorry for the slow review. I went through the change carefully and want to explain why I'm hesitant about the fallback before we go further.

The leaf nodes from get_leaf_nodes are meant to be the in-degree-0 roots of the dependency graph, i.e. the entry points nothing else depends on. They're not supposed to cover every file. The idea is that they seed the clustering step and each module agent, and the agent then explores downstream from those roots (it gets the full file for each core component plus read_code_components to pull in dependencies). So a root set that only touches 20% of files isn't a bug by itself; the other 80% is supposed to be reached through exploration.

The new fallback works against that. When coverage drops below 0.5 it stops using roots and instead ranks each file by descending in-degree, so it picks the most-depended-upon nodes first. Those are hubs and shared helpers, which is pretty much the opposite of an entry point. On a CLI-shaped repo with a few main functions as the true roots, we'd end up handing the clustering step utility code as "core components" and dropping the actual entry points. That flips which branch runs on a 50% threshold, so the same repo could get very different module trees depending on which side it lands on.

That said, I suspect you hit a real problem to motivate this. If the docs were missing large parts of a repo, my guess is the exploration side is the weak link rather than the root selection: the agent has to guess component IDs from file content, and there's no tool that lists a component's dependencies. Could you share which repo you were testing on? If that's the issue, I'd rather fix it in a way that keeps roots as the seeds, e.g.: Give the module agent a tool that returns the direct dependencies of a component, so exploration is reliable instead of inferred.

Happy to discuss this direction. Thanks again for digging into this.

[StefanZoerner]

Thanks for taking the time. That explanation is exactly what I was missing. I had interpreted “leaf nodes” as an attempt at coverage, not as in-degree-0 roots that seed the exploration.

My fallback sorts by descending in-degree and thus picks hubs and shared helpers, which is pretty much the opposite of an entry point — you're right about that. And the 0.5 threshold was admittedly somewhat arbitrary (experimental).

The repository I’ve been testing extensively is umoria, which contains 77 C++ files, 765 functions, and 5 structs. I used a script to search through the Markdown output generated by CodeWiki for source file names. The documentation mentioned 12 of the 77 files using qwen3-coder-flash over several runs, with minor variations; see the table above in the PR text. This aligns with your assumption that the weak point lies in the exploration.

Your idea of a tool that returns a component's direct dependencies sounds great. I'd be happy to test it against umoria (and others) as soon as you have it ready, and I'll gladly report on the file coverage before and after.

I am closing this PR and opening two smaller ones for the parts that are independent of the root selection. The identifier filter in filter_leaf_nodes rejects names that contain “error,” “exception,” “failed,” or “invalid” as a substring. Consequently, handleInvalidInput is silently skipped. And the reduction step is logged at the DEBUG level. A reduction from 770 to 27 nodes, as in umoria, leaves no trace at the normal log level.

[StefanZoerner]

I am closing this PR and opening two smaller ones for the parts that are independent of the root selection. The first one is #102 for the identifier filter.