Skip to content

fix: regenerate invalidated modules in whole-repository-mode --update runs - #100

Merged
anhnh2002 merged 2 commits into
FSoft-AI4Code:mainfrom
kiboook:fix/incremental-update-whole-repo-mode
Sep 4, 2026
Merged

fix: regenerate invalidated modules in whole-repository-mode --update runs#100
anhnh2002 merged 2 commits into
FSoft-AI4Code:mainfrom
kiboook:fix/incremental-update-whole-repo-mode

Conversation

@kiboook

@kiboook kiboook commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Fixes #99

Problem

--update --compare-to <commit> never regenerates an invalidated module
when the repo went through whole-repository documentation mode (module
clustering skipped because the repo fits in one context window). It
correctly deletes the affected module's .md, then fails every time with
IncompleteGenerationError. See #99 for the full trace of why.

Root cause

generate_module_documentation() computed the processing order only from
first_module_tree.json, which stays {} forever for whole-repo-mode
repos (only the LLM clustering path writes it) — even though
module_tree.json can have real sub-modules a prior whole-repo agent run
inserted. So the regeneration loop had nothing to iterate over for an
invalidated module.

Fix

Extracted the processing-order lookup into
DocumentationGenerator.resolve_processing_order(), which falls back to
module_tree only when first_module_tree produces an empty order. The
normal clustered-repo path is unaffected, since first_module_tree is
non-empty there and the fallback never triggers.

Testing

  • Added tests/test_processing_order_fallback.py (4 unit tests, no LLM/
    network needed) covering: fallback when first_module_tree is empty,
    first_module_tree still wins when non-empty (regression safety for
    the normal path), both-empty doesn't crash, and nested children are
    still walked leaf-first through the fallback.
  • Ran the new tests plus the existing tests/test_overview_structure.py
    against this change — all pass.
  • End-to-end verified against the minimal repro from --update --compare-to never regenerates invalidated modules in whole-repository documentation mode (always ends in IncompleteGenerationError) #99 (tiny
    auth/payment/notification TypeScript fixture, 2 commits): before the
    fix, the run reproducibly failed with IncompleteGenerationError;
    after the fix, generate --update --compare-to <sha> succeeds, the
    changed module regenerates, the two unaffected modules are correctly
    skipped ("✓ Module docs already exists"), and the overview is rebuilt
    referencing all three.

… runs

first_module_tree.json is only ever written by the LLM clustering path,
so it stays {} for any repo small enough to skip clustering
("whole-repository documentation mode"). generate_module_documentation()
computed the processing order exclusively from that file, so once
--update/--compare-to invalidated (deleted) an affected sub-module's .md
via module_tree.json, the regeneration loop had nothing to iterate and
the run always ended in IncompleteGenerationError.

Fall back to module_tree for the processing order only when
first_module_tree is empty. The normal clustered-repo path is untouched,
since first_module_tree is non-empty there and the fallback never
triggers.

Fixes FSoft-AI4Code#99
@kiboook
kiboook force-pushed the fix/incremental-update-whole-repo-mode branch from 54615ec to e27accc Compare August 31, 2026 08:37
…ent-inserted sub-modules

The fallback added for FSoft-AI4Code#99 only kicked in when first_module_tree.json
produced an empty order, so it fixed whole-repository mode but left the
same failure in clustered mode: first_module_tree.json never learns about
the sub-modules agents insert into module_tree.json while documenting a
complex module. Invalidating one of those with --update/--compare-to
deleted its .md (plus the parent and overview), the loop only revisited
the parent, and the run ended in IncompleteDocumentationError.

Order from module_tree.json unconditionally instead. It is always a
superset of the first tree, and modules whose .md already exists
short-circuit in run_module_agent / generate_parent_module_docs, so
plain re-runs still cost no LLM calls. This makes
resolve_processing_order unnecessary, so it is removed.

Tests: replace test_processing_order_fallback.py with
test_processing_order_update.py, which adds fake-backend integration
tests for the clustered nested case, the whole-repo case from FSoft-AI4Code#99, and
a no-op re-run. get_processing_order / is_leaf_module become
staticmethods since they use no instance state.
@anhnh2002

Copy link
Copy Markdown
Collaborator

Thanks for the fix and the clear write-up in #99. I pushed a follow-up commit (fd060a3) to this branch that generalizes it.

Why

The fallback only fires when the order from first_module_tree.json is empty, so it covers whole-repository mode but the same failure still exists in clustered mode. first_module_tree.json never learns about sub-modules agents insert into module_tree.json via generate_sub_module_documentation_tool. If a change lands under one of those, _invalidate_affected_modules deletes the child .md, the parent .md, and overview.md, but the processing order only contains the parent. The parent is rebuilt, the child never is, and the run ends in IncompleteDocumentationError exactly like #99.

What changed

  • generate_module_documentation now orders from module_tree.json unconditionally. It is always a superset of the first tree, and modules whose .md already exists short-circuit in run_module_agent / generate_parent_module_docs, so plain re-runs still make no LLM calls. resolve_processing_order is no longer needed and is removed.
  • tests/test_processing_order_fallback.py is replaced by tests/test_processing_order_update.py: unit tests on get_processing_order plus fake-backend integration tests for the clustered nested case, the whole-repo case from --update --compare-to never regenerates invalidated modules in whole-repository documentation mode (always ends in IncompleteGenerationError) #99, and a no-op re-run. The nested-case test fails against the original fallback logic and passes with this change.
  • get_processing_order / is_leaf_module became staticmethods since they use no instance state.

Not addressed here

Regenerated modules reuse the components list stored in module_tree.json, so a symbol newly added to a changed file is not treated as a core component of the regenerated doc. That is pre-existing on the clustered path as well and is better handled in a follow-up issue.

@anhnh2002
anhnh2002 merged commit 2584854 into FSoft-AI4Code:main Sep 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--update --compare-to never regenerates invalidated modules in whole-repository documentation mode (always ends in IncompleteGenerationError)

2 participants