Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Changed

- **BREAKING: `--pycg-shard-timeout` is removed** (#145). It bounded PyCG's
fixpoint a second time, by the clock, after `--pycg-max-iter` had already
bounded it by iteration count — and that second bound is what made the output
load-dependent. Anyone passing `--pycg-shard-timeout` must drop the flag;
`--pycg-max-iter` remains the knob that trades analysis depth against
runtime. Note that nothing bounds wall-clock time anymore: `--pycg-max-iter`
caps fixpoint *passes*, not their duration (see Fixed below), so a
pathological shard can still run long at any setting, and
`--pycg-max-iter -1` (run to convergence, no cap) can run indefinitely.
- **BREAKING: the msgpack output format is removed** (#118, TS parity): the
`--format msgpack` CLI choice, the `analysis.msgpack` artifact, the msgpack
serialization mixin on schema models, and the `msgpack` dependency are gone.
Expand All @@ -32,6 +42,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
cp39 manylinux wheels for x86_64 and aarch64, so the source build stops
happening there too.

### Fixed

- **Sharded PyCG no longer drops shards by wall clock** (#145): `--pycg-shard` decided which
shards to keep by wall-clock timeout, so which shards survived depended on
machine load and Ray scheduling. A dropped shard contributed *zero* edges —
three byte-identical invocations over one 2,364-file fixture produced 48,595 /
43,431 / 40,224 call edges, an 11% spread with PyCG's own contribution swinging
44%. Shard outcomes are now decided by PyCG's own convergence
(`has_converged()`): a shard is a runaway when its fixpoint stopped at
`--pycg-max-iter` instead of converging, which is a function of the input
alone.

**`--pycg-max-iter` is not a termination guarantee.** It bounds fixpoint
*passes*, and PyCG consults the cap only between passes, so one expensive
pass escapes it — a single 100-file shard has run for over an hour inside
its pass budget without completing. Removing the wall-clock timeout
therefore removes the only wall-clock bound that existed; that bound was
load-dependent and had to go, but nothing replaces it yet — #148 tracks
a deterministic fallback that routes predicted-expensive shards to
Jedi-only coverage.

Adaptive decomposition is unchanged: a runaway is still re-partitioned at a
tighter file budget to recover recall, and a shard that cannot be split
further now keeps the edges it did produce instead of being discarded — a
capped fixpoint is a sound under-approximation, so those edges are real.

**Scope:** this removes the load-dependent shard-dropping mechanism, which was
the 11% effect. Output is not yet byte-identical across runs: a separate,
much smaller source remains in Jedi's overload resolution for `open()` —
`f.read()` resolves to `_TextIOBase.read` or `_BufferedIOBase.read` depending
on the run, accounting for **0.1–0.3%** of edges on the Flask fixture. That is
tracked as #146 and is not addressed here.

## [1.1.1] - 2026-07-27

### Fixed
Expand Down
483 changes: 203 additions & 280 deletions README.md

Large diffs are not rendered by default.

23 changes: 5 additions & 18 deletions codeanalyzer/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -255,21 +255,6 @@ def main(
min=1,
),
] = 100,
pycg_shard_timeout: Annotated[
int,
typer.Option(
"--pycg-shard-timeout",
help=(
"Per-shard wall-clock timeout in seconds when --pycg-shard is "
"active (default 120). A shard that exceeds this limit is skipped "
"gracefully. PyCG's fixpoint is bimodal: it either converges "
"quickly or diverges indefinitely, so the timeout acts as a final "
"safety net after the file-count ceiling. Set to 0 to disable. "
"POSIX only (macOS / Linux); ignored on Windows."
),
min=0,
),
] = 120,
pycg_shard_strategy: Annotated[
ShardStrategy,
typer.Option(
Expand All @@ -294,8 +279,11 @@ def main(
"changing, but its access-path domain has no convergence bound, "
"so heavy metaclass/mixin code (e.g. an ORM) can loop with each "
"pass costing seconds. The cap returns a sound-but-incomplete "
"call graph instead of looping until the timeout kills it. "
"Set to -1 for PyCG's unbounded run-to-convergence behaviour."
"call graph instead of looping indefinitely. Lowering it does "
"not reliably bound runtime — per-pass cost dominates — and a "
"low cap makes nearly every shard hit it. Set to -1 for "
"unbounded run-to-convergence, which has no wall-clock net, so "
"a divergent shard can then run indefinitely."
),
min=-1,
),
Expand Down Expand Up @@ -391,7 +379,6 @@ def main(
verbosity=verbosity,
pycg_shard=pycg_shard,
pycg_shard_ceiling=pycg_shard_ceiling,
pycg_shard_timeout=pycg_shard_timeout,
pycg_shard_strategy=pycg_shard_strategy,
pycg_max_iter=pycg_max_iter,
entrypoint_rules=tuple(entrypoint_rules or ()),
Expand Down
1 change: 0 additions & 1 deletion codeanalyzer/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -965,7 +965,6 @@ def _get_pycg_call_graph(
skip_tests=self.skip_tests,
shard=self.options.pycg_shard,
shard_ceiling=self.options.pycg_shard_ceiling,
shard_timeout=self.options.pycg_shard_timeout,
shard_strategy=self.options.pycg_shard_strategy,
max_iter=self.options.pycg_max_iter,
using_ray=self.using_ray,
Expand Down
1 change: 0 additions & 1 deletion codeanalyzer/options/options.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,6 @@ class AnalysisOptions:
verbosity: int = 0
pycg_shard: bool = False
pycg_shard_ceiling: int = 100
pycg_shard_timeout: int = 120
pycg_shard_strategy: ShardStrategy = ShardStrategy.JEDI
pycg_max_iter: int = 50
entrypoint_rules: Tuple[Path, ...] = ()
Loading