[CI] Update test fetcher script by DN6 · Pull Request #14607 · huggingface/diffusers

DN6 · 2026-08-26T09:21:51Z

What does this PR do?

Migrating PR CI to use the test fetcher by default. Doing this in parts to keep review manageable and testable.

First step is to just update the fetcher script.

The test fetcher determines which tests to run based on which files have been changed in a PR. It works in the following way

First fetch the changed files based on the PR diff
Read all the files under diffusers/src and tests/ to build a reverse dependency map. This map is a dictionary from a file in our repo to every file that would be affected if the file were changed. It is built in the following way
1. For every .py under src/diffusers/ and tests/ return the files it imports. e.g. test_pipeline_flux.py -> [src/diffusers/pipelines/flux/pipeline_flux.py,...]
2. Each file's list is expanded with its dependencies' dependencies, until this list stops growing. e.g. test_pipeline_flux.py's list also contains transformer_flux.py, attention.pyand so on, because these are modules are imported by pipeline_flux.py
3. Invert this forward map. Each file entry's list becomes a key in the reverse map that maps back to the original file. So src/diffusers/pipelines/flux/pipeline_flux.py -> [tests/pipelines/flux/test_pipeline_flux.py, tests/pipelines/flux/test_pipeline_flux_control.py, ...]
Look up the the changed files from the diff against reverse dependency map and filter just the test files to give us a final set of tests.
This set of tests is then bucketed based on test type. e.g. pipelines, models, schedulers, etc.
For models and pipelines. The fetcher parses the test file to learn which Mixin class carries which marker (LoraTesterMixin → lora, MemoryTesterMixin → memory, cpu_offload, group_offload, …). It uses this to construct a feature based matrix for pipeline/models tests so that the actual tests jobs run in feature based buckets. e.g model-core, model-memory, model-attention, pipeline-core, pipeline-lora. Here is an example of a workflow running this new test matrix: https://github.com/huggingface/diffusers/actions/runs/32945379597/job/98105816662

Once the fetcher script is merged into main, we can update the relevant workflows and test them side by side with the existing CI to ensure no gaps. FYI the fetcher tends to over select tests rather than miss them.

… with feature-split matrix output

…so the fetcher does not over-select through shared mixins

…an HF token, not an accelerator

…sformer_flux.py` instead of a separate `utils.py`

DN6 · 2026-08-26T10:20:48Z

                assert False, "Parameters not the same!"

    def test_local_files_only_with_sharded_checkpoint(self):
+        from diffusers.models import FluxTransformer2DModel


Similar issue as AutoPipelines. Importing Flux and SD3 at the top of the file suggests to the test fetcher that these model objects are a dependency for anything that imports from this file.

Based on this observation, we should update the test related sections in our agent guides, then.

Additionally, a simple example of how this affects the dependency map would be helpful. Currently, it feels confusing.

yiyixuxu

thanks!, I left a comment

yiyixuxu · 2026-09-02T22:11:16Z

        return inputs

    def test_pag_cfg(self):
+        from diffusers import AutoPipelineForText2Image


I wonder if we can make the "dependency map" more explainable and its changes visible (e.g. maybe we can keep the map as a file in the reop so changes would show up in PR diffs)

e.g. when someone writes the test_pag_sd.py for the first time in a PR (the way it was in the current main with this import at the module level), they should be able to see how it changes the dependency map and make changes accordingly - so that we can always make sure the map makes sense

Right now it's a black box, no? It's built on fresh, in-memory each time in CI and we cann't see why a given test was or wasn;t selected etc

@yiyixuxu I don't think we can store the dependency map as a repo file because the map will probably update quite frequently. Another challenge is that PRs modifying many files, will likely trigger many tests. So a dependency map explainer might be too hard interpret in those cases.

Proposing a minimal summary section in the workflow that would run on each PR. Example here
https://github.com/huggingface/diffusers/actions/runs/37287991101

It shows how many test files were triggered by a modified file. If the PR contains modified test files (which end up triggering themselves) we just report the count instead of listing every test file in the table.

I think this should help catch weird cases. e.g. if a file outside the core modules (modeling_utils, pipeline_utils, etc.) triggers hundreds of tests, an import problem is the likely cause. We can of course store the changed dependency map as a workflow asset, and perhaps use /diffusers-bot to provide a more detailed explainer if something looks weird.

wdyt?

@DN6
thanks, I think the approach make sense. but this particular case won't show up as anything in the current summary table though. i.e for the PR adds the test_pag_sd.py with the top level import of AutoPipelineForText2Image, I think the test file does not trigger any other test file other than itself, but it makes a lot other files rigger a lot more tests, even though these other files are not modified in the PR) - maybe we can make it more informative? e.g. we can show how many files they depends on (the difference between PR and main) etc so something like this would be visible too

It shows how many test files were triggered by a modified file. If the PR contains modified test files (which end up triggering themselves) we just report the count instead of listing every test file in the table.

But how is this count useful to a contributor / maintainer, though? Could you provide an example?

DN6 · 2026-09-16T08:47:25Z

/diffusers-bot review

sergereview

🤗 Serge says:

Rewrites utils/tests_fetcher.py from regex-based to AST-based import extraction, simplifies the CI matrix to flat {name, paths, markers} entries, and moves several AutoPipeline* imports to function-level to shrink the dependency graph.

Correctness — silent test drop for class-less test files

_feature_groups_for returns [] when class_markers is empty (every selected file has only top-level functions, no ClassDef). _matrix_entries then produces zero entries for that bucket, silently dropping those tests. At least one such file exists today (tests/pipelines/ltx2/test_ltx2_diffusion_decode.py). Fix: default to ["core"] when groups is empty but paths is non-empty.

Robustness — potential KeyError in transitive closure

_merged_nested_deps indexes direct_deps[d] without a guard. If d is absent (file deletion race, conftest outside scanned trees), this raises an unhandled KeyError. The old code had an explicit check. direct_deps.get(d, []) would be safer and consistent with the "over-select rather than miss" philosophy.

serge v0.1.0 · model: claude-opus-4-6 · 21 LLM turns · 24 tool calls · 312.1s · 1094467 in / 6075 out tokens

sergereview · 2026-09-16T08:56:06Z

+    for d in list(direct_deps[m]):
+        if d.endswith("__init__.py"):
+            continue
+        new_deps = set(direct_deps[d]) - set(direct_deps[m])


If d is absent from direct_deps (file deleted between glob and import resolution, or a conftest outside scanned trees), this raises an unhandled KeyError. The old code had an explicit guard.

Use .get() for robustness:

Suggested change

new_deps = set(direct_deps[d]) - set(direct_deps[m])

new_deps = set(direct_deps.get(d, [])) - set(direct_deps[m])

… checkout has no local `main` ref

…_fetcher.py` in the container

…ecorators `tests/conftest.py` derives `NON_CORE_MARKERS` from the markers the `is_*` decorators in `testing_utils.py` apply and marks every test carrying none of them as `core` at collection time. The fetcher imports that set instead of keeping its own list and emits `-m core` for the core jobs in place of the `not (...)` complement expression. Also drop `-v` from the pytest run, which no other workflow passes.

…`; legacy mixins share their names The legacy cache and IP-Adapter mixins in `test_pipelines_common.py` are unmarked but have the same class names as the marked ones, so matching by bare name scheduled a `pipelines-cache` job whose `-m cache` run collected nothing and failed with exit code 5.

fix(ci): harden workflow files flagged on #14607 Co-authored-by: hf-security-analysis[bot] <265538906+hf-security-analysis[bot]@users.noreply.github.com>

…ed test files outside the listed reach

…script

sayakpaul

FYI the fetcher tends to over select tests rather than miss them.

Do we know why?

Can we also see an action where the changeset would be based on other important features like quantization, context-parallelism? What is the workflow there?
Is there a way to get a comparative view of the coverage with and without this PR for a given diff? If the coverage is reduced without any explanation, then that is a concern.
Are we running the GPU tests by default? Or are we asking the maintainers to run them (through a comment)? I think the latter is better because for small and obvious changes, it can save time. Doesn't have to be this PR.
Also, we should test the functionalities implemented, though.

Another question:

https://github.com/huggingface/diffusers/actions/runs/37564579479/job/112610663236?pr=14900 - stuff like this should be caught in the pipeline/model PRs (#14949, for example added Kandinsky6). Would the fetcher resolve this kind of problems?

sayakpaul · 2026-10-07T07:26:03Z

+      # The checkout step only marks the workspace safe in a temporary git config, so git processes the
+      # fetcher spawns in the container would otherwise refuse the repo as dubiously owned.
      run: |
+        git config --global --add safe.directory "$GITHUB_WORKSPACE"


The fetcher is currently in workflow_dispatch. Should it have a pull_request trigger? And then we should also make sure that the fetching script is always run from the base upstream (for security reasons)?

sayakpaul · 2026-10-07T07:30:50Z

        if [ -f test_map.json ]; then
-            keys=$(python3 -c 'import json; fp = open("test_map.json"); test_map = json.load(fp); fp.close(); d = list(test_map.keys()); print(json.dumps(d))')
-            test_map=$(python3 -c 'import json; fp = open("test_map.json"); test_map = json.load(fp); fp.close(); print(json.dumps(test_map))')
+            matrix=$(python3 -c 'import json; print(json.dumps(json.load(open("test_map.json"))))')


Is it possible to have an example matrix for quick references?

sayakpaul · 2026-10-07T07:31:34Z

    - name: Run all selected tests on CPU
      run: |
-        pytest -n 2 --dist=loadfile -v --make-reports=${{ matrix.modules }}_tests_cpu ${{ fromJson(needs.setup_pr_tests.outputs.test_map)[matrix.modules] }}
+        pytest -n 2 --dist=loadfile -m "${{ matrix.markers }}" --make-reports=${{ matrix.name }}_tests_cpu ${{ matrix.paths }}


Possible to include a comment to show example markers and name?

If possible, I would also increase the number of workers to 4. Doesn't hurt I guess.

sayakpaul · 2026-10-07T07:32:14Z

                assert False, "Parameters not the same!"

    def test_local_files_only_with_sharded_checkpoint(self):
+        from diffusers.models import FluxTransformer2DModel


Based on this observation, we should update the test related sections in our agent guides, then.

sayakpaul · 2026-10-07T07:37:50Z

-    AutoPipelineForImage2Image,
-    AutoPipelineForText2Image,


I think it's then better to replace them with their exact pipeline class so that we don't have to do the inline import. Doing it this way also preserves the expected behaviour in the dependency map? Applies elsewhere, too.

sayakpaul · 2026-10-07T07:44:31Z

 from huggingface_hub.utils import HfHubHTTPError, is_jinja_available

-from diffusers.models import FluxTransformer2DModel, SD3Transformer2DModel, UNet2DConditionModel
+from diffusers.models import UNet2DConditionModel


Why is this import not being inlined then?

sayakpaul · 2026-10-07T07:48:51Z

                assert False, "Parameters not the same!"

    def test_local_files_only_with_sharded_checkpoint(self):
+        from diffusers.models import FluxTransformer2DModel


Additionally, a simple example of how this affects the dependency map would be helpful. Currently, it feels confusing.

sayakpaul · 2026-10-07T07:51:30Z

        return inputs

    def test_pag_cfg(self):
+        from diffusers import AutoPipelineForText2Image


It shows how many test files were triggered by a modified file. If the PR contains modified test files (which end up triggering themselves) we just report the count instead of listing every test file in the table.

But how is this count useful to a contributor / maintainer, though? Could you provide an example?

sayakpaul · 2026-10-07T07:55:12Z

 warnings.simplefilter(action="ignore", category=FutureWarning)


+def marker_decorators():


Consider providing an example of how a minimal decorators would look like.

sayakpaul · 2026-10-07T07:56:34Z



+@pytest.hookimpl(tryfirst=True)
+def pytest_collection_modifyitems(items):


What is this for?

…e fetcher summary

…cher summary

DN6 added 3 commits August 26, 2026 10:34

[CI] Rewrite tests_fetcher.py as an AST-based import-graph selector…

341e55f

… with feature-split matrix output

[tests] Import pipeline/model classes inside the tests that use them …

c7c8cc4

…so the fetcher does not over-select through shared mixins

[CI] Give single_file model tests their own fetcher job; they need …

4242566

…an HF token, not an accelerator

github-actions Bot added tests CI size/L PR with diff > 200 LOC labels Aug 26, 2026

[tests] Keep create_flux_ip_adapter_state_dict in `test_models_tran…

bd8d8b8

…sformer_flux.py` instead of a separate `utils.py`

DN6 requested review from sayakpaul and yiyixuxu and a lite review from Copilot and removed request for Copilot August 26, 2026 10:43

Copilot started reviewing on behalf of DN6 August 26, 2026 10:44 View session

DN6 commented Aug 26, 2026

View reviewed changes

yiyixuxu reviewed Sep 2, 2026

View reviewed changes

sergereview Bot requested changes Sep 16, 2026

View reviewed changes

update

b19052c

hf-security-analysis Bot added a commit that referenced this pull request Sep 30, 2026

fix(ci): harden workflow files flagged on #14607

0e6701b

hf-security-analysis Bot mentioned this pull request Sep 30, 2026

fix(ci): harden GitHub Actions workflows (#14607) #14915

Closed

[CI] Detect the main branch by name in tests_fetcher.py; a branch…

7e674e0

… checkout has no local `main` ref

hf-security-analysis Bot added a commit that referenced this pull request Sep 30, 2026

fix(ci): harden workflow files flagged on #14607

a62b881

[CI] Mark the workspace as a git safe directory before running `tests…

dbc8d8a

…_fetcher.py` in the container

hf-security-analysis Bot added a commit that referenced this pull request Sep 30, 2026

fix(ci): harden workflow files flagged on #14607

2c5d4e9

hf-security-analysis Bot mentioned this pull request Sep 30, 2026

fix(ci): harden GitHub Actions workflows (#14607) #14916

Closed

hf-security-analysis Bot added a commit that referenced this pull request Sep 30, 2026

fix(ci): harden workflow files flagged on #14607

452c4f4

hf-security-analysis Bot mentioned this pull request Sep 30, 2026

fix(ci): harden GitHub Actions workflows (#14607) #14919

Closed

hf-security-analysis Bot added a commit that referenced this pull request Oct 5, 2026

fix(ci): harden workflow files flagged on #14607

a81a66f

hf-security-analysis Bot mentioned this pull request Oct 5, 2026

fix(ci): harden GitHub Actions workflows (#14607) #14945

Merged

hf-security-analysis Bot and others added 3 commits October 5, 2026 10:37

fix(ci): harden GitHub Actions workflows (#14607) (#14945)

a8b2c8e

fix(ci): harden workflow files flagged on #14607 Co-authored-by: hf-security-analysis[bot] <265538906+hf-security-analysis[bot]@users.noreply.github.com>

[CI] Count test files in the fetcher summary and add a row for modifi…

a5af804

…ed test files outside the listed reach

Merge remote-tracking branch 'origin/main' into update-tests-fetcher-…

bc5488a

…script

sayakpaul reviewed Oct 7, 2026

View reviewed changes

DN6 added 2 commits October 13, 2026 17:07

[CI] Compare each modified file's test reach to the base commit in th…

bcfeaa9

…e fetcher summary

[CI] Show each modified file's reach on branch and on main in the fet…

7e67c09

…cher summary

	new_deps = set(direct_deps[d]) - set(direct_deps[m])
	new_deps = set(direct_deps.get(d, [])) - set(direct_deps[m])

		warnings.simplefilter(action="ignore", category=FutureWarning)


		def marker_decorators():



		@pytest.hookimpl(tryfirst=True)
		def pytest_collection_modifyitems(items):

Conversation

DN6 commented Aug 26, 2026

What does this PR do?

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DN6 commented Sep 16, 2026

Uh oh!

sergereview Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sergereview Bot Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sayakpaul left a comment •

edited

Loading