Repository navigation
Qualcomm AI Engine Direct - [GenAI Pipeline] PR5: Model preparation & quantization strategy implementations - #21899
Merged
Merged
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21899
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
Contributor
Author
|
@pytorchbot label "release notes: qualcomm" |
pytorch-bot
Bot
added
the
release notes: qualcomm
Contributor
|
@claude review this PR also review this quant_dtype is a required positional here, but the strategy omits it on the default path expecting export_utils's use_8a8w default — so ExecuTorchQuantizationStrategy() with defaults raises TypeError. Could we give quant_dtype a default here (and create_autospec the adapter in tests)? The bare MagicMock accepts any signature, so this path currently passes CI but would crash in production. |
|
Claude finished @psiddh's task in 2m 6s —— View job PR Review: PR5 — Model preparation & quantization strategies
1. Confirmed:
|
Contributor
|
Accepting to unblock, @qti-horodnic can you take a look at #1 and possibly #2 (its ok if you want to address in upcoming PRs) |
psiddh
approved these changes
Aug 28, 2026
… quantization strategy implementations
Contributor
Author
|
Thanks @psiddh, I've addressed the 2 comments. Should be good to merge |
Contributor
Thanks again for the refactor , it is really looking solid |
psiddh
pushed a commit
that referenced
this pull request
Aug 29, 2026
…ence strategy implementations (#22284) ## Summary This PR implements the __compilation__ and __inference__ strategy implementations, completing the strategy layer. Each strategy delegates to injectable adapter interfaces for testability. ### What's included #### Strategy implementations: - `ExecuTorchCompilationStrategy`: Compiles model to .pte artifacts via `CompilerAdapter` - Validates model, example_inputs, soc_model, and backend_type are present - Passes `example_inputs` explicitly to the adapter (not via `extra_options`) — mirrors the PR5 fix for the quantization stage - Delegates to adapter with example_inputs, compile specs, artifact dir, soc_model, backend_type - Filters `context.extra_options` to a compilation-relevant allow-list - Returns artifact paths and optional `ETRecord` - `ExecuTorchInferenceStrategy`: Runs on-device inference via `DeviceRunnerAdapter` - Validates artifact_paths and adapter are present (no default adapter — device config is required) - Push → execute → pull results flow - Two-step protocol: uses `output_data` from execute if present, falls back to pulled file paths - Returns inference results, performance metrics, and optional `ETDump` #### `DefaultCompilerAdapter`: - `compile_model` signature finalised — mirrors `to_edge_transform_and_lower_to_qnn` argument-for-argument, so per-graph lowering inputs (`compile_specs`, `dep_table`, `passes_job`, `constant_methods`) are explicit parameters rather than `extra_options` keys - Body deliberately raises `NotImplementedError`: the version this package needs is the multi-graph one (graph-name-keyed dicts, single multi-method `.pte` for weight sharing), so it lands with the strategy-level fan-out that calls it rather than being written single-graph and then replaced. Inject a custom `CompilerAdapter` for now. #### Config addition: - `CompilationInputConfig.example_inputs: Optional[Tuple[Any, ...]]` — sourced from the model via `ModelLoaderAdapter.get_example_inputs` #### Orchestrator wiring: - `genai_pipeline.py` `_run_compilation`: passes `example_inputs=model_prep_output.example_inputs` #### Unit tests: - `test_executorch_compilation_strategy.py` - `test_executorch_inference_strategy.py` - `test_compilation_input_config.py` ### PR Review Checklist - All new classes follow single responsibility (one class per file) - Yes. - All dependencies are injected via constructor with sensible defaults - Yes. - All external calls are behind injectable interfaces - Yes (adapter pattern). - Unit tests cover every public method - Yes (100% coverage on strategy impls). - Docstrings on all public classes and methods - Yes. - Type annotations on all function signatures - Yes. - Logging follows the strategy in the LLD - Yes (info on entry/exit, debug per step). ### Related PRs - PR 1: Core data model, engine routing & exceptions: #20409 - PR 2: Strategy interfaces & stage wrappers: #20795 - PR 3: Pipeline orchestrator: #21149 - PR 4: Adapter interfaces, default implementations & dataset providers: #21751 - PR 5: Model preparation & quantization strategies: #21899 - PR 6: Compilation & inference strategy implementations: this pr. - PR 7: Integration & E2E tests: pending. ## Test plan ### Run only tests added in this PR: ``` python -m pytest \ backends/qualcomm/genai_pipeline/tests/strategies/compilation/ \ backends/qualcomm/genai_pipeline/tests/strategies/inference/ \ backends/qualcomm/genai_pipeline/tests/configs/test_compilation_input_config.py -v ``` ### Run only this PR's tests with coverage: ``` python -m pytest \ backends/qualcomm/genai_pipeline/tests/strategies/compilation/ \ backends/qualcomm/genai_pipeline/tests/strategies/inference/ \ backends/qualcomm/genai_pipeline/tests/configs/test_compilation_input_config.py \ --cov=backends/qualcomm/genai_pipeline/strategies/compilation \ --cov=backends/qualcomm/genai_pipeline/strategies/inference \ --cov=backends/qualcomm/genai_pipeline/configs/compilation_input_config \ --cov-report=term-missing ``` Result: ``` Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------------------------------------------------------------------------------------- backends/qualcomm/genai_pipeline/strategies/compilation/compilation_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/compiler_adapter.py 11 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/executorch_compilation_strategy.py 45 0 10 0 100% backends/qualcomm/genai_pipeline/strategies/inference/device_runner_adapter.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/inference/executorch_inference_strategy.py 43 0 6 0 100% backends/qualcomm/genai_pipeline/strategies/inference/inference_strategy.py 7 0 0 0 100% ---------------------------------------------------------------------------------------------------------------------------------------- TOTAL 127 0 16 0 100% ``` ### Run all `genai_pipeline` tests: ``` python -m pytest backends/qualcomm/genai_pipeline/tests/ -v ``` ### Run all tests with coverage: ``` python -m pytest backends/qualcomm/genai_pipeline/tests/ \ --cov=backends/qualcomm/genai_pipeline \ --cov-config=backends/qualcomm/.coveragerc \ --cov-report=term-missing ``` Result: ``` Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------------------------------------------------------------------------------------------------- backends/qualcomm/genai_pipeline/configs/compilation_input_config.py 12 0 0 0 100% backends/qualcomm/genai_pipeline/configs/compilation_output_config.py 8 0 0 0 100% backends/qualcomm/genai_pipeline/configs/inference_input_config.py 12 0 0 0 100% backends/qualcomm/genai_pipeline/configs/inference_output_config.py 9 0 0 0 100% backends/qualcomm/genai_pipeline/configs/model_preparation_input_config.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/configs/model_preparation_output_config.py 12 0 0 0 100% backends/qualcomm/genai_pipeline/configs/quantization_input_config.py 13 0 0 0 100% backends/qualcomm/genai_pipeline/configs/quantization_output_config.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/datasets/calibration_data_adapter.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/datasets/default_calibration_data_adapter.py 26 0 6 0 100% backends/qualcomm/genai_pipeline/datasets/default_training_data_adapter.py 14 0 2 0 100% backends/qualcomm/genai_pipeline/datasets/training_data_adapter.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/engine_proxy.py 20 0 4 0 100% backends/qualcomm/genai_pipeline/exceptions.py 20 0 6 0 100% backends/qualcomm/genai_pipeline/genai_pipeline.py 99 7 12 1 93% 190-201 backends/qualcomm/genai_pipeline/pipeline_context.py 52 0 14 0 100% backends/qualcomm/genai_pipeline/pipeline_stage.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/stages/compilation_stage.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/stages/inference_stage.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/stages/model_preparation_stage.py 14 2 0 0 86% 30, 37 backends/qualcomm/genai_pipeline/stages/quantization_stage.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/compilation_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/compiler_adapter.py 11 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/executorch_compilation_strategy.py 45 0 10 0 100% backends/qualcomm/genai_pipeline/strategies/inference/device_runner_adapter.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/inference/executorch_inference_strategy.py 43 0 6 0 100% backends/qualcomm/genai_pipeline/strategies/inference/inference_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/model_preparation/executorch_model_preparation_strategy.py 68 0 14 0 100% backends/qualcomm/genai_pipeline/strategies/model_preparation/model_loader_adapter.py 9 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/model_preparation/model_preparation_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/quantization/executorch_quantization_strategy.py 61 0 18 0 100% backends/qualcomm/genai_pipeline/strategies/quantization/quantization_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/quantization/quantizer_adapter.py 9 0 0 0 100% ---------------------------------------------------------------------------------------------------------------------------------------------------- TOTAL 668 9 92 1 99% ```
psiddh
pushed a commit
that referenced
this pull request
Sep 28, 2026
…on foundation (#22846) ## Summary First PR of __Phase 2__ and the start of __Stream B (compilation)__. Phase 1 added the skeleton, interfaces, stage wrappers, typed configs with stubbed logic. This PR adds the compilation groundwork the rest of Stream B builds on and makes `DefaultCompilerAdapter` actually compile a graph to a `.pte`. Nothing under `examples/` changes. The legacy `llama.py` flow is untouched and stays CI's reference; the two coexist until Phase 3, whose deletion of the legacy path is gated on a parity test added in PR-A2. ### What's included __Two separate name spaces__: `artifact_keys.py`, `graph_names.py` An __artifact key__ names a `.pte` file; a __graph name__ names a method inside one. A hybrid decoder produces one `.pte` (`text_decoder`) holding two methods (`kv_forward`, `prefill_forward`), so these are deliberately distinct key spaces. - Artifact keys are the six the on-device runner already uses for `pte_paths`, so a compiled bundle reaches inference without translation. Plus `DECODE_QDQ_FILENAME`, a *filename* (a `.pt2` the SQNR eval reads, never seen by the runner) and therefore deliberately absent from `ALL_ARTIFACT_KEYS`. - `DECODER_GRAPH_NAMES` is __decode first__, and the order matters: `kv` mode builds no prefill graph, and decode is the authoritative source of the artifact's constant methods. - Both sets duplicate strings from `examples/.../decoder_constants.py` rather than importing them, so this package does not depend on the example scripts. Agreement tests keep them in step; the legacy copies go away with the legacy flow. __`graph_bundle.py`__: `GraphBundle`, the frozen per-graph unit quantization hands to compilation (`module`, `inputs`, `meta`, `quant_io_dtypes`, `modality_inputs`, `executorch_config`) - One decoder is exported several times from the same weights, so the stages after model preparation work on a *set* of graphs. Bundling them means the pipeline threads one `{graph_name: GraphBundle}` map rather than several parallel dicts that can drift apart. - `frozen=True` stops in-place edits across a stage boundary. Immutability stops at the field boundary though, `convert_pt2e` mutates `meta` in place and the docstring says so, so the guarantee is not overread. - `quant_io_dtypes` is the one thing compilation cannot derive: the graph-boundary dtypes follow from the quantization recipe's bit widths. `None` means quantization was skipped. `__post_init__` requires __both keys or neither__, because the tagger indexes both unconditionally so a partial mapping is a `KeyError` at lowering, not a partially-quantized boundary. - `CompilationInputConfig.graphs` is the consumer side, added here; PR-A1 adds the producer side, PR-B2 is the first stage to read it. Defined here so both streams import one definition. __`compilation/compile_spec_builder.py`__: `QnnCompileSpecBuilder`, `resolve_soc_model`, `resolve_backend_type` - Pairs `generate_htp_compiler_spec` / `generate_gpu_compiler_spec` with `generate_qnn_executorch_compiler_spec` behind one `build()`; `llama.py` repeats that branch three times today. - `resolve_soc_model` converts `str` → `QcomChipset`. `PipelineContext` carries the SoC as a string so nothing in the pipeline imports the QNN schema; lowering needs the enum. Per earlier review this belongs in the adapter layer. - `resolve_backend_type` uses a lookup map built from the enum itself, since `QnnExecuTorchBackendType.__str__` already yields the CLI name so no hardcoded table, and a bad name raises `ValueError` listing the valid ones. - `enable_x86_64` is __constructor__ state, not a per-call flag: the emulator supports neither weight sharing nor shared buffers, which is exactly why `llama.py` writes `not args.enable_x86_64` at all three call sites. Hardcoding either on would break the x86 CI run without failing anything locally. - __Trap worth knowing:__ the builder's device default for `shared_buffer` is *on*, while `ControlArgs.shared_buffer` defaults *off* to mirror `llama.py`'s parser. A caller driving the builder from a `ControlArgs` must pass it explicitly. Documented on `build()` and pinned by two tests. __`control_args.py`__: typed bridge from `PipelineContext` to the existing LLM components - Those components read configuration off an `argparse.Namespace` (`args.max_seq_len`, `args.enable_x86_64`, …), so this is a dataclass carrying the same 65 attributes, buildable from a `PipelineContext`. - It __subclasses `argparse.Namespace`__, and that is load-bearing: `QnnConfig.load_config` dispatches on `isinstance(config, argparse.Namespace)`, so a plain dataclass would break the device path. - `build_parser()` derives an `argparse` parser from the same fields, so a CLI entry point stops maintaining a second default table. - Defaults are asserted field-for-field against the real `llama.py` parser by a test, since this is a second configuration surface and drift would be silent. Two fields are explicitly exempt (`train_config`, `lr_config` default to `None` rather than to YAML paths under `examples/`), with a test pinning that the exemption is deliberate. - Documented as __temporary__: fields leave as their consumers move behind pipeline interfaces. __`artifact_paths`: `List[Path]` → `Dict[str, Path]`__ (`CompilationResult`, `CompilationOutputConfig`, `InferenceInputConfig`, `DeviceRunnerAdapter`, `DefaultDeviceRunnerAdapter`) Keyed because the runner addresses artifacts individually and which ones exist varies by model, so position carries no reliable meaning. Absent artifacts are __omitted__, never mapped to `None` so the runner tests key membership to decide whether a model is multimodal. This is the frozen B→A2 interface. __`DefaultCompilerAdapter` implementation__: replaces PR6's `NotImplementedError` - Lowers the graph, converts with the same config the legacy path uses (`MemoryPlanningPass(alloc_graph_input=False, alloc_graph_output=False)` + `BuildQuantIo()`, since with a shared buffer the caller supplies graph I/O addresses from RPC memory), writes the `.pte`, and returns it under the `artifact_key` the caller names keyed by the caller because a 1:1 adapter cannot tell a vision encoder from a text decoder when both arrive as `model`. - `soc_model` and `backend_type` are __validated against `compile_specs`__ rather than trusted. Lowering takes its target only from the specs, so these parameters would otherwise be decorative: a caller could validate ops for one SoC and compile for another and see nothing until the artifact ran on device. The target is read back out of the specs with `flatbuffer_to_option` and a disagreement raises. Specs with no QNN entry skip the check. - Two things stay deliberately outside the adapter, stated in its docstring: __multi-graph grouping__ (fanning out into a multi-method `.pte` is the *strategy*'s job, so the adapter stays a 1:1 wrapper) and __spill-fill sizing__ (a property of the group, computed from the lowered program). Both land in PR-B2. __Two bugs found along the way__ (worth a look, not specific to this feature) - `DefaultDeviceRunnerAdapter.push_artifacts` did `[str(p) for p in artifact_paths]`. Once that is a dict, iteration yields __keys__, so it would have pushed the strings `"text_decoder"`, `"tok_embedding"` … instead of paths. Now `.values()`, matching `EvalBase._get_adb`. This is why the type change could not be a pure annotation edit. - `CompilationInputConfig` imported `CompileSpec` from `executorch.exir.backend.compile_spec`, which does not exist (it is `compile_spec_schema`). Hidden at runtime by `TYPE_CHECKING`; mypy flags it. __Tests__: 6 new files, 8 updated. Beyond the unit tests, `DefaultCompilerAdapter` was run against the real stack (a small module compiled through QNN to a non-empty `.pte` on the x86 path); that takes ~15s so the committed adapter tests mock lowering, while the builder tests keep two real-API cases (specs really are `CompileSpec`s; `use_multi_contexts` + `online_prepare` is rejected rather than masked). ### PR Review Checklist - All new classes follow single responsibility (one class per file) - Yes. - All dependencies are injected via constructor with sensible defaults - Yes. - All external calls are behind injectable interfaces - Yes (adapter pattern; lowering is imported lazily inside the adapter). - Unit tests cover every public method - Yes. - Docstrings on all public classes and methods - Yes. - Type annotations on all function signatures - Yes. - Logging follows the strategy in the LLD - Yes (info on entry/exit, debug per step, warning on degraded paths). - No existing behaviour changed - Yes (no `examples/` files touched; `llama.py` and the wrappers import unchanged). ### Related PRs __Phase 2 flow.__ `PR-A1` and `PR-B1` are independent and can land in either order. `PR-B2` depends on `PR-B1`. `PR-A2` depends on all three, and closes Phase 2 by wiring the runner and adding the parity test that gates the Phase-3 deletion of the legacy flow. ``` PR-A1 ─────────────────────────┐ ├─► PR-A2 PR-B1 (this PR) ──► PR-B2 ──────┘ ``` Phase 1 (merged): - PR 1: Core data model, engine routing & exceptions: #20409 - PR 2: Strategy interfaces & stage wrappers: #20795 - PR 3: Pipeline orchestrator: #21149 - PR 4: Adapter interfaces & default implementations: #21751 - PR 5: Model preparation & quantization strategies: #21899 - PR 6: Compilation & inference strategy implementations: #22284 - PR 7: Integration & E2E tests: cancelled, will be raised after Phase 2 is done. Phase 2: - PR A1: Model registry, `model_lookup`, CLI, `LLMQuantizerAdapter`, multi-graph quantization & encoding reconciliation. Consumes `ControlArgs`, `GraphBundle` and `graph_names` from this PR, but does not depend on it landing first; pending - PR B1: Compilation foundation (__this PR__) - PR B2: Multi-graph lowering, weight sharing, sharding & spill-fill. Depends on this PR; pending - PR A2: Device-runner adapter, pipeline-runner wiring, README & the `llama_stories_260k` E2E parity test. Depends on PR-A1 and PR-B2; pending ## Test plan ### Run only tests added in this PR: ``` python -m pytest \ backends/qualcomm/genai_pipeline/tests/test_artifact_keys.py \ backends/qualcomm/genai_pipeline/tests/test_control_args.py \ backends/qualcomm/genai_pipeline/tests/compilation/ \ backends/qualcomm/genai_pipeline/tests/strategies/compilation/test_default_compiler_adapter.py \ -v ``` Result: ``` 72 passed, 76 subtests passed in 7.55s ``` ### Run all `genai_pipeline` tests: ``` python -m pytest backends/qualcomm/genai_pipeline/tests/ -v ``` Result: ``` 246 passed, 92 subtests passed in 8.08s ``` ### Run all tests with coverage: ``` python -m pytest backends/qualcomm/genai_pipeline/tests/ \ --cov=backends/qualcomm/genai_pipeline \ --cov-config=backends/qualcomm/.coveragerc \ --cov-report=term-missing ``` Result: ``` Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------------------------------------------------------------------------------------------------- backends/qualcomm/genai_pipeline/artifact_keys.py 9 0 0 0 100% backends/qualcomm/genai_pipeline/compilation/compile_spec_builder.py 55 1 16 0 99% 181 backends/qualcomm/genai_pipeline/configs/compilation_input_config.py 13 0 0 0 100% backends/qualcomm/genai_pipeline/configs/compilation_output_config.py 8 0 0 0 100% backends/qualcomm/genai_pipeline/configs/inference_input_config.py 12 0 0 0 100% backends/qualcomm/genai_pipeline/configs/inference_output_config.py 9 0 0 0 100% backends/qualcomm/genai_pipeline/configs/model_preparation_input_config.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/configs/model_preparation_output_config.py 12 0 0 0 100% backends/qualcomm/genai_pipeline/configs/quantization_input_config.py 13 0 0 0 100% backends/qualcomm/genai_pipeline/configs/quantization_output_config.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/control_args.py 154 1 24 1 99% 276 backends/qualcomm/genai_pipeline/datasets/calibration_data_adapter.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/datasets/default_calibration_data_adapter.py 26 0 6 0 100% backends/qualcomm/genai_pipeline/datasets/default_training_data_adapter.py 14 0 2 0 100% backends/qualcomm/genai_pipeline/datasets/training_data_adapter.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/engine_proxy.py 20 0 4 0 100% backends/qualcomm/genai_pipeline/exceptions.py 20 0 6 0 100% backends/qualcomm/genai_pipeline/genai_pipeline.py 99 7 12 1 93% 218-229 backends/qualcomm/genai_pipeline/graph_bundle.py 18 0 4 0 100% backends/qualcomm/genai_pipeline/graph_names.py 8 0 0 0 100% backends/qualcomm/genai_pipeline/pipeline_context.py 52 0 14 0 100% backends/qualcomm/genai_pipeline/pipeline_stage.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/stages/compilation_stage.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/stages/inference_stage.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/stages/model_preparation_stage.py 14 2 0 0 86% 30, 37 backends/qualcomm/genai_pipeline/stages/quantization_stage.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/compilation_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/compiler_adapter.py 12 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/executorch_compilation_strategy.py 45 0 10 0 100% backends/qualcomm/genai_pipeline/strategies/inference/device_runner_adapter.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/inference/executorch_inference_strategy.py 43 0 6 0 100% backends/qualcomm/genai_pipeline/strategies/inference/inference_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/model_preparation/executorch_model_preparation_strategy.py 68 0 14 0 100% backends/qualcomm/genai_pipeline/strategies/model_preparation/model_loader_adapter.py 9 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/model_preparation/model_preparation_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/quantization/executorch_quantization_strategy.py 61 0 18 0 100% backends/qualcomm/genai_pipeline/strategies/quantization/quantization_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/quantization/quantizer_adapter.py 9 0 0 0 100% ---------------------------------------------------------------------------------------------------------------------------------------------------- TOTAL 914 11 136 2 99% ``` ### Confirm the legacy flow is unaffected: ``` python -c " from executorch.examples.qualcomm.oss_scripts.llama.llama import _build_parser, export_llama from executorch.examples.qualcomm.oss_scripts.llama.wrappers import MultiModalManager, HybridAttentionSinkEvictor print('LEGACY IMPORTS OK') " ``` Result: ``` `LEGACY IMPORTS OK` ```
psiddh
pushed a commit
that referenced
this pull request
Oct 28, 2026
…paration and Quantization (#23050) ## Summary First PR of Phase 2 Stream A. This PR makes the GenAI Pipeline runnable for LLM and MLLM models through the front half of the flow: ```text CLI -> model lookup -> source transforms -> model preparation -> dataset-backed calibration -> quantization -> save QDQ model ``` `examples/` and the legacy `llama.py` path remain unchanged. Compilation and device inference remain in Stream B / PR-A2. ## What's Included ### CLI Entry Point: `cli.py` Adds a GenAI Pipeline CLI that resolves model, dataset, and quantization options, builds `PipelineContext`, and invokes model preparation and quantization. FP16 omits quantization; QAT, embedding quantization, and attention sink are rejected for now. The rejected feature will be raised in following PR. ### Registry-Backed Model lookup: `model_lookup.py` Maps a user-facing model name to the model-specific inputs consumed by the pipeline. Callers resolve one registry config, then derive from that same entry: - **Model construction:** component and graph keyed constructors, including decoder decode/quantize graphs, optional prefill graphs, and multimodal embedding variants. - **Source preparation:** component-keyed state-dict weight transforms and module G2G transforms, plus the Hugging Face state-dict loader when needed. - **Pipeline configuration:** component sharding, quantization dtypes and recipe classes, and LLM or MLLM loader/quantizer adapters. This centralizes model-family branching at the registry boundary; the model preparation and quantization stages consume only the derived component and graph keyed configuration. ### LLM and Multimodal LLM model components: `model_components/` Adds pipeline import surfaces for decoder, token embedding, and encoder components over existing `examples` implementations. ### Module G2G Transforms: `source_transform/`: Collects existing `llama.py` transformations into reusable pipeline functions: - Weight transforms. - Model graph-to-graph (G2G) transforms. ### LLM and MLLM Model Loading and Preparation: `strategies/model_preparation/`: `ExecuTorchModelPreparationStrategy` implements the shared preparation flow: 1. Load one weighted module per component and graph variant. 2. Read per-graph metadata. 3. Build model-native example inputs for each graph. 4. Select one weight-sharing module per component. 5. Apply component-level module transforms. 6. Load the tokenizer. 7. Build the model-specific inference helper used by quantization, if any. 8. Optionally export the tokenizer for runtime. 9. Extract a tokenizer chat template, falling back to `extra_options`. `LLMLoaderAdapter` handles text decoders; `MLLMLoaderAdapter` handles multimodal models. The default adapter owns shared tokenizer and transforms logic. ### Dataset stack: `datasets/` Adds typed dataset options, lookup, loaders, calibration adapters, and LLM/MLLM collators. It supports random fallback data, lm-eval samples, JSON messages, Hugging Face chat datasets, and multimodal messages. `generate_calibration_data()` now receives `example_inputs` explicitly because collators require the calibration-graph signature. ### Quantization: `strategies/quantization/`: `ExecuTorchQuantizationStrategy` implements native PTQ for component and graph keyed models: 1. Create one QNN quantizer per component graph. 2. Export every graph variant and prepare it with observers. 3. Initialize observers for deploy graphs. 4. Build real calibration inputs through the calibration data adapter. 5. Run PTQ calibration on graphs that consume real data. 6. Convert prepared graphs to QDQ modules. 7. Propagate encodings from quantize graphs to deploy graphs. 8. Remove quantize-only graphs from the output. Decoder and token embedding use separate quantize and deploy graphs; encoders use one shared graph. Encodings are propagated before lowering. Quantizer creation belongs to the strategy; LLM/MLLM adapters own only model-family-specific export, preparation, conversion, and calibration. ### Quantization Helpers: `quant_utilities.py`: Provides QNN quantizer construction, recipe application, QDQ saving, logits/KV-cache attributes, and encoding propagation. KV-cache override is enabled only when `n_cache_layers` is provided. ### Quantization Output `QuantizationOutputConfig` now returns component and graph keyed `GraphBundle` objects. Each bundle carries the quantized graph module, export inputs, metadata, and optional quantized IO dtypes. Quantize-only graphs are removed after their encodings have been propagated, so downstream compilation sees only deployment graphs. ## Tests Covers CLI/config construction, source transforms, datasets and collators, loader adapters, model preparation, quantization adapters and strategy, and preparation/quantization stage integration. ## PR Review Checklist - All dependencies are injected via constructor with sensible defaults - Yes. - All external calls are behind injectable interfaces - Yes - Hugging Face, `TokenizerWrapper`, `torch.export`, PT2E, QNN quantizer construction, dataset loading, and calibration execution sit behind adapters or lookup-created callables. - Yes. - Unit tests cover the new public behavior - Yes. - Docstrings on public classes and methods - Yes. - Type annotations on function signatures - Yes. - Logging follows the strategy in the LLD - Yes; info for stage-level progress, debug for per-step details, warning for degraded/normalized CLI behavior. No existing behavior changed - Yes - `examples/` is not modified and `llama.py` remains the reference path. - Yes ## Related PRs Phase 2 flow. PR-A1 and PR-B1 are independent and can land in either order. PR-B2 depends on PR-B1. PR-A2 depends on PR-A1 and PR-B2, and closes Phase 2 by wiring the runner and adding the parity test that gates Phase 3 deletion of the legacy flow. ```text PR-A1 (this PR) ────────────────┐ ├─► PR-A2 PR-B1 ───────────► PR-B2 ───────┘ ``` Phase 1 (merged): - PR 1: Core data model, engine routing & exceptions: #20409 - PR 2: Strategy interfaces & stage wrappers: #20795 - PR 3: Pipeline orchestrator: #21149 - PR 4: Adapter interfaces, default implementations & dataset providers: #21751 - PR 5: Model preparation & quantization strategy implementations: #21899 - PR 6: Compilation & inference strategy implementations: #22284 - PR 7: Integration & E2E tests: cancelled, will be raised after Phase 2 is done. Phase 2: - PR A1: Model registry, model_lookup, CLI, Multimodal LLM & LLM model preparation, dataset-backed calibration, component/graph-aware quantization, and encoding reconciliation. [This PR]. - PR B1: Compilation foundation: #22846 - PR B2: Multi-graph lowering, weight sharing, sharding, and spill-fill. Depends on PR-B1. - PR A2: Device-runner adapter, pipeline-runner wiring, README, and E2E parity test. Depends on PR-A1 and PR-B2. ## Test plan ### Run only tests added in this PR: ``` bash python -m pytest \ backends/qualcomm/genai_pipeline/tests/strategies/model_preparation/ \ backends/qualcomm/genai_pipeline/tests/strategies/quantization/ \ backends/qualcomm/genai_pipeline/tests/source_transform/ \ -v ``` Result: ``` log 126 passed, 2 subtests passed in 11.59s ``` ### Run all genai_pipeline tests: ``` bash python -m pytest backends/qualcomm/genai_pipeline/tests/ -v ``` Result: ``` log 238 passed, 13 subtests passed in 14.37s ``` ### Run all tests with coverage: ``` bash python -m pytest backends/qualcomm/genai_pipeline/tests/ \ --cov=backends/qualcomm/genai_pipeline \ --cov-config=backends/qualcomm/.coveragerc \ --cov-report=term-missing ``` Result: ``` log 238 passed, 13 subtests passed in 17.41s ``` ### Confirm the legacy flow is unaffected: ``` bash python -c " from executorch.examples.qualcomm.oss_scripts.llama.llama import _build_parser, export_llama from executorch.examples.qualcomm.oss_scripts.llama.wrappers import MultiModalManager, HybridAttentionSinkEvictor print('LEGACY IMPORTS OK') " ``` Result: ``` log `LEGACY IMPORTS OK` ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements the model preparation and quantization strategy implementations, replacing the
NotImplementedErrorstubs with real logic. Each strategy delegates to injectable adapter interfaces (from PR4) for testability.What's included
Strategy implementations (2 files + 1
__init__fix):ExecuTorchModelPreparationStrategy: 5-step flowload_model→load_tokenizer(viaModelLoaderAdapter)generate_calibration_data(via separately-injectableCalibrationDataAdapter)extra_optionsfallback)model_name,soc_modelrequired)ExecuTorchQuantizationStrategy: Full PT2E single-graph pipeline viaQuantizerAdapterquant_dtype,quant_recipe, and per-channel options viaextra_optionsIterableas calibration data (lists, DataLoaders, generators)training_datais provided (QAT deferred)strategies/model_preparation/__init__.py: adds missingExecuTorchModelPreparationStrategyimport to__all__Unit tests:
test_executorch_model_preparation_strategy.pytest_executorch_quantization_strategy.pytest_default_model_preparation_adapter.pytest_default_model_preparation_adapter.pyPR Review Checklist
Related PRs
Test plan
Run only tests added in this PR:
Run only this PR's tests with coverage:
Result:
Run all
genai_pipelinetests:Run all
genai_pipelinetests with coverage:Result: