CASR: Cluster-Aware Model Selection and Sequential Refinement for Code Generation

CASR is a test-free, prompt-level ensemble method for improving first-try code generation. It combines:

1. Offline cluster-aware model selection on the APPS training split to choose a complementary pool of code LLMs.
2. Sequential multi-model refinement on the target benchmark, where models refine the previous step's candidate programs without seeing target-task tests or execution feedback.

The repository contains the code and compact artifacts needed to reproduce the model-selection procedure, run CASR on HumanEval+, MBPP+, and APPS, evaluate the baselines used in the experiments, and reproduce the main ablations.

Method overview

1. Semantic labeling of APPS train

Each APPS training problem is assigned one dominant problem type using google/gemma-4-E2B-it. The seven types are:

• string_parsing
• array_sequence
• math_number_theory
• graph_tree
• data_structures
• greedy_dp_optimization
• simulation_implementation

The precomputed labels are included in apps_labels.jsonl. cluster_apps_samples.py converts these labels into the subset files and manifest under apps_label_clusters_primary/.

2. Cluster-balanced model selection

Candidate models are evaluated once on the APPS training split. For a candidate model $M_j$, the selector measures how many previously unsolved problems it adds in every semantic cluster. The cluster-balanced marginal gain is

$$\mathrm{Gain}(M_j \mid S) = \frac{1}{L} \sum_{\ell=1}^{L} \frac{\Delta_\ell(M_j \mid S)}{|G_\ell|}$$

where $S$ is the current selected pool, $G_\ell$ is semantic cluster $\ell$, and $\Delta_\ell$ is the number of newly solved tasks contributed by $M_j$ in that cluster.

Each cluster therefore receives equal weight regardless of its size. The objective favors models with complementary strengths rather than simply selecting the models with the highest standalone APPS-train accuracy.

The main three-model pool selected by this procedure is, in order:

1. Qwen/Qwen3.6-27B
2. google/gemma-4-31B-it
3. Qwen/Qwen3.6-35B-A3B

The order matters: the first model is the strongest standalone member and is used for the final CASR integration step.

3. Sequential refinement

The main experiments use (K=3) selected models and (N=3) total steps.

• Step 0 — initialization: every selected model independently generates one solution from the original problem.
• Step 1 — refinement: every selected model receives the original problem plus all candidates from step 0 and generates one improved solution.
• Step 2 — final integration: only the strongest selected model receives all candidates from step 1 and produces the single final program.

The target benchmark tests are not used to choose, rank, or refine intermediate candidates. Candidate programs are treated as fallible hints; the problem specification and examples remain authoritative.

The offline selection stage is different: APPS-train programs are executed once to build model pass sets used by the selector.

Repository layout

.
├── run_ensemble.py                    # Main CASR refinement runner
├── run_single_model.py                # One-shot single-model baseline
├── gemma4_apps_labeler.py             # APPS semantic labeling
├── cluster_apps_samples.py            # Build semantic-cluster subsets/manifest
├── apps_run_models_on_clusters.py     # Evaluate candidate models on APPS-train clusters
├── select_cluster_aware_ensemble.py   # Greedy cluster-balanced selector
├── apps_aggregate_cluster_metrics.py  # Aggregate APPS cluster evaluation metrics
├── apps_stratified_cluster_audit.py   # Stratified manual label-audit helper
├── apps_labels.jsonl                  # Precomputed semantic labels
├── apps_label_clusters_primary/       # Cluster manifest and task-id subsets
├── selection_data/
│   ├── apps_train_model_pass_sets.json
│   └── candidate_model_accuracy.csv
├── baselines/
│   ├── self-refine/                   # Code-generation Self-Refine adapter
│   └── reflexion/                     # Code-generation Reflexion adapter
├── evalplus/                          # Modified EvalPlus code used by the experiments
├── stats_evalplus.py                  # Aggregate repeated-run statistics
├── summarize_eval_results.py
├── summarize_ensemble_comparison.py
├── summarize_model_step_accuracy.py
└── evalplus_results/                  # Generated results (not tracked)

Large raw experiment outputs are intentionally not committed. evalplus_results/ is the expected location for target-benchmark runs. The much larger per-model APPS-train outputs can be regenerated under evalplus_results_label_primary/. A compact first-attempt pass-set artifact is included in selection_data/, so the reported model-selection trajectory can be reproduced without the raw APPS-train result directory.

Installation

Create a Python environment and install the core dependencies:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

For models run through vLLM:

pip install -r requirements-vllm.txt

GPU inference requires a CUDA/PyTorch setup compatible with the selected models. Some Hugging Face models may also require accepting their model license and authenticating with Hugging Face.

Reproduce the semantic clusters

The repository already includes the labels and generated cluster files. To rebuild the cluster files from the included labels:

python cluster_apps_samples.py

To regenerate the labels from APPS train:

python gemma4_apps_labeler.py \
  --output-path apps_labels.jsonl \
  --model google/gemma-4-E2B-it \
  --overwrite

Then rebuild the subsets:

python cluster_apps_samples.py

Reproduce model-pool selection

The compact APPS-train pass sets are included, so no model inference is needed for this step:

python select_cluster_aware_ensemble.py \
  --max-models 5 \
  --out-json selection_data/selected_cluster_ensemble.json

The first five selected models should be:

Qwen/Qwen3.6-27B
google/gemma-4-31B-it
Qwen/Qwen3.6-35B-A3B
microsoft/Phi-4-reasoning
Qwen/Qwen3-Coder-30B-A3B-Instruct

The default selector uses the CASR cluster-balanced objective:

task_weight    = 0
cluster_weight = 1

To reproduce the greedy task-coverage selection ablation instead:

python select_cluster_aware_ensemble.py \
  --max-models 4 \
  --task-weight 1 \
  --cluster-weight 0

For the top APPS-train models ablation, take the first (K) rows of selection_data/candidate_model_accuracy.csv.

Regenerating the raw APPS-train pass sets

To rerun a candidate model on all semantic clusters:

python apps_run_models_on_clusters.py \
  --cluster_manifest apps_label_clusters_primary/cluster_manifest.json \
  --model Qwen/Qwen3.6-27B \
  --backend hf \
  --temperature 0.6 \
  --root evalplus_results_label_primary \
  --skip_existing

Repeat this for the candidate models being evaluated. Then run the selector directly from the regenerated raw result files:

python select_cluster_aware_ensemble.py \
  --use-raw-results \
  --results-dir evalplus_results_label_primary \
  --max-models 5

Run CASR

The full refinement prompt is the default. It can also be selected explicitly:

export CASR_REFINEMENT_PROMPT=mbpp_guard_hint

Run the main three-model configuration on HumanEval+:

python run_ensemble.py \
  --dataset humaneval \
  --ensemble_name casr_selected3_run0 \
  --refinement_steps 3 \
  --model_temperature 0.6 \
  --max_new_tokens 6384 \
  --models \
    "Qwen/Qwen3.6-27B|hf|" \
    "google/gemma-4-31B-it|hf|" \
    "Qwen/Qwen3.6-35B-A3B|hf|"

For MBPP+:

python run_ensemble.py \
  --dataset mbpp \
  --ensemble_name casr_selected3_run0 \
  --refinement_steps 3 \
  --model_temperature 0.6 \
  --max_new_tokens 6384 \
  --models \
    "Qwen/Qwen3.6-27B|hf|" \
    "google/gemma-4-31B-it|hf|" \
    "Qwen/Qwen3.6-35B-A3B|hf|"

For APPS test:

python run_ensemble.py \
  --dataset apps \
  --ensemble_name casr_selected3_run0 \
  --refinement_steps 3 \
  --model_temperature 0.6 \
  --max_new_tokens 6384 \
  --models \
    "Qwen/Qwen3.6-27B|hf|" \
    "google/gemma-4-31B-it|hf|" \
    "Qwen/Qwen3.6-35B-A3B|hf|"

The experiments use temperature 0.6, nucleus sampling with top_p=0.95, a maximum of 6384 new tokens, and a 16,384-token model context for the vLLM backend. Repeat runs with different output names such as run0, run1, and run2 to keep independent results separate.

Intermediate generations and evaluation files are written under:

evalplus_results/test/<ensemble_name>/<dataset>/refinement_attempt_<step>/

The final CASR answer is the output of the first model in the final refinement step.

Baselines

Single-model one-shot

CASR uses Qwen/Qwen3.6-27B as the strongest standalone comparison and as its final-step integrator. Run the one-shot baseline with the same decoding settings:

python run_single_model.py \
  --dataset humaneval \
  --model Qwen/Qwen3.6-27B \
  --backend hf \
  --temperature 0.6 \
  --max_new_tokens 6384 \
  --trust_remote_code

Change --dataset to mbpp or apps for the other benchmarks.

Self-Refine

The included adapter follows a generate-feedback-refine loop using the same local model backend. The reported configuration allows up to six attempts and retains the full refinement history.

PYTHONPATH=baselines/self-refine \
python baselines/self-refine/src/codegen_bench/run.py \
  --dataset humaneval \
  --output evalplus_results/baselines/self_refine_humaneval.jsonl \
  --evalplus_path . \
  --model Qwen/Qwen3.6-27B \
  --provider evalplus \
  --backend hf \
  --temperature 0.6 \
  --max_new_tokens 6384 \
  --max_attempts 6 \
  --trust_remote_code \
  --device_map auto \
  --evaluate

Change --dataset and the output path for MBPP+ or APPS.

Reflexion

The Reflexion adapter generates internal tests, executes them, creates a verbal reflection, and uses the most recent reflection for the next attempt. The reported configuration uses up to six attempts, two generated internal tests, and a reflection-memory window of one.

python baselines/reflexion/programming_runs/casr_evalplus_reflexion.py \
  --dataset humaneval \
  --output evalplus_results/baselines/reflexion_humaneval.jsonl \
  --evalplus_path . \
  --model Qwen/Qwen3.6-27B \
  --backend hf \
  --temperature 0.6 \
  --max_new_tokens 6384 \
  --max_attempts 6 \
  --internal_test_count 2 \
  --memory_limit 1 \
  --trust_remote_code \
  --device_map auto \
  --evaluate

Reflexion is not execution-free: execution of its generated internal tests is part of that baseline.

Ablations

Pool size and selection strategy

The model-pool experiments use (K \in {2,3,4}) with the same (N=3) refinement schedule. Generate the desired model list using:

• top APPS-train standalone accuracy,
• greedy task coverage, or
• greedy cluster-balanced coverage,

then pass the selected ordered models to run_ensemble.py.

Refinement-prompt ablation

The following CASR_REFINEMENT_PROMPT values are implemented:

mbpp_guard_hint
no_refinement
no_boundary_operator_emphasis
no_extra_assumptions
no_readability_improvement
no_hidden_tests_mention
no_edge_case_emphasis
no_efficiency_priority

Example:

CASR_REFINEMENT_PROMPT=no_edge_case_emphasis \
python run_ensemble.py \
  --dataset humaneval \
  --ensemble_name prompt_ablation_no_edge_cases \
  --refinement_steps 3 \
  --models \
    "Qwen/Qwen2.5-Coder-14B-Instruct|hf|" \
    "Qwen/Qwen3-14B|hf|" \
    "microsoft/NextCoder-14B|hf|" \
    "nvidia/NVIDIA-Nemotron-Nano-12B-v2|vllm|"

The prompt ablation uses this fixed four-model pool to isolate prompt design; it is separate from the cluster-balanced pool-selection experiment.

Reported main results

Mean Pass@1 over three independent runs:

Method	HumanEval+	MBPP+	APPS
Single model	90.85 ± 1.06	74.51 ± 0.67	57.83 ± 0.48
Self-Refine	91.26 ± 0.35	77.69 ± 0.93	67.37 ± 0.77
Reflexion	91.26 ± 0.93	76.81 ± 1.00	66.10 ± 0.36
CASR	94.72 ± 0.93	80.86 ± 0.15	68.41 ± 0.34

The raw generated result directories are not tracked because they are large. The analysis scripts operate on the regenerated or locally restored evalplus_results/ directory.

Cluster-label audit

A stratified manual audit helper is included in apps_stratified_cluster_audit.py, together with the sampled items and completed audit files:

apps_cluster_manual_audit_sample.json
apps_cluster_manual_audit.tsv

The audit samples 20 problems from each of the seven semantic clusters.

Safety note

APPS evaluation and the Reflexion baseline execute model-generated Python. Run untrusted generated code only in an appropriately isolated environment.

Third-party code

The repository includes modified EvalPlus components and small adapters derived from the Self-Refine and Reflexion implementations. Their corresponding license files are retained in the relevant directories.