Pin batch-size invariance of safety and judge generation in fp32 (batch_size 1 / 2 / 4 / 8) #922

Activity

1. cemililik

   added this to the Backlog — Wave 4: low, info and watch items milestone on Oct 9, 2026
2. cemililik

   added

   area: eval-gatesCode area: eval-gatesCode area: eval-gates

   severity: lowMinor defect, inaccuracy or rough edgeMinor defect, inaccuracy or rough edge

   wave: 4Roadmap wave 4: low / info / watch, when touching the moduleRoadmap wave 4: low / info / watch, when touching the module

   source: roadmapPlanned or deferred work recorded in docs/roadmap/Planned or deferred work recorded in docs/roadmap/

   on Oct 9, 2026
3. cemililik

   added this to ForgeLM Project Planon Oct 9, 2026
4. cemililik

   added a parent issue on Oct 9, 2026

   • [Epic] Deferred work · conditions and budgets recorded in risks-and-decisions.md #929
5. cemililik

   added

   good first issueGood for newcomersGood for newcomers

   on Oct 11, 2026
6. cemililik commented on Oct 11, 2026

   

   cemililik

   on Oct 11, 2026

   CollaboratorAuthor

   More actions

   Where to start

   Thanks for taking a look. Please confirm the issue is unassigned and comment before starting so work does not overlap.

   Start with these files on development:

   • forgelm/safety/_generate.py
   • forgelm/judge.py
   • tests/test_safety_advanced.py
   • tests/test_judge_functions.py
   • docs/roadmap/risks-and-decisions.md

   Construct a tiny seeded causal model and tokenizer locally in fp32/eval mode, with short prompts of differing lengths and left padding. Assert identical decoded outputs for batch sizes 1, 2, 4 and 8 through both real helpers; do not mock the generation helper or download a Hub model. Include a case sensitive to padding errors. Follow the deferral table's removal contract: reference the implementation now and remove the row only when the fix is on main; name F-PR29-A8-10 in the changelog.

   Test-only task, no bf16 determinism claim or production model benchmark. Needs familiarity with a tiny local Transformers fixture; allow ~2 h, subject to maintainer review of that estimate.

   Run the focused tests:

   pytest tests/test_safety_advanced.py tests/test_judge_functions.py --no-cov

   Run these guards/checks as applicable to the changed files:

   ruff format --check forgelm/safety/_generate.py forgelm/judge.py tests/test_safety_advanced.py tests/test_judge_functions.py
   ruff check forgelm/safety/_generate.py forgelm/judge.py tests/test_safety_advanced.py tests/test_judge_functions.py
   python3 tools/check_bilingual_parity.py --strict
   python3 tools/check_bilingual_code_blocks.py --strict
   python3 tools/check_anchor_resolution.py --strict
   python3 tools/check_tr_links_prefer_mirror.py --strict
   python3 tools/check_usermanual_self_contained.py --strict
   python3 tools/check_usermanual_schema_drift.py --strict

   If user-facing documentation changes, update the existing Turkish mirror in the same PR; keep commands, code and terminal output in English and structurally aligned. English-only contributor standards need no new mirror.

   Open the PR against development with Fixes #922 once every finding is resolved or explicitly closed with a reason; use Refs #922 for partial work and list the resolved finding IDs.
7. cemililik

   added

   help wantedExtra attention is neededExtra attention is needed

   and removed

   good first issueGood for newcomersGood for newcomers

   on Oct 11, 2026