feat: add on-demand models with shared LoRA and exclusive training

cookbook/client/server/on_demand_training.yaml

@@ -0,0 +1,40 @@
+# On-demand training: applications stays the public configuration entry.
+training:
+  allocation_timeout_s: 3
+  startup_timeout_s: 1800
+  shutdown_timeout_s: 60
+  sweep_interval_s: 5
+applications:
+  - name: model-qwen
+    route_prefix: /api/v1/model/Qwen/Qwen3-1.7B
+    import_path: model
+    args:
+      model_id: ms://Qwen/Qwen3-1.7B
+      backend: transformers
+      device_type: cuda
+      modes:
+        lora:
+          num_devices: 2
+          device_mesh: {device_type: cuda, dp_size: 2}
+          min_replicas: 1
+          max_loras: 5
+          max_r: 32
+          idle_timeout_s: 1800
+          target_modules: all-linear
+        full:
+          num_devices: 4
+          device_mesh: {device_type: cuda, dp_size: 4}
+          min_replicas: 0
+          idle_timeout_s: 1800
+  - name: model-small
+    route_prefix: /api/v1/model/Qwen/Qwen3-0.6B
+    import_path: model
+    args:
+      model_id: ms://Qwen/Qwen3-0.6B
+      backend: megatron
+      device_type: cuda
+      modes:
+        lora:
+          num_devices: 2
+          device_mesh: {device_type: cuda, dp_size: 2}
+          max_loras: 5

docs/design/on-demand-training.md

@@ -0,0 +1,169 @@
+# On-demand training and device reuse
+
+The catalog, controller, allocation, and SDK routing are implemented. Review fixes
+are under local validation. Real-device reruns are deferred at the user's request;
+see the [validation record](validation/on-demand-training/README.md).
+
+## Scope and requirements
+
+A fixed device pool serves a catalog larger than its resident capacity. The demand
+unit is a training handle; the allocation unit is a model instance on N devices.
+Sampling retains its existing deployment lifecycle. This change does not introduce
+resource waiting queues, preemption, training batching, or automatic checkpointing.
+
+| Requirement | Behavior |
+| --- | --- |
+| R01 | Support the existing Transformers and Megatron training backends. |
+| R02 | Administrators declare model sources, modes, parallelism, and N devices. |
+| R03 | New mode specifications default to lazy startup; `min_replicas` keeps resident capacity. |
+| R04 | Initialize residents before accepting dynamic training; all instances use Ray resources. |
+| R05 | An instance reserves N devices on one node; independent bases do not share that reservation. |
+| R06 | A model may offer both LoRA and full training through separate instances. |
+| R07 | Fill compatible LoRA instances first; expand when slots and devices permit. |
+| R08 | Each full handle exclusively owns an instance, subject to user quota. |
+| R09 | Capacity failures are terminal; callers explicitly issue a new create to retry. |
+| R10 | New specifications use business inactivity, defaulting to 1800 seconds. |
+| R11 | Heartbeats, polling, health, and information queries do not renew business inactivity. |
+| R12 | Remove an empty dynamic instance immediately; retain clean resident capacity. |
+| R13 | Expired handles stay invalid; users explicitly recreate. No checkpoint is resumed implicitly. |
+| R14 | Worker failure or controller restart invalidates affected handles without replaying training. |
+| R15 | Retain the existing instance queue and backend execution gate. |
+| R16 | Keep `applications`, old model arguments, and SDK methods; use one controller. |
+| R17 | Every create gets a distinct training ID; identical adapter labels cannot reattach old state. |
+| R18 | Validate both backends and modes on real devices before marking the PR ready. |
+
+## Configuration and compatibility
+
+The public entry point remains `applications` with `import_path: model`.
+The [configuration example](../../cookbook/client/server/on_demand_training.yaml)
+shows `args.modes.lora/full`. Each mode sets `num_devices`, `device_mesh`,
+`min_replicas`, `max_replicas`, and `idle_timeout_s`. LoRA additionally sets
+`max_loras`, `max_r`, and its existing backend options.
+
+- New modes default to `min_replicas=0`, `idle_timeout_s=1800`, five LoRA slots,
+  and maximum LoRA rank 32. Full capacity is one handle.
+- Mesh world size must equal N; N and capacities are positive integers. Timeouts
+  must be finite and positive. Minimum replicas cannot exceed the maximum.
+- `modes` cannot be mixed with legacy single-mode settings or deployment replicas.
+- Legacy `train_mode`, device counts, mesh, strategy, LoRA parameters, and environment
+  variables normalize into the same runtime catalog.
+- Existing deployment replica bounds keep their resident-minimum/instance-limit
+  meaning. Explicit `args.min_replicas` and `args.max_replicas` take precedence.
+- **Legacy `adapter_timeout` remains a session heartbeat timeout.** It is not converted
+  to business inactivity. `adapter_max_lifetime` remains an absolute handle lifetime.
+  Their existing defaults are 1800 and 36000 seconds respectively.
+- A legacy configuration enables business inactivity only by explicitly setting
+  `args.idle_timeout_s`. Its heartbeat and absolute lifetime bounds still apply.
+- The Native SDK and server must upgrade together for the internal training-ID
+  handshake. Native and Tinker retain their existing initialization differences.
+
+Controller allocation/startup/shutdown/sweep bounds default to 3/1800/60/5 seconds
+under `training`. Base startup and user initialization each receive a bounded
+startup window; waiting for a base does not consume the user's initialization window.
+The create future covers both phases without allowing late success.
+
+## Ownership and lifecycle
+
+`TrainingCatalog` performs pure configuration lookup. `TrainingController` is the
+single lifecycle writer. `RayInstanceDriver` performs Ray/Serve effects, and
+`ModelManagement` executes backend operations on the existing queue.
+
+The controller is a named actor owned by the launcher, not a detached service that
+another launcher adopts. Initialization failure cleans and terminates that actor.
+While its owner is alive, a controller restart starts a new epoch and invalidates
+old handles. Runtime constructors register an incarnation once; Serve cannot
+silently replace a process while preserving its old training state.
+
+The state lock covers decisions and commits. Model startup, user initialization,
+cleanup, and resident rebuilding run outside it. A resident rebuilding for one
+model/mode does not block another specification or the sweep loop.
+
+Every instance owns a named Ray Placement Group with one accelerator/CPU bundle
+and `STRICT_PACK`. The allocator bounds `pg.ready()`, obtains actual device IDs
+through a scheduled probe, and injects that allocation into workers without a
+second reservation. Worker, application, registry, and PG names identify their
+server namespace, controller epoch, and instance ID.
+
+A model has a stable CPU router. Creating/releasing handles enters the controller;
+training requests route to their fixed instance. Internal model lifecycle HTTP
+routes explicitly reject direct creation or unloading. Backend workers do not
+restart or replay stateful calls automatically.
+
+| Record | States |
+| --- | --- |
+| Training | PENDING → ACTIVE → CLOSING → RELEASED / EXPIRED / FAILED / CANCELLED |
+| Instance | ALLOCATING → STARTING → READY; RESETTING for resident full reset; DRAINING → STOPPED |
+| Unconfirmed shutdown | QUARANTINED, with devices/quota still owned until death is confirmed |
+
+## Creation, isolation, and expiry
+
+Create identity is `(client type, token, session, create key)`: Native uses its UUID
+and Tinker uses `model_seq_id`. Identical retries return the original future;
+changed parameters are rejected. Pending users count toward quota and LoRA slots.
+Compatible pending creates share a base startup. Full handles never share an instance.
+
+A failed user initialization closes only that user. Its slot remains reserved while
+queued or late initialization may still execute. Cleanup follows initialization in
+the serial lifecycle queue, and a late result cannot publish an ACTIVE handle.
+Worker death, base startup failure, or unsafe backend execution can fail an entire
+instance; a create RPC timeout alone is not evidence of that failure.
+
+Controller tickets protect accepted operations from expiry. The runtime exposes
+queued/executing membership through the compute worker's public interface. Sweep
+reconciles both uncommitted admission and committed operations whose completion
+notification was lost. A terminal future alone does not establish that a background
+call stopped. Reconciled activity uses the original completion time, not sweep time.
+
+Native and Tinker keep bounded sequence retry receipts per training stream. An
+expired-sequence fence survives while the handle is live, so expiry cannot cause an
+old optimizer operation to run again. Retained receipts tolerate out-of-order
+requests; claims are released only when submission never created a future. Retiring
+the handle removes its stream ledgers. Redis enumeration uses the existing SCAN API.
+
+Expiry closes inactive users only after their operations drain. Business inactivity
+uses executed operations; heartbeat-only clients expire under the new policy.
+Legacy heartbeat and maximum-lifetime checks retain their original meanings.
+
+The last user leaving closes a dynamic instance. Clean residents keep their base;
+full residents reset weights, optimizer state, gradients, and template hooks before
+another user is admitted. A resident modified by an instance-level patch is rebuilt
+when empty. A cleanup failure cannot release a slot or kill another active LoRA user
+merely because its cleanup RPC timed out.
+
+Shutdown kills owned workers, deletes the owned Serve app, removes the PG, confirms
+worker/resource termination, and then retires quota. Failed confirmation preserves
+ownership and retries in the background. Recovery never clears a shared registry or
+shuts down another component's Ray cluster. Launcher shutdown retains its normal
+Serve teardown behavior.
+
+## Acceptance
+
+| Case | Required observation |
+| --- | --- |
+| A01 | Startup/query/health on a lazy catalog loads no base or training PG. |
+| A02 | Concurrent first creates coalesce into the required LoRA slot count. |
+| A03 | Full LoRA slots expand with devices available; exhausted capacity fails terminally. |
+| A04 | Both backends and SDKs isolate LoRA tenants and exclusive full handles. |
+| A05 | Disjoint allocations are released and reused by another model. |
+| A06 | Fragmented resources across nodes cannot satisfy one instance. |
+| A07 | Heartbeats do not renew new-policy business inactivity; legacy semantics remain unchanged. |
+| A08 | Idle LoRA cleanup preserves an active tenant's optimizer continuation. |
+| A09 | Queued/running/save operations drain before expiry or release. |
+| A10 | Unsafe background execution retains ownership until workers terminate. |
+| A11 | Partial startup, cancellation, and allocation failure leave correct quota and ownership. |
+| A12 | Worker/controller failure fences old handles and preserves unrelated instances. |
+| A13 | Same-label recreation, old requests, and sequence retries cannot mutate another handle. |
+| A14 | Explicit recreation starts with clean weights and optimizer state; checkpoint restoration is separate. |
+| A15 | Resident full reset leaves a clean base for the next user. |
+| A16 | Ten cross-model reload cycles leave no accumulated workers, PGs, or device memory. |
+| A17 | Invalid mode/config, expired IDs, and cross-user access fail without allocation. |
+| A18 | Related SDK, YAML, and repository CPU regressions pass together. |
+| A19 | Late startup/user initialization cannot publish success or prematurely reuse a slot. |
+| A20 | Memory and Redis recovery invalidate old state and rebuild quota correctly. |
+| A21 | Checkpoint upload needs no base; patched residents rebuild before reuse. |
+
+Review regressions also cover nonblocking resident recovery, launcher initialization
+rollback, lost completion notification, and sequence-receipt expiry. CPU/logical Ray
+resources are not device training evidence. The standalone resource lifecycle is accepted with DP on both backends.
+Checkpoint restoration and FSDP/TP/SP backend repairs are tracked separately in #315.
+Raw device evidence is retained outside the public source tree.

docs/design/validation/on-demand-training/README.md

@@ -0,0 +1,30 @@
+# On-demand training validation
+
+PR #312 is based directly on main. It does not require the checkpoint,
+FSDP optimizer, or Megatron TP/SP repairs in #315. Its backend changes are
+limited to resetting resident full-training state and using the supported
+Transformers configuration import needed to initialize Megatron.
+
+Collected CPU tests under `tests/server/training/` exercise the controller,
+allocation, recovery, and both installed SDKs over HTTP using a mock backend.
+The HTTP and allocation probes run in isolated subprocesses. Existing
+applications YAML startup can also be checked with memory and Redis state.
+
+Standalone validation programs live in `tools/validation/on_demand_training/`.
+The DP resource-lifecycle device suite is explicitly invoked:
+
+```bash
+python -m pytest tools/validation/on_demand_training/device_validation.py -vv
+```
+
+It runs DP=2, TP=1, and no FSDP for both backends and both SDKs in shared
+LoRA/exclusive full modes. Assertions cover actual training and tenant isolation,
+release, cross-model reuse, business-idle reclamation, expired handles, and clean
+recreation. Checkpoint I/O and FSDP/TP repairs belong to #315, not this acceptance.
+Results from older revisions containing #315 do not validate standalone #312.
+
+Device validation requires explicit `TWINKLE_DEVICE_*` configuration, a separate
+evidence directory, and exclusively available devices. Record the exact source
+SHA, import origins, raw JUnit/logs, numerical assertions, and resource cleanup.
+Current results and remaining limitations are recorded in the English PR body;
+results from an older revision are not acceptance evidence for the new head.

docs/source_en/Usage Guide/Server and Client/Server.md

@@ -479,3 +479,9 @@ raises. During that window newly arriving tasks fail fast with a `server`/503 er
 Each persisted future stores its immutable `absolute_deadline` when it is created.
 Cleanup therefore reaches the same decision regardless of which deployment process
 holds the cleanup lease.
+
+## On-demand training and device reclamation
+
+Existing `applications` entries with `import_path: model` remain valid. Set `args.min_replicas: 0` for lazy startup and explicitly set `args.idle_timeout_s` for business inactivity. Legacy `adapter_config.adapter_timeout` remains a heartbeat timeout and `adapter_max_lifetime` retains its absolute lifetime. New `modes` default to 1800 seconds of business inactivity; heartbeats and polling do not renew that deadline; queued and running operations protect training state. Empty instances above the resident minimum are destroyed so other models can reuse their devices. Existing deployment min/max replicas become training instance minimums/limits.
+
+To offer both LoRA and full training for one model, configure `num_devices`, `device_mesh`, `min_replicas`, and `idle_timeout_s` separately under `args.modes.lora/full`. LoRA users share compatible instances; each full-training handle owns an instance. Sampling configuration stays the same. See the [specification](../../../design/on-demand-training.md) and `cookbook/client/server/on_demand_training.yaml`.

docs/source_zh/使用指引/服务端和客户端/服务端.md

@@ -466,3 +466,9 @@ twinkle-server check-config -c server_config.yaml
 任务记录进入终态后，per-replica 准入闸门额外保持关闭的最长时长为 `max(0, Collect_Width × R − T)`：此时记录已是终态，但泄漏的 executor 线程可能仍持有闸门，直到其 `ray.get` 返回或抛出。在该窗口内新到达的任务会以 `server`/503 错误快速失败。
 
 每条持久化 future 在创建时写入不可变的 `absolute_deadline`，因此无论哪个 deployment 进程持有 cleanup lease，清理结果都由任务自身契约决定。
+
+## 按需训练与设备回收
+
+原有 `applications` 和 `import_path: model` 配置继续有效。单模式配置增加 `args.min_replicas: 0` 即可懒启动，`args.idle_timeout_s` 显式设置训练句柄的业务闲置时间。旧 `adapter_config.adapter_timeout` 仍按心跳判断会话存活，`adapter_max_lifetime` 保留绝对寿命；不再把旧字段转换成业务闲置时间。新增 `modes` 的业务闲置默认 1800 秒，心跳和轮询不延长该期限，运行中或已排队的任务保护训练状态。最后一个用户释放或过期后，超出保底数量的实例立即销毁，释放的卡可用于其他模型。已有 deployment 的 min/max_replicas 转成训练实例的保底/上限。
+
+同一模型需要同时提供 LoRA 和全参数训练时，在 `args.modes.lora/full` 内分别设置 `num_devices`、`device_mesh`、`min_replicas` 和 `idle_timeout_s`；不用改成另一套顶层模型目录。LoRA 用户共享兼容实例，全参数用户各占一个实例。采样服务继续使用原配置。详见 [完整规范](../../../design/on-demand-training.md) 与仓库 `cookbook/client/server/on_demand_training.yaml`。

src/twinkle/infra/_ray/ray_helper.py

@@ -71,13 +71,19 @@ def get_config(self, key: str):
             def clear(self):
                 self.config.clear()
 
+        registry_name = os.environ.get('TWINKLE_RAY_REGISTRY_NAME', 'config_registry')
+        registry_namespace = os.environ.get('TWINKLE_RAY_NAMESPACE')
         try:
-            RayHelper._registry = ray.get_actor('config_registry')
+            RayHelper._registry = ray.get_actor(registry_name, namespace=registry_namespace)
         except ValueError:
             try:
-                RayHelper._registry = WorkerRegistry.options(name='config_registry', lifetime='detached').remote()
+                RayHelper._registry = WorkerRegistry.options(
+                    name=registry_name,
+                    namespace=registry_namespace,
+                    lifetime='detached',
+                    num_cpus=0.01 if registry_namespace else 1).remote()
             except ValueError:
-                RayHelper._registry = ray.get_actor('config_registry')
+                RayHelper._registry = ray.get_actor(registry_name, namespace=registry_namespace)
         assert RayHelper._registry is not None
 
     @staticmethod
@@ -359,7 +365,7 @@ def create_workers(worker_cls: Type[T],
                     'scheduling_strategy':
                     PlacementGroupSchedulingStrategy(placement_group=deploy_pg['placement_group']),
                     'name': worker_name,
-                    'namespace': 'default',
+                    'namespace': os.environ.get('TWINKLE_RAY_NAMESPACE', 'default'),
                     'runtime_env': runtime_env,
                     'num_cpus': 0.01,
                 }
@@ -383,6 +389,8 @@ def create_workers(worker_cls: Type[T],
                 if max_concurrency is not None:
                     worker_options['max_concurrency'] = max_concurrency
 
+                if os.environ.get('TWINKLE_RAY_NAMESPACE'):
+                    worker_options.update(max_restarts=0, max_task_retries=0)
                 worker = worker_cls.options(**worker_options).remote(*args, **kwargs)
                 workers.append(worker)
         else:
@@ -417,14 +425,16 @@ def create_workers(worker_cls: Type[T],
                     'scheduling_strategy':
                     PlacementGroupSchedulingStrategy(placement_group=deploy_pg['placement_group']),
                     'name': worker_name,
-                    'namespace': 'default',
+                    'namespace': os.environ.get('TWINKLE_RAY_NAMESPACE', 'default'),
                     'runtime_env': runtime_env,
                     'num_cpus': 0.01,
                 }
 
                 if max_concurrency is not None:
                     worker_options['max_concurrency'] = max_concurrency
 
+                if os.environ.get('TWINKLE_RAY_NAMESPACE'):
+                    worker_options.update(max_restarts=0, max_task_retries=0)
                 worker = worker_cls.options(**worker_options).remote(*args, **kwargs)
                 workers.append(worker)
         return workers