feat: per-request request_id/token_range pipeline (generate -> native -> host -> ClickHouse) - #19
Conversation
- Add task size support on BackendFuture and submit as budget in submit to host engine.
padded tokens. - DMX host engine shard_rank support for TP. - DMX host engine batch to request breaking. - DMX host engine support for dropping padded tokens. - NOTE: Still require changes from monitoring engine for submit API change. Need to track per request id, track effective start_token_idx and end_token_idx.
…update - dmx_host_utils: fix std::move bug in hook loop (use const ref params) - dmx_host_utils: add inner/outer length validation - dmx_host_engine: clean up submit() signature to use const refs - future_process: fix variable name error (token_end/start_idx) - future_process: add tensor.size(0) vs request_ids.size() validation - future_process: per-request select+narrow+clone, skip empty ranges - future_process: special handling for attn hook token/key dim narrowing - future_process: remove unsqueeze(0) to preserve correct tensor shape - clickhouse_client: update schema comment to 8-column with shard_rank - bindings: expose BackendFuture.size() - native_engine/hooks: fix BackendFuture task size type to pair<int64_t,int64_t>
- engine.py: replace single request_id with per-batch request_id list - engine.py: add _db_state_lock, _auto_batch_group_id, per-request start_idx and finished state - engine.py: shape-based prefill detection (input_ids.shape[1] > 1) as fallback for HF cache modes that pass non-None past_key_values - engine.py: EOS/PAD detection checks last_ids before advancing range (not after), fixing off-by-one row count for finished requests - engine.py: update submit() call to new C++ signature - hook_points.py: pass attention_mask to _register_db_step - generate.py: sync shape-based prefill detection in monitored_forward
- validate_request_id_pipeline.py: E2E test via full DB pipeline
- validates per-request token ranges are contiguous and correct
- validates prefill length matches attention_mask effective length
- validates row count matches effective generated length (EOS-aware)
- validates tensor shape rank for final_logits and attn hooks
- --with-attn-hook: exercises is_attn narrowing path in future_process
- --exercise-eos-path: forces early EOS via LogitsProcessor to validate
finished-request row count
- prompts_varlen_validation.txt: variable-length prompts for padding test
- test_monitoring_engine_request_id.py: unit tests for engine state logic
|
(proj-dmx) nengneng@FROOT-Lab:~/AIPrometheus/HF_Prometheus$ ./tests/run_request_id_tests.sh ========================================
|
|
Follow-up update: removed Reason: in the current runtime model, we do not have multiple workers concurrently mutating the same Python engine instance (single inference owner per engine / per-rank engine separation), so this lock is non-essential for current behavior and only adds complexity. The previously lock-protected fields were:
|
DMI-vLLM-Integration PR #21 merged (squash 23717cb): attach_model accepts layers=LayerSelection(...), applies it to the local specs and the model-wide candidate-rank sets, and stays importable on DMI builds without the layer-range facade. The pin moves old-main 29f26c3 -> 23717cb, picking up both the V2-runner promotion (#19) and the layer range (#21).
DMI-vLLM-Integration PR #21 merged (squash 23717cb): attach_model accepts layers=LayerSelection(...), applies it to the local specs and the model-wide candidate-rank sets, and stays importable on DMI builds without the layer-range facade. The pin moves old-main 29f26c3 -> 23717cb, picking up both the V2-runner promotion (#19) and the layer range (#21).
DMI-vLLM-Integration PR #21 merged (squash 23717cb): attach_model accepts layers=LayerSelection(...), applies it to the local specs and the model-wide candidate-rank sets, and stays importable on DMI builds without the layer-range facade. The pin moves old-main 29f26c3 -> 23717cb, picking up both the V2-runner promotion (#19) and the layer range (#21).
2 participants
Background
This PR finalizes per-request tracking in the monitoring pipeline:
HF generate() -> monitoring engine -> native backend -> host engine -> ClickHouse.Main objective:
v0_host_sideWhat changed
1) Python-side per-request tracking (monitoring engine)
Files:
monitoring/engine.pymonitoring/hook_points.pymonitoring/generate.pyChanges:
_register_db_step(...)now uses:request_idsper active batchstart/endtoken range per requestattention_maskto compute prefill lengths(start, start)and stops growthpast_key_values is Noneinput_ids.shape[1] > 1(for cache-mode edge cases)(model_id, shard_rank, request_ids, token_ranges, cache_dict)2) Native interface alignment
Files:
monitoring/csrc/native_engine.hmonitoring/csrc/native_engine_internal.hmonitoring/csrc/native_engine.cppmonitoring/csrc/hooks.cppmonitoring/csrc/bindings.cppChanges:
BackendFuturenow carriestask_size(token, task_size)for host queue budgetingBackendFuture.size()3) Host-engine side (delta vs
origin/v0_host_side) — for focused reviewFiles:
monitoring/csrc/dmx_host_engine.hmonitoring/csrc/dmx_host_utils.hmonitoring/csrc/dmx_host_utils.cppmonitoring/csrc/future_process.cppmonitoring/csrc/clickhouse_client.hmonitoring/csrc/clickhouse_client.cppmonitoring/csrc/bindings.cppPost-merge fixes/hardening (no architecture redesign):
std::move(request_ids[i])/std::move(token_range_per_request[i])request_ids.size == token_ranges.sizetensor.size(0) == request_ids.sizestart >= end, negative)start_token_idx/end_token_idxvariables for effective range slicingt_future.size()in host enqueue budgetBackendFuture.size()is available from bindingsTests added
Files:
tests/test_monitoring_engine_request_id.pytests/validate_request_id_pipeline.pybenchmark/data/prompts_varlen_validation.txtCoverage:
--exercise-eos-path)--with-attn-hook)Compatibility / notes
shard_rankis preserved in host->DB schema pathshard_rank=0as placeholder (TP/distributed to follow)Review request