Official repo for the ACSOS 2021 paper on how to manage many deep learning models at the edge!
-
Updated
Sep 29, 2021 - Python
#
inference-serving
Here are 10 public repositories matching this topic...
NexusKV is a model-state intelligence layer for inference systems.
python rust golang distributed-systems inference radix-tree model-state kv-cache inference-serving llm-inference ai-infrastructure state-reuse
-
Updated
Aug 21, 2026 - Python
Local inference serving with adaptive batching, benchmark sweeps, and regression gates for batching tradeoffs.
python benchmark machine-learning real-time latency websocket load-testing inference regression-testing performance-testing model-serving quality-gates mlops fastapi inference-serving llm-inference adaptive-batching
-
Updated
May 20, 2026 - Python
Reproducible Apple-Silicon (Apple MLX) deployment and one-token verification kit for the FreeToken edge-native MoE serving API — pinned, loopback-only, schema-validated evidence.
macos benchmark reproducibility mlx mixture-of-experts research-reproducibility apple-silicon inference-serving llm-inference freetoken
-
Updated
Aug 29, 2026 - Python
Reactive LLM inference.
inference incremental-computation inference-optimization kv-cache inference-serving llm llm-inference kvcache-reuse recurrent-memory
-
Updated
Sep 10, 2026 - Python
Machine-readable companion to the IEEE OJ-CS survey 'Semantic Caching and Response Reuse for Large Language Model Services: A Survey' (Chukkapalli, Mishra, Naik, 2026): 21-work evidence matrix, systematic-search log, proposed benchmark trace schema, stdlib-only contract validator, and CPU pilot. Code MIT; data CC-BY-4.0.
-
Updated
Jul 11, 2026 - Python
cost-aware-inference-cluster: FastAPI inference-serving prototype with Redis queues, worker heartbeats, dynamic batching, autoscaling simulation, and local Docker Compose benchmark evidence.
docker kubernetes redis grafana prometheus autoscaling ml-infrastructure fastapi dynamic-batching inference-serving
-
Updated
Jun 29, 2026 - Python
GPU-aware LLM serving runtime with async scheduling, dynamic batching, streaming, admission control, metrics, and mock/llama.cpp/vLLM backends.
-
Updated
Jul 21, 2026 - Python
GSM artifact for exact online GNN inference serving with degree-based scheduling and memory-aware batch division.
scheduling cuda memory-management graph-neural-networks gnn dgl inference-serving online-inference graph-inference
-
Updated
Jun 15, 2026 - Python
Add this topic to your repo
To associate your repository with the inference-serving topic, visit your repo's landing page and select "manage topics."