Transition to efficient background processing #3551

Activity

1. r4victor

   self-assigned this

   on Feb 6, 2026
2. r4victor

   added

   internals

   performance

   on Feb 6, 2026
3. github-actions commented on Mar 12, 2026

   

   github-actionsbot

   on Mar 12, 2026 – with GitHub Actions

   More actions

   This issue is stale because it has been open for 30 days with no activity.

   👎React with 👎1Reacted by Victor Skvortsov
4. github-actions

   added

   stale

   on Mar 12, 2026
5. r4victor

   removed

   stale

   on Mar 13, 2026
6. r4victor commented on Mar 31, 2026

   

   r4victor

   on Mar 31, 2026

   Last edited by r4victor

   CollaboratorAuthor

   More actions

   Benchmarking old tasks vs pipelines (one server replica, SQLite):

   • Provisioning N instances (pending => idle, non-cluster fleet, AWS):
     • N=1: 1m:10s => 45s
     • N=50: ~4m => 1m:30s
     • N=200: ~11 m => ~3-4m.
   • Terminating N instances (idle => terminated, AWS):
     • N=1: 9s => 3s
     • N=50: 1m => 9s
     • N=200: no time => 30s
   • Running N-nodes task on a provisioned fleet (submitted => running / any job running, first time):
     • N=1: 1m:20s => 1m:25s
     • N=50: 3m:40s => 3m:40s
     • N=200: >25m => 4m:30s
   • Running N-nodes task on a provisioned fleet (submitted => running / any job running, second time, no pull):
     • N=1: 50s => 50s
     • N=50: 3m:40s => 2m
     • N=200: no time => 4m:30s
   • Terminating N-nodes task (graceful stop):
     • N=1: 25s => 35s
     • N=50: ~3m => 35s
     • N=200: no time => 45s
   • Running N 1-job runs on a provisioned fleet (submitted => running, includes time to accept N apply requests, 200 idle instances):
     • N=10: >6m => 1m:25s
     • N=50: no time => 1m:30s
     • N=200: no time => 4m
   • Peak memory usage (heap):
     • 10 instances and 10 jobs: 521MB => 532MB
     • 100 instances and 100 jobs 598MB => 624MB
   • Number of used DB connections (Postgres):
     • Provisioning 100 instances: 5-11 average , 12 max => 1-2 average , 7-8 max.
     • Provisioning 100 instances and 100 jobs: 5-11 average, 16 max => 1-2 average, 35 max (short peaks when processing many submitted jobs).

   Benchmarking notes:

   • Most measurements are median of 3.
   • No time means either I didn't want to wait for the benchmarks to finish or there were problems with running the setup.
   • Postgres connections count "active" + "idle in transaction" and don't count "idle" since dstack server uses a client connection pool.

   Observations:

   • Processing >150 resources on a single replica was impractical due to high processing latency. Now a single replica can handle hundreds and possibly thousands of active resources without performance degradation.
   • Provisioning many instances became almost as fast as provisioning one instance – the dstack overhead is minimal. With the sufficient number of instances, processing throughput is limited by the provider's rate limits, e.g. 5 RPS for AWS RunInstances.
   • Some one-resource processing may not improve since it does not depend on processing throughput but on other factors such the number of state transitions and min processing intervals. In cases where pipelines have bigger latency, it's possible to optimize by tweaking processing intervals and implementing "fast" paths to skip unnecessary waits.
   • Provisioning many jobs wasn't limited by pulling but by processing latency so there was no difference between running on instances with and without pulled image before.
   • Running even a small number of runs was slow when the server already had many active resources, e.g. instances, possible due to the session pool limit. Pipelines don't have this issue, e.g. having many idle instances does not effect run processing much.
   • Running one N-node task takes more time than running N one-node tasks. This is expected since multi-node tasks need extra synchronization between jobs.
   • Pipelines may slightly increase memory usage due to higher processing throughput.
   • As expected, most DB connections before pipelines are "idle in transaction" and <3 are "active". With pipelines, there can be both many "active" and "idle in transaction" but less in total on average. Pipelines can have higher spikes during active provisioning. This is not a problem since they are short, can be handled with a client-side pool, and can be further optimized. TooManyConnectionsError should be less likely.

   Benchmarks against #3719.
7. r4victor

   closed this as completedin #3749on Apr 7, 2026