Determine how to do resource allocation

[RaasAhsan]

Hosts have limited resources and therefore they shouldn't be scheduled jobs that would exhaust or leave its resources in high contention.

resources:

• cpus
• memory
• disk space
• network bandwidth?

Each Gemini node should be able to specify what resources it is offering to the cluster capacity. Every Gemini task/job should specify what resources they require and they should be scheduled on a free Gemini node.

could investigate things like cpu sharing/burst

[PipelineBaron]

Memory, disk space and bandwidth are fairly intuitive for most people in terms of requesting a % of them or a raw number requirement. How do we represent shares of CPU "units" though? What's the most intuitive way for users?

[RaasAhsan]

I was thinking it would just be # of CPUs or # of cores/hyperthreads. I guess we should also consider that hardware isn't homogenous, so we could also develop classes/affinity?

Regarding "shared CPU", I was thinking a node could advertise that it has shared CPUs to offer to the cluster. Any job/task that specifies that its OK to run on shared CPUs could get scheduled on one of those machines (doesn't necessarily have to), and the master could even overschedule those nodes. Basically those jobs/tasks are forced to shared CPUs instead of getting allocated a dedicated one.