process_submitted_jobs should create InstanceModel before provisioning instances #3665

process_submitted_jobs creates InstanceModel (ComputeGroupModel) models only after provisioning succeeds. This is problematic:

• Possible untracked instances.
• Duplicate instance_num race because the fleet lock cannot be held for a long time.
• Fleet nodes limits are soft because the fleet lock cannot be held for a long time ([Bug]: Can over-allocate instances in fleet past nodes-max when using rest API #3663).
• Master instance determination is implicit (addressed in Implement submitted jobs pipeline #3670).

The solution is to add a process_submitted_jobs step that creates and commits InstanceModel (ComputeGroupModel) "reserved" models. Then provision against those models and either promote them or delete them. Introduce InstanceModel.provisioning_job_id that means that the instance is being provisioned by process_submitted_jobs and ignore it in the instance background task.