Restricted populate with reserve_jobs deletes other workers' reservations and error records #1594

populate(*restrictions, reserve_jobs=True) deletes jobs outside its restriction, including jobs reserved by other workers and error records, once they are older than stale_timeout (3600 s by default).

@schema
class Raw(dj.Manual):
    definition = """
    raw_id : int32
    ---
    v : float32
    """

@schema
class Proc(dj.Computed):
    definition = """
    -> Raw
    ---
    w : float32
    """

    def make(self, key):
        self.insert1(dict(key, w=2 * (Raw & key).fetch1("v")))

Raw.insert([dict(raw_id=i, v=float(i)) for i in range(3)])
Proc.jobs.refresh()
Proc.jobs.reserve({"raw_id": 1})            # another worker is computing raw_id=1
Proc.jobs.reserve({"raw_id": 2})
Proc.jobs.error({"raw_id": 2}, "failed")    # raw_id=2 failed earlier
# ... more than an hour later ...
Proc.populate({"raw_id": 0}, reserve_jobs=True)
Proc.jobs.proj("status").to_dicts()         # [] — the reservation and the error record are gone

Effect. Workers that populate different restrictions in parallel delete each other's long-running reservations, so another worker can reserve the same key and compute it twice. Error records are deleted, so their history is lost and the failed keys are pended again.

Cause. populate calls self.jobs.refresh(*restrictions). Job.refresh narrows key_source by the restrictions to add new jobs (step 1). It then reuses the narrowed key_source in the stale-job cleanup (step 3), so every job outside the restriction looks as if its key had vanished.

Workaround until the fix ships: populate without a restriction when using reserve_jobs=True, or raise jobs.stale_timeout beyond the longest make().

Fix: the cleanup checks keys against the unrestricted key_source. The restrictions still limit which jobs are added and re-pended.