Skip to content

Jobs: completed_time is truncated to the second, so it can precede reserved_time #1600

Closed

Description

@dimitri-yatsenko

Summary

The jobs table declares reserved_time and completed_time as datetime(3) (jobs.py:167-168). reserve() writes reserved_time as CURRENT_TIMESTAMP(3) (jobs.py:464), but complete() and error() take completed_time from SELECT CURRENT_TIMESTAMP (jobs.py:499, 529), which has no fractional seconds. MySQL truncates it to the whole second, so any job that finishes in the second it was reserved records a completed_time earlier than its reserved_time.

Error records carry no duration, so completed_time - reserved_time is the only measure of how long a failed job ran, and it comes out negative or up to a second short. With keep_completed = True, success records show the same inversion.

Reproduction

DataJoint 2.3.5.dev8 (master, a00b89f), MySQL 8.0.46, Python 3.11:

import time
import datajoint as dj

dj.config.jobs.keep_completed = True
schema = dj.Schema("scratch_job_times")

@schema
class Item(dj.Manual):
    definition = """
    item : int32
    """

@schema
class Result(dj.Computed):
    definition = """
    -> Item
    ---
    value : int32
    """

    def make(self, key):
        time.sleep(0.2)
        self.insert1(dict(key, value=key["item"]))

Item.insert({"item": i} for i in range(5))
Result.populate(reserve_jobs=True)
for job in Result.jobs.to_dicts(order_by="item"):
    flag = "  <- completed before reserved" if job["completed_time"] < job["reserved_time"] else ""
    print(job["item"], job["reserved_time"], job["completed_time"], flag)

Output:

0 2026-10-17 21:37:31.124000 2026-10-17 21:37:31   <- completed before reserved
1 2026-10-17 21:37:31.335000 2026-10-17 21:37:31   <- completed before reserved
2 2026-10-17 21:37:31.546000 2026-10-17 21:37:31   <- completed before reserved
3 2026-10-17 21:37:31.754000 2026-10-17 21:37:31   <- completed before reserved
4 2026-10-17 21:37:31.966000 2026-10-17 21:37:32

Expected behavior

completed_time is recorded at the column's declared precision and on the same clock as reserved_time, so completed_time >= reserved_time always holds: SELECT CURRENT_TIMESTAMP(3) in complete() and error(). On PostgreSQL, CURRENT_TIMESTAMP is the start of the transaction rather than the statement; the statement-time adapter expression proposed in #1584 would serve here too.

Activity

  1. added
    bugIndicates an unexpected problem or unintended behavior
    on Oct 17, 2026
  2. added this to the v2.3.5 milestone on Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIndicates an unexpected problem or unintended behavior

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions