Repository navigation
[Bug]: Job processing tasks may raise "database is locked" errors #825
Description
dstack version
master
Python version
3.11
Host OS
macOS 13.4
Host Arch
ARM
What happened?
We occasionally observe "database is locked" errors in sentry:
OperationalError
(sqlite3.OperationalError) database is locked
[SQL: UPDATE jobs SET last_processed_at=? WHERE jobs.id = ?]
In sqlite, a writer transaction locks the db. We have busy_timeout=5000, so other writers can wait up to 5s. The reasons for "database is locked" can be:
- litestream locks the db causing app connections to timeout (unlikely since it runs every second).
- long-running writing transactions lock the db for.
CLI logs
No response
Server logs
No response
Runner logs
No response
Additional Information
No response
Activity
r4victor commentedon Jan 9, 2024
To lower probability of "database is locked" errors, ensure there are no long-running writing transactions. That is, ensure the commit takes place immediately after any DML operation.