Skip to content

fix: Replace DB connection crash with passive reconnect polling - #3155

Merged
amirbiron merged 3 commits into
mainfrom
claude/fix-db-timeout-lJR9K
Apr 20, 2026
Merged

amirbiron merged 3 commits into
mainfrom
claude/fix-db-timeout-lJR9K

Conversation

@amirbiron

Copy link
Copy Markdown
Owner

✨ תיאור קצר

שינוי התנהגות כשל התחברות למסד הנתונים: במקום לעצור את התהליך (SystemExit) לאחר timeout, התהליך יישאר פעיל ויחכה בפולינג פסיבי עד שהחיבור יתחדש. זה מונע CrashLoop ו-Sentry storms בעת הפסקות זמניות של מסד הנתונים.

📦 שינויים עיקריים

  • קוד (Backend)

פירוט:

  • main.py: החלפת SystemExit(1) בלולאת פולינג פסיבית שמחכה לחיבור מחדש

    • הוספת משתנה סביבה DB_RECONNECT_POLL_INTERVAL (ברירת מחדל: 30 שניות)
    • שינוי לוג מ-critical ל-warning כדי לא להעלות אזעקות מיותרות
    • הוספת הערה המסבירה את הרציונל של השינוי
    • תיקון לוג ההצלחה להציג את הזמן הכולל של ההמתנה
  • database/manager.py: הקטנת delay בפונקציית _schedule_background_reconnect מ-30 שניות ל-5 שניות

    • מאפשר ניסיונות חיבור מחדש תכופים יותר בתהליך הרקע

🧪 בדיקות

  • CI Required Checks: 🔍 Code Quality & Security; Unit Tests (3.11); Unit Tests (3.12)
  • השינוי הוא בעיקר בהתנהגות error handling ולא משנה את הלוגיקה הקיימת של חיבור מסד הנתונים
  • בדיקות יחידה קיימות צריכות להמשיך לעבור

📝 סוג שינוי

  • fix: תיקון באג

✅ צ'קליסט

  • הקוד עוקב אחרי הסגנון (Black/isort/flake8/mypy)
  • בדיקות רצות ועוברות
  • אם נוספו משתני סביבה – עודכן docs/environment-variables.rst וגם services/config_inspector_service.py
    • ✅ נוסף DB_RECONNECT_POLL_INTERVAL – יש לעדכן את התיעוד
  • אין סודות/מפתחות בקוד
  • אין מחיקות מסוכנות

🧩 השפעות/סיכונים

  • חיובי: מונע CrashLoop של Pod בעת הפסקות זמניות של מסד הנתונים, מפחית רעש ב-Sentry
  • סיכון: התהליך יישאר תלוי בחיבור מסד הנתונים ולא יעצור אם החיבור לא יתחדש. יש לוודא שיש מנגנון חיצוני (health check, timeout) שיוכל לעצור את התהליך במקרה של כשל ממושך.

🧯 סיכון / החזרה

https://claude.ai/code/session_01Dgaz4FwhYhLtERMozH2UZF

Two related changes to make startup resilient to a brief DB outage:

1. database/manager.py: reduce initial background-reconnect delay from 30s
   to 5s. Previously, inside the 120s startup wait only 2 reconnect attempts
   fit (t=30, t=75). With 5s first delay we get several early attempts where
   recovery is most likely.

2. main.py: after the 120s window, don't SystemExit(1). Enter a passive wait
   loop (poll every DB_RECONNECT_POLL_INTERVAL, default 30s) until the
   background task reconnects. Avoids CrashLoop + Sentry storms on transient
   outages (e.g. the fatal events on 2026-04-03).

https://claude.ai/code/session_01Dgaz4FwhYhLtERMozH2UZF
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@github-actions

github-actions Bot commented Apr 20, 2026 •

Copy link
Copy Markdown
Contributor

🧯 Dangerous deletes guard report

Policy: see .cursorrules — dangerous deletions are blocked unless wrapped safely.

Summary:

  • Flagged findings (blocking): 0
    0
  • Excluded matches (not blocking): 15
  • Total matches (all files): 129

Flagged findings (file:line:snippet):
(none)

Excluded matches (by path pattern)
./README.md:842:find . -name "__pycache__" -exec rm -rf {} +
./node_modules/mermaid/dist/chunks/mermaid.esm.min/chunk-4HFYJGYH.mjs:1:var r={name:"mermaid",version:"11.12.0",description:"Markdown-ish syntax for generating flowcharts, mindmaps, sequence diagrams, class diagrams, gantt charts, git graph … [truncated]
./node_modules/mermaid/dist/chunks/mermaid.esm.min/chunk-4HFYJGYH.mjs.map:4:  "sourcesContent": ["{\n  \"name\": \"mermaid\",\n  \"version\": \"11.12.0\",\n  \"description\": \"Markdown-ish syntax for generating flowcharts, mindmaps, sequen … [truncated]
./node_modules/mermaid/dist/chunks/mermaid.esm/chunk-2M32CCKP.mjs.map:4:  "sourcesContent": ["{\n  \"name\": \"mermaid\",\n  \"version\": \"11.12.0\",\n  \"description\": \"Markdown-ish syntax for generating flowcharts, mindmaps, sequence d … [truncated]
./node_modules/mermaid/dist/chunks/mermaid.core/chunk-KS23V3DP.mjs.map:4:  "sourcesContent": ["{\n  \"name\": \"mermaid\",\n  \"version\": \"11.12.0\",\n  \"description\": \"Markdown-ish syntax for generating flowcharts, mindmaps, sequence  … [truncated]
./node_modules/mermaid/dist/mermaid.js.map:4:  "sourcesContent": ["/**\n* Default values for dimensions\n*/\nconst defaultIconDimensions = Object.freeze({\n\tleft: 0,\n\ttop: 0,\n\twidth: 16,\n\theight: 16\n});\n/**\n* Default values for tr … [truncated]
./node_modules/mermaid/dist/mermaid.min.js.map:4:  "sourcesContent": ["/**\n* Default values for dimensions\n*/\nconst defaultIconDimensions = Object.freeze({\n\tleft: 0,\n\ttop: 0,\n\twidth: 16,\n\theight: 16\n});\n/**\n* Default values fo … [truncated]
./node_modules/mermaid/dist/mermaid.min.js:1524:`,"getStyles"),c1e=RQe});var h1e={};dr(h1e,{diagram:()=>NQe});var NQe,f1e=N(()=>{"use strict";$ge();a1e();l1e();u1e();NQe={parser:Fge,db:n1e,renderer:o1e,styles:c1e}});var m1e,g1e=N(()=>{"use  … [truncated]
./node_modules/katex/src/fonts/Makefile:139:	rm -rf pfa ff otf ttf woff woff2
./node_modules/katex/package.json:153:    "build": "rimraf dist/ && mkdirp dist && cp README.md dist && rollup -c --failAfterWarnings && webpack && node update-sri.js package dist/README.md",
./Dockerfile:42:    rm -rf /var/lib/apt/lists/*
./Dockerfile:121:    rm -rf /var/lib/apt/lists/*
./webapp/static/js/md_preview.bundle.js.map:4:  "sourcesContent": ["// Markdown-it plugin to render GitHub-style task lists; see\n//\n// https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments\n// https://github.com/blog/1825-t … [truncated]
./docs/DOCUMENTATION_GUIDE.md:453:rm -rf _build
./docs/Makefile:24:	rm -rf $(BUILDDIR)

@github-actions

github-actions Bot commented Apr 20, 2026 •

Copy link
Copy Markdown
Contributor

⏱️ Performance report

(No performance test durations collected. Mark tests with @pytest.mark.performance.)

@github-actions

github-actions Bot commented Apr 20, 2026 •

Copy link
Copy Markdown
Contributor

📖 Documentation Preview

The documentation has been built successfully!

To view locally:

  1. Download the artifacts
  2. Extract the zip file
  3. Open index.html in your browser

Comment thread main.py
@sentry

sentry Bot commented Apr 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 11 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
main.py 0.00% 7 Missing ⚠️
database/manager.py 0.00% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

Follow-up to df08735. Cursor-bot flagged that main.py's new passive wait
loop could hang forever: after max_bg_attempts (10) in
_schedule_background_reconnect, no more attempts were scheduled, so
is_connected would stay False and the main loop would busy-wait doing
nothing useful.

Fix: remove the hard cap on background reconnect attempts. Exponential
backoff is already capped at 300s, so worst-case load is one attempt per
5 minutes — cheap to keep trying. We emit a one-time "escalating" event
when the historical max is crossed, so observability is preserved.

https://claude.ai/code/session_01Dgaz4FwhYhLtERMozH2UZF

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 158c732. Configure here.

Comment thread main.py
Cursor-bot flagged that the new DB_RECONNECT_POLL_INTERVAL (introduced in
df08735) wasn't registered in services/config_inspector_service.py or
docs/environment-variables.rst. Also adds the previously-undocumented
DB_RECONNECT_WAIT_BEFORE_POLL so both tunables are visible to operators
using the config inspector.

https://claude.ai/code/session_01Dgaz4FwhYhLtERMozH2UZF
@amirbiron
amirbiron merged commit 752f43b into main Apr 20, 2026
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants