Closed
Description
Original author: Josh Duffney (@duffney)
Problem
Currently, if MCP server deregistration fails at the end of a run, stale servers remain in the gateway until purgeAll() clears them at the start of the next run. This means:
- The next run starts on a pod with a contaminated gateway state
- There's no warning surfaced to the operator that the pod is in a degraded state
- A silent
purgeAll()is the only recovery mechanism
Proposed solution
Instead of silently purging on the next run, detect stale state at startup and recycle the pod — let Kubernetes rebuild it clean rather than trying to recover in-process.
Options:
- Pod self-termination: if
purgeAll()finds stale servers at the start ofsetup(), log an error andprocess.exit(1)— KEDA will reschedule a fresh pod - Pre-flight health check: before accepting a new task from the queue, query the gateway for stale servers; if any exist, reject the message (nack/return to queue) and exit — guarantees the next worker picks it up clean
- Contamination block: expose a
isGatewayClean()check inWorkerProcessor.setup()and fail fast before the run starts, with a clear log message
Option 1 is simplest and leverages the existing K8s + KEDA self-healing — a recycled pod is always cleaner than in-process recovery.
Related
- PR feat: MCP gateway support for ACP workers #521 — MCP gateway support (introduced
purgeAll()as crash recovery) - Issue security: migrate MCP server secrets (headers, env) from MongoDB to Key Vault #525 — MCP server secrets in Key Vault
Metadata
Metadata
Assignees
Labels
No labels
Activity
cedricvidal commentedon Jul 29, 2026
Original author: Cedric Vidal (@cedricvidal)
What if after receiving the request, if there are stale MCP servers, we try purging them? if it fails, then we can go seppuku ;)
cedricvidal commentedon Jul 29, 2026
Original author: Cedric Vidal (@cedricvidal)
@jduffney_microsoft is still relevant?
cedricvidal commentedon Jul 29, 2026
Original author: Josh Duffney (@duffney)
It's no longer relvant, I'll close the issue. :)