Repository navigation
Add composite MySQL safety checks and stable recovery - #196
pradeepkintali wants to merge 11 commits into
Conversation
Allow replica-lag checks to require cached primary and ProxySQL safety metrics, fail closed for configured probes, and hold recovery until a cooling period completes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0fbc396f-32f9-449a-8537-6264e8665c31
Require a full healthy window after process or leader startup, expose the blocking metric in check responses, and publish recovery latch state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0fbc396f-32f9-449a-8537-6264e8665c31
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0fbc396f-32f9-449a-8537-6264e8665c31
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Recovery semantics and direct-memcache behavior can bypass or inconsistently apply the new safety gates.
Review effort: Balanced
Findings: 1
Open (4)
What changed in this PR
Adds composite MySQL safety checks, recovery hysteresis, and configurable Vitess tablet discovery.
Changes:
- Composes replica checks with required primary and ProxySQL metrics.
- Adds fail-closed behavior, recovery thresholds, cooling periods, and metrics.
- Supports validated
REPLICA,MASTER, andPRIMARYVitess tablet configuration.
| File | Description |
|---|---|
pkg/base/throttle_metric.go |
Adds recovery-state metric errors. |
pkg/config/config_test.go |
Tests new configuration validation. |
pkg/config/mysql_config.go |
Adds composite and recovery settings. |
pkg/config/vitess_config.go |
Validates tablet types. |
pkg/throttle/check.go |
Evaluates required clusters. |
pkg/throttle/check_result.go |
Reports the blocking metric. |
pkg/throttle/throttler.go |
Implements recovery latching. |
pkg/throttle/throttler_test.go |
Tests composition and recovery. |
pkg/vitess/api_client.go |
Filters configured tablet types. |
pkg/vitess/api_client_test.go |
Tests primary discovery. |
doc/mysql.md |
Documents configuration and rollout scope. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Preserve the replica validation API, apply read overrides consistently, reset recovery state on leadership regain, and enforce required clusters for direct memcache consumers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0fbc396f-32f9-449a-8537-6264e8665c31
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0fbc396f-32f9-449a-8537-6264e8665c31
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dea961fa-5766-415f-a0f5-2562c55a0385
Preserve MASTER as a deprecated input alias and the existing numeric tablet mapping. Clarify that shadow collection precedes enforcement and is not composite dry-run counting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fbc83df2-d1af-406f-b97b-51cf0aaa570d
Evaluate prospective required metrics without changing admission or memcache publication. Report separate HTTP and background would-reject counters, validate dependency graphs, and document safe activation and recovery behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fbc83df2-d1af-406f-b97b-51cf0aaa570d
Preserve static-host ignore-settings propagation and both regression test suites. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fbc83df2-d1af-406f-b97b-51cf0aaa570d
Explain cached source and dependency metrics with effective thresholds, recovery settings and enforcement modes without running checks or exposing connection details. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fbc83df2-d1af-406f-b97b-51cf0aaa570d
preetha-su
left a comment
There was a problem hiding this comment.
Can you validate the following?
-
Send real requests through the HTTP handler. Verify that an overloaded required primary returns
429and identifies the primary metric -
The same primary as above configured as a shadow dependency, verify the request still gets the original
200response.
3.Recovery after error: Start the healthy recovery window, make the probe fail, then restore healthy readings. Verify the error interrupts recovery and Freno requires a fresh full healthy window before allowing work again.


Summary
RequiredClustersdependencies.ShadowRequiredClusterswithout changing the source cluster's admission decision, and report would-reject observations./explain/mysql/<cluster>diagnostics for loaded thresholds, recovery settings, exclusions, and cached required/shadow metric state.PRIMARYdiscovery while acceptingMASTERas a deprecated input alias.Configuration and compatibility
RequiredClusters,ShadowRequiredClusters,FailOnNoHosts, andRecoveryDurationMillisare opt-in. Existing clusters retain legacy behaviorwhen these settings are absent.
Shadow observations do not simulate application-specific throttling policies.
Background observations represent aggregation cycles, not client requests or
independent database samples. Missing observations are not evidence of health.
Diagnostics do not run admission checks or SQL, advance recovery, or record
synthetic check/shadow observations. Missing measurements are
null; credentials,queries, connection details, and raw probe errors are not exported. A successful
diagnostic request is not permission to write.
Deployment and rollback
Deploy a compatible binary before configuring the new fields. Validate probe
discovery, permissions, sampling cost, and recovery behavior before enabling
required dependencies. Review thresholds and client retry/wait budgets for each
workload, and enable enforcement incrementally.
To roll back dependency enforcement, remove the relevant
RequiredClustersentry while preserving existing replica-lag checks. Keep it in
ShadowRequiredClustersif continued observation is desired. Configurationdeployment and startup recovery still apply; this is not an instantaneous
runtime toggle.
See
doc/mysql.mdfor configuration examples, behavior changes, and rolloutwarnings, and
doc/http.mdfor the diagnostic endpoint contract.Validation
go test -mod=vendor -race ./cmd/... ./pkg/...go vet -mod=vendor ./cmd/... ./pkg/...go build -mod=vendor -o /tmp/freno ./cmd/frenogit diff --check.Scope
These are cooperative admission checks, not universal database load shedding.
Deployments remain responsible for metric selection, thresholds, discovery,
monitoring, and safe client retry/defer behavior.