Add signer balance metric - #137
Open
jannikluhn wants to merge 3 commits into
Open
Conversation
TotalSuccessfulIdentityRegistration, TotalDecryptionKeysReceived and TotalFailedRPCCalls are only ever incremented, so Counter is the correct type. Series names are unchanged, so existing queries keep working, and rate() over them is now legitimate rather than an accident.
Prometheus counters carry _total as a suffix, not a prefix, and the
plural belongs on the thing being counted:
shutter_api_total_successful_identities_registration
-> shutter_api_successful_identity_registrations_total
shutter_api_total_decryption_keys_received
-> shutter_api_decryption_keys_received_total
shutter_api_total_failed_rpc_calls
-> shutter_api_failed_rpc_calls_total
Go identifiers drop their now-redundant Total prefix to match. Existing
series keep their history under the old names but stop being written to,
so any dashboard or alert rule querying them needs updating.
The signer pays gas for every identity registration, but the service never queried its balance, so an account draining to empty was only visible once registrations started failing. At that point it surfaced as failed_rpc_calls_total from the transaction send sites, indistinguishable from an RPC outage. shutter_api_signer_balance_ether is published by a new BalancePoller running as a service alongside the metrics server, so it is gated on METRICS_ENABLED. It reads once at startup and then every 60s, each read bounded by a 10s timeout so a hung endpoint cannot stall the loop. A failed read logs, increments failed_rpc_calls_total and leaves the gauge alone; it never returns an error, because a transient RPC failure must not bring the service down through the error group. The metric is a GaugeVec with no labels rather than a plain Gauge. A plain Gauge is registered holding 0, so a restart while the RPC endpoint was down would publish 0 ether and fire the low-balance alert this metric exists to raise. With no value set the series is simply absent, and the exposed series is otherwise identical. Balance is reported in ether rather than wei so that alert thresholds are readable; the conversion goes through big.Float, as an integer quotient would truncate everything below 1 ether to zero. Known gap: a reading that stops being refreshed keeps its last value indefinitely, so a dead poller looks healthy. Detection relies on failed_rpc_calls_total, which now also moves for background polls and can therefore rise with no traffic. Co-Authored-By: Claude <noreply@anthropic.com>
ylembachar
reviewed
Aug 25, 2026
| defer cancel() | ||
|
|
||
| wei, err := p.client.BalanceAt(ctx, p.address, nil) | ||
| if err != nil { |
Collaborator
There was a problem hiding this comment.
This counts shutdown as an RPC failure. A cancelled parent context lands in the same error branch as a real failure, so a redeploy that catches a poll in flight logs an error and increments FailedRPCCalls. DeadlineExceeded should stay counted though, a node that doesn't answer in 10s is a genuine failure. So guard on cancellation only:
if err != nil {
if errors.Is(err, context.Canceled) {
return
}
log.Err(err).Str("address", p.address.Hex()).Msg("failed to query signer balance")
FailedRPCCalls.Inc()
return
}
| return nil | ||
| } | ||
|
|
||
| // poll reads the balance and updates the gauge. A failed read leaves the gauge |
Collaborator
There was a problem hiding this comment.
This gives the wrong reason for swallowing errors. Swallowing them is correct, but the stated reason isn't: the comment says the poller "shares an error group with the API and a transient RPC failure must not shut the service down". It shares the group at main.go:206 with the metrics server, and that group's error is only logged at :209, never cancelled.
The actual reason is: errgroup.WithContext cancels the group on the first non-nil error, so returning one would take metricsServer down with the poller and we'd lose the metrics entirely over a transient RPC blip. Suggested:
// Errors are never returned, because the poller shares an errgroup with the
// metrics server (main.go:206) and returning one would cancel the group,
// taking the whole /metrics endpoint down over a transient RPC failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #135
The balance is polled instead of updated after every transaction for simplicity and to detect balance changes made by potential other users of the key.