Skip to content

docs(operators): sync image versions, Grafana password and snapshot default with the compose repo - #2376

Merged
GigaHierz merged 4 commits into
celo-org:mainfrom
eddiemessiah:docs/operator-config-sync
Oct 8, 2026
Merged

GigaHierz merged 4 commits into
celo-org:mainfrom
eddiemessiah:docs/operator-config-sync

Conversation

@eddiemessiah

@eddiemessiah eddiemessiah commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

The hole, and the fix

The operator pages had drifted from celo-l2-node-docker-compose, the setup they document. An operator who followed them would pull the wrong op-node and eigenda-proxy tags if building outside the compose repo, fail to log in to Grafana, misread the snapshot default, try to pin op-node with a variable the compose file never reads, and open the healthcheck's metrics while expecting op-node's.

This PR changes only values checked against compose main at 9634d6e465197a120e6bd7d80ba9f249b4a7a8d9 ("Bump op-node image to celo-v2.2.3 (#107)", 2026-10-06):

  • network-config.mdx: op-node celo-v2.1.0 -> celo-v2.2.3 and eigenda-proxy v1.8.2 -> v2.6.0, mainnet and Celo Sepolia. op-reth celo-v1.0.5 and the Celo L1 client geth-all:1.8.9 already match and are unchanged. The snapshot line now says snapshots are on by default.
  • monitoring.mdx: Grafana admin password optimism -> celo, plus a warning: compose publishes Grafana on every host interface, GF_SECURITY_ADMIN_PASSWORD must be set in .env before the first start with monitoring enabled (Grafana reads it only when it creates the admin user, and the user persists in the grafana_data volume), after that the password is changed in Grafana, and port 3000 should be firewalled. The op-node metrics line no longer points at http://localhost:7300/metrics; that host port belongs to the healthcheck container. op-node's port 7300 is scraped by Prometheus on the compose network and is not published to the host.
  • configuration.mdx, run-node.mdx, archive-node.mdx: OP_RETH__SNAPSHOT is true in the shipped .env files, not false. "leave OP_RETH__SNAPSHOT=false" becomes "set".
  • maintenance.mdx: the pinning example is now IMAGE_TAG__OP_RETH only, with one sentence saying the op-node and eigenda-proxy tags are fixed in docker-compose.yml and change only on git pull. IMAGE_TAG__OP_NODE has no effect.

What this does NOT do / residual risk

  • Archived notices (operate/notices/archive/*) still list op-node v2.1.0 and eigenda-proxy v1.8.2. That is correct for each notice's date, so they are unchanged.
  • The image tags are a snapshot of compose main on 2026-10-06. They will drift again on the next compose bump.
  • Grafana dashboard names ("Simple Node Dashboard", "Succinct Challenger") and the admin username were not checked against the dashboard JSON or a running Grafana. They are unchanged.
  • The Grafana warning does not give a CLI reset command. The CLI name differs between Grafana 9.3 (grafana-cli, the compose default image) and current releases (grafana cli), and I did not confirm which is on the path in the container.
  • IMAGE_TAG__OP_NODE is still listed in mainnet.env / celo-sepolia.env in the compose repo even though docker-compose.yml does not read it. That belongs in the compose repo and is not fixed here.

Judgement calls

  • monitoring.mdx op-node metrics line: I corrected the endpoint instead of only changing the password, because the old URL returns healthcheck metrics. One-line revert if a maintainer would rather publish op-node's 7300 in compose.
  • maintenance.mdx: I dropped IMAGE_TAG__OP_NODE rather than opening a compose-repo issue to wire the variable up. Happy to drop this hunk; the rest stands without it.

Issues

Refs: none filed. Found by an audit of the operator pages against the compose repo.

Stacking / conflicts

Based on current main through the maintainer's merge of main (3c20055b, parent 418506f6). The merge touched no file in this PR. Touches only operate/operators/{archive-node,configuration,maintenance,monitoring,network-config,run-node}.mdx. Should merge before or with #2377, whose notices link to Network config.

Verification evidence

Source: celo-org/celo-l2-node-docker-compose @ 9634d6e465197a120e6bd7d80ba9f249b4a7a8d9 (main on 2026-10-08), fetched with gh api repos/celo-org/celo-l2-node-docker-compose/contents/<path>.

Docs claim (new) Compose file:line Line
op-node celo-v2.2.3 docker-compose.yml:70 image: us-west1-docker.pkg.dev/devopsre/celo-blockchain-public/op-node:celo-v2.2.3
eigenda-proxy v2.6.0 docker-compose.yml:32 image: ghcr.io/layr-labs/eigenda-proxy:v2.6.0
op-reth celo-v1.0.5 (unchanged) docker-compose.yml:48 ...op-reth}:${IMAGE_TAG__OP_RETH:-celo-v1.0.5}
Celo L1 client geth-all:1.8.9 (unchanged) docker-compose.yml:3 image: us-docker.pkg.dev/celo-org/us.gcr.io/geth-all:1.8.9
Grafana password celo envs/common/grafana.env:1 GF_SECURITY_ADMIN_PASSWORD=celo
.env overrides grafana.env docker-compose.yml:133-135 env_file: - ./envs/common/grafana.env - .env
Grafana state persists in a named volume docker-compose.yml:141 - grafana_data:/var/lib/grafana
Grafana published on all interfaces docker-compose.yml:144 - ${PORT__GRAFANA:-3000}:3000
Grafana runs only with monitoring enabled scripts/start-grafana.sh if [ "$MONITORING_ENABLED" = "true" ]; then exec /run.sh
OP_RETH__SNAPSHOT=true (mainnet) mainnet.env:55 OP_RETH__SNAPSHOT=true
OP_RETH__SNAPSHOT=true (Celo Sepolia) celo-sepolia.env:45 OP_RETH__SNAPSHOT=true
Same, in the start script scripts/start-op-reth.sh:26 # scratch. On by default (OP_RETH__SNAPSHOT=true); set it to false to sync from
Host port 7300 is the healthcheck docker-compose.yml:27-28 (healthcheck service) ports: - ${PORT__HEALTHCHECK_METRICS:-7300}:7300
op-node 7300 not published to host docker-compose.yml:80-83 (op-node service) publishes only 9222/udp, 9222/tcp, 9545
op-node metrics on 7300 inside the network scripts/start-op-node.sh:50; docker/prometheus/prometheus.yml:20 --metrics.port=7300; targets: ["op-node:7300"]
IMAGE_TAG__OP_NODE not read docker-compose.yml grep -c IMAGE_TAG__OP_NODE docker-compose.yml returns 0

Grafana: the admin_password setting is documented as "Set once on first-run" in both the current and v9.3 configuration docs (compose defaults to grafana/grafana:9.3.0). The in-app path (user icon, then Change Password) is from the v9.3 user preferences docs.

Line endings: the six files keep CRLF in the working tree. The PR diff against main (418506f6) is 6 files changed, 15 insertions(+), 11 deletions(-).

Preview

npx mint dev --port 3642 on this branch: /operate/operators/monitoring returns 200 with the new warning text. Checked in a browser at 1280px and 412px: the warning renders inside the callout, no horizontal overflow (scrollWidth 412 at 412px), 0 console errors.

mint broken-links (Windows): found 586 broken links in 125 files on this branch and the same list, line for line, with the six files reverted to main (418506f6). The report flags pages that exist (for example /operate/operators/monitoring), so it is a Windows path-resolution artifact. This PR adds no internal links. CI's Linux run is the authoritative check.

Remaining ops steps

  • none

Checklist

  • Title is the commit message I want on main
  • Ships the test that fails on pre-fix code (n/a: docs-only; evidence table above instead)
  • Covers the seam (n/a: docs-only)
  • Error responses audited (n/a)
  • Judgement calls / bundled product changes flagged above
  • lint / typecheck / test pass locally (n/a; mint broken-links result above)
  • Re-read acceptance criteria of every Closes issue (none closed)
  • Development sidebar links match Closes/Refs above
  • README / runbook / examples updated for the world this creates
  • Lockfile touched (n/a)
  • Wallet/provider tree touched (n/a)
  • UI touched (n/a)
  • Money/security path (n/a)
  • No secrets in the diff (the Grafana password is the published compose default)
  • Questions for the maintainer: should the IMAGE_TAG__OP_NODE gap be fixed in the compose repo (wire the variable into docker-compose.yml) rather than documented around?

🤖 Generated with Claude Code

@eddiemessiah
eddiemessiah requested a review from a team as a code owner October 6, 2026 08:26
@eddiemessiah
eddiemessiah requested review from ezdac and karlb and removed request for a team October 6, 2026 08:26

@GigaHierz GigaHierz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

head 214fac8 · API headRefOid == local refs/pull ref (re-fetched); 1 commit, base 50ed71d, 1 behind main (a12466b, the home/ to learn/ rename); merge-tree against origin/main is clean; mint broken-links --check-redirects and check-orphans.sh pass on the merge; CI green (Check for broken links, GitGuardian, StepSecurity); rendered monitoring and network-config at 1280 and 412, 0 console errors

Verdict: REQUEST-CHANGES

Thanks for the careful audit of the operator pages against the compose repo. Most of what you changed checks out against primary sources, and the evidence table made it quick to verify. One finding is urgent, because the compose repo moved on after you opened the PR. Everything else is small.

Findings

1. High · operate/operators/network-config.mdx:60 and :93 · op-node tag celo-v2.2.1 is already stale and would mislead operators on a day it matters

The compose repo's main is now 9634d6e4 ("Bump op-node image to celo-v2.2.3 (#107)", 2026-10-06 08:46 UTC, about 20 minutes after this PR was opened), and docker-compose.yml:70 reads op-node:celo-v2.2.3. The celo-org/optimism release for celo-v2.2.3 says it is "required for the Glamsterdam activation" and that Celo Sepolia operators "must upgrade to this version prior to the Glamsterdam hardfork on October 6th". Release celo-v2.2.2 is batcher-only. So v2.2.1 is not just older: it is the version that predates the Amsterdam L1 header support.

Failure scenario: an operator building outside the compose repo (the exact case this page serves, per the PR text) copies the linked op-node:celo-v2.2.1 tag and runs it against Ethereum Sepolia after Glamsterdam. op-node cannot handle the new L1 headers and the node stops following the chain.

Fix shape: change both lines (mainnet and Celo Sepolia) to celo-v2.2.3, and re-read compose docker-compose.yml on the day you push, since it can move again. Both tags (v2.2.1 and v2.2.3) exist in the registry; eigenda-proxy:v2.6.0, op-reth:celo-v1.0.5 and geth-all:1.8.9 also resolve and still match compose. Optionally, to stop this drifting a third time, replace the pinned image list with a sentence plus a link to docker-compose.yml as the source of truth (your PR text already names this risk). That is a judgement call for the maintainers, so either approach resolves the finding.

2. Medium · operate/operators/monitoring.mdx:24 · a documented default credential with no instruction to change it

The value is correct (envs/common/grafana.env contains GF_SECURITY_ADMIN_PASSWORD=celo, and the compose README says the same), and publishing a default credential is fine. But compose publishes Grafana as ${PORT__GRAFANA:-3000}:3000, which binds all host interfaces, and the page is written for people running production nodes.

Failure scenario: an operator runs monitoring on a VPS without a firewall rule for 3000, and anyone who reads this page can log in as admin.

Fix shape: add a short <Warning> under the credentials saying these are defaults, and to set GF_SECURITY_ADMIN_PASSWORD in .env (compose lists .env after grafana.env in env_file, so it overrides) or to firewall port 3000. Do not describe the override as tested unless you run it.

3. Low · operate/operators/configuration.mdx:20 and run-node.mdx:79 · "default" is true for the shipped env templates, not for the start script

scripts/start-op-reth.sh:29 only downloads a snapshot when OP_RETH__SNAPSHOT is exactly true; if the variable is missing from a .env (for example one copied before the variable existed) there is no snapshot. The default comes from mainnet.env:55 and celo-sepolia.env:45. The compose README calls true "the default", so your wording matches upstream, but the narrower phrasing is the precise one.

Fix shape: write true (default in the shipped .env files), false, and keep run-node as "set OP_RETH__SNAPSHOT=false (the shipped .env files set it to true)". This is also the wording in #2345, which avoids a second conflict (see merge-order hazards).

4. Low · operate/operators/monitoring.mdx:36 · the corrected op-node line says where metrics are not, but not where to read them

"scraped by Prometheus at op-node:7300 on the compose network; this port is not published to the host, where localhost:7300 is the healthcheck" is accurate (prometheus.yml target op-node:7300, start-op-node.sh:48-50, op-node publishes only 9222 and 9545), but a reader skimming a "Key metrics to watch" list still has no URL to open.

Fix shape: "From op-node (query in Prometheus at http://localhost:9090 or in Grafana; the op-node port 7300 is not published to the host, where localhost:7300 is the healthcheck)". Prometheus is published as ${PORT__PROMETHEUS:-9090}:9090 and the page already names it two paragraphs above.

Verified good

Checked against compose main (9634d6e4 at review time, and e90c955d which you cited), the registries and release tags:

  • eigenda-proxy v2.6.0: compose docker-compose.yml:32 literal; manifest returns HTTP 200 from ghcr.io; Layr-Labs/eigenda has release v2.6.0. (The old Layr-Labs/eigenda-proxy repo is archived, which is consistent with the image now being published from the monorepo.)
  • op-reth celo-v1.0.5 unchanged and correct: compose line 48 default; tag present in the Artifact Registry tag list (v1.0.6 also exists but compose has not adopted it, so leaving v1.0.5 is right).
  • geth-all 1.8.9: compose lines 3; manifest returns 200.
  • Same images on both networks: the image lines take no network variable.
  • Grafana user admin / password celo: envs/common/grafana.env:1 and README lines 262-263.
  • OP_RETH__SNAPSHOT=true in mainnet.env:55 and celo-sepolia.env:45; README "Snapshot modes" says "(the default)"; script comment at start-op-reth.sh:26. Agrees with the new text in configuration, run-node, archive-node and the Celo Sepolia line of network-config. historical-proofs.mdx:123 already says "the default", so it now agrees.
  • Healthcheck owns host port 7300 (docker-compose.yml:27-28); op-node publishes only 9222/udp, 9222/tcp, 9545 (:81-83); op-node metrics on 7300 inside the network.
  • IMAGE_TAG__OP_NODE has no effect: the op-node image tag is a literal in docker-compose.yml, and the variable appears only as an empty line in the env templates. The maintenance.mdx sentence is accurate.
  • Not read: nothing was blocked. I read envs/common/grafana.env, mainnet.env and celo-sepolia.env through gh api. I did not run compose or log in to a Grafana, so the claim that .env overrides grafana.env rests on compose env_file ordering, not on a run.
  • Anchors and links: the diff adds no internal links; mint broken-links --check-redirects and scripts/check-orphans.sh pass on a merge into current main. Frontmatter, heading case and page structure are untouched; no promotional wording.

What the PR got right

  • Scoped to values you checked, with file:line evidence and an honest residual-risk section. That is why the one stale value is easy to fix.
  • Catching that localhost:7300 is the healthcheck and not op-node was a real operator trap.
  • Removing IMAGE_TAG__OP_NODE as a pin example is correct, and raising the compose-repo gap as a question rather than papering over it is the right call.
  • Leaving archived notices alone is right; their versions are correct for their dates (but see the cross-PR note below).
  • Line endings and diff size are tight (6 files, 11 insertions, 11 deletions).

Merge-order hazards

  • #2345 (draft, other contributor) edits the same lines and conflicts with this PR in archive-node.mdx (the snapshot paragraph, line 70), configuration.mdx (the OP_RETH__SNAPSHOT row, line 20) and run-node.mdx (the "From genesis" bullet, line 79). git merge-tree of the two heads reports exactly those three files; maintenance.mdx and network-config.mdx merge cleanly. Both PRs agree the default is true, so there is no factual contradiction, but #2345 wording ("default in the shipped .env files") is the more precise one, so the second merger should keep it.
  • Both this PR and #2345 leave or restate "it provides the pre-L2 (Celo L1) history" for the mainnet snapshot in those three places; #2345 argues that claim is wrong (the snapshot holds only the Celo L1 state at the migration). Whichever merges first, the other should not reintroduce the old sentence when resolving the conflict.
  • #2351 is already merged and this branch is based on it; no overlap remains.
  • #2377 touches different files; the two merge in any order. #2367 (home/ to learn/) is already on main; this PR touches only operate/, so a rebase is trivial. #2368 (build/ move) has not landed and does not touch these files.
  • Once the op-node tag is bumped, the archived notices still say "We recommend the latest released versions: ... op-node v2.1.0" (jello, fusaka, Celo Sepolia launch). That present-tense recommendation is now wrong; it is tracked in the #2377 review.
  • Separate from this PR: docs on main have no Glamsterdam notice even though op-node celo-v2.2.3 is required for it on Celo Sepolia; a maintainer should decide whether to file an issue for that.

@eddiemessiah

Copy link
Copy Markdown
Contributor Author

Previewed in a local live deployment (mint dev, this branch merged with the other open docs fixes, no overlapping files) at desktop 1280px and phone 412px:

  • /operate/operators/network-config (new image tags are in the op-node and eigenda-proxy links for both networks)
  • /operate/operators/monitoring
  • /operate/operators/configuration
  • /operate/operators/run-node
  • /operate/operators/archive-node
  • /operate/operators/maintenance

Each page returns 200 and renders the new text, with no horizontal overflow on phone, no broken images and 0 console errors.

eddiemessiah and others added 2 commits October 6, 2026 17:53
…efault with the compose repo

Align the operator pages with celo-l2-node-docker-compose main (e90c955):

- network-config: op-node celo-v2.1.0 -> celo-v2.2.1 and eigenda-proxy v1.8.2 -> v2.6.0 for mainnet and Celo Sepolia (docker-compose.yml); Celo Sepolia snapshot is on by default
- monitoring: Grafana admin password is celo (envs/common/grafana.env); op-node metrics on 7300 are scraped inside the compose network, not published to the host
- configuration, run-node, archive-node: OP_RETH__SNAPSHOT defaults to true in both network .env files and scripts/start-op-reth.sh
- maintenance: IMAGE_TAG__OP_NODE does not pin op-node; its tag is fixed in docker-compose.yml

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a warning, snapshot wording, metrics URL)

- network-config: op-node celo-v2.2.1 -> celo-v2.2.3 on the mainnet and Celo Sepolia lines, matching compose main 9634d6e4 (2026-10-06); tag confirmed in the registry
- monitoring: add a Warning under the Grafana credentials to set GF_SECURITY_ADMIN_PASSWORD in .env or firewall port 3000
- monitoring: the op-node metrics line now says where to read them (Prometheus at localhost:9090 or Grafana)
- configuration, run-node: "default" is scoped to the shipped .env files, matching celo-org#2345

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@eddiemessiah
eddiemessiah force-pushed the docs/operator-config-sync branch from 214fac8 to db81e21 Compare October 6, 2026 18:08
@eddiemessiah

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Head is now db81e21cbc40905f66839ddec1d0a917d965acc9 (rebased onto main at 7b8f97c2, then force-pushed with lease; the two commits are the original plus one fix-up on top).

# Finding Status Where New wording
1 op-node celo-v2.2.1 stale FIXED operate/operators/network-config.mdx:60 and :93 op-node:celo-v2.2.3 on the mainnet and Celo Sepolia lines
2 Grafana default credentials, no instruction to change them FIXED operate/operators/monitoring.mdx:26-28 <Warning> under the credentials: "These are the compose defaults, and compose publishes Grafana on every host interface (${PORT__GRAFANA:-3000}:3000). Before the node is reachable from a network, set GF_SECURITY_ADMIN_PASSWORD in .env (compose loads .env after grafana.env, so it takes precedence) or firewall port 3000." The override rests on env_file order in docker-compose.yml:133-135; I did not run it.
3 "default" is true for the shipped env files, not the start script FIXED operate/operators/configuration.mdx:20; operate/operators/run-node.mdx:79 `true` (default in the shipped `.env` files), `false`; "set OP_RETH__SNAPSHOT=false (the shipped .env files set it to true)"
4 op-node metrics line says where metrics are not, not where to read them FIXED operate/operators/monitoring.mdx:40 "From op-node (query in Prometheus at http://localhost:9090 or in Grafana; the op-node port 7300 is not published to the host, where localhost:7300 is the healthcheck)"

Source for the tag: celo-org/celo-l2-node-docker-compose main at 9634d6e465197a120e6bd7d80ba9f249b4a7a8d9 ("Bump op-node image to celo-v2.2.3 (#107)", 2026-10-06 08:46 UTC), read through gh api today, 2026-10-06, before pushing. docker-compose.yml:70 is op-node:celo-v2.2.3. The other images are unchanged there and on the page: eigenda-proxy:v2.6.0 (:32), op-reth default celo-v1.0.5 (:48), geth-all:1.8.9 (:3). The Artifact Registry returns HTTP 200 for the op-node:celo-v2.2.3 manifest and lists the tag. I left the page as a pinned list; replacing it with a sentence plus a link to docker-compose.yml as the source of truth is yours to call, and I am happy to do it in this PR or a follow-up.

Checks on db81e21c:

  • git merge-tree --write-tree origin/main HEAD: clean (and the rebase applied with no conflicts).
  • scripts/check-orphans.sh: no orphan pages (225 navigation entries, 224 page files). jq is not installed on this machine, so I ran the script's two jq filters through an equivalent Node one-liner and the rest of the script as written.
  • git grep -n "celo-v2.2.1" operate/operators: nothing. The only remaining hit under operate/ is operate/notices/archive/jovian-upgrade.mdx:55, an archived notice that is correct for its date and intentionally unchanged.
  • mint broken-links --check-redirects (mint 4.2.988): on Windows it flags links to pages that exist, the same path artifact described in the PR body, so the local exit code is not meaningful either way. The diff adds or removes no internal links (only the two image URLs change). The authoritative run is CI's Documentation Validation on Linux; for this head it is in action_required, waiting for a maintainer to approve the workflow run from the fork (same as on the previous head). GitGuardian and both StepSecurity checks passed on db81e21c.
  • Local preview and screenshots: not done. The drive this worktree is on was below 0.7 GB free while four sibling mint dev previews were running, and mint dev --port 3473 writes another ~300 MB preview copy, so I did not start it. I have not seen the rendered pages for this head; the previous head rendered clean at 1280 and 412 in your review, and the changes are four lines of text plus one <Warning>, in the same form as the other <Warning> blocks in the repo.

Merge order: #2345 edits three of the same lines (archive-node.mdx:70, configuration.mdx:20, run-node.mdx:79). Finding 3 now uses the same "default in the shipped .env files" wording, so the two agree; whichever merges second should keep that wording and not reintroduce the "pre-L2 (Celo L1) history" sentence that #2345 corrects.

@eddiemessiah

Copy link
Copy Markdown
Contributor Author

Follow-up on the one item I had left open: the local preview of db81e21c is now done (mint dev --port 3473 in the worktree, pages fetched over HTTP and rendered in headless Chromium).

All six operator pages return 200:

Page Status
/operate/operators/network-config 200
/operate/operators/monitoring 200
/operate/operators/configuration 200
/operate/operators/run-node 200
/operate/operators/archive-node 200
/operate/operators/maintenance 200

The new text is in the served HTML and renders: op-node:celo-v2.2.3 on both network-config lines (no celo-v2.2.1 left on the page), the Grafana <Warning> block under the credentials on monitoring, the reworded op-node metrics line, and the "shipped .env files" wording on configuration and run-node. Zero browser console errors at 1280x900 and 412x915; the Warning block fits at 412 px with no horizontal overflow.

Screenshots of the four changed pages at both sizes are available if you want them attached here.

@GigaHierz GigaHierz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

head db81e21 · API headRefOid == local refs/pull ref (re-fetched); 2 commits on base 7b8f97c = current origin/main (0 behind); merge-tree against origin/main clean; mint broken-links --check-redirects and check-orphans.sh pass on the head; compose main still 9634d6e4 (celo-v2.2.3); CI: Documentation Validation is action_required (waiting for a maintainer to approve the workflow run, not a failure; GitGuardian and StepSecurity pass); rendered monitoring on port 3621: warning present, password celo, no horizontal overflow, 0 console errors

Verdict: REQUEST-CHANGES

Thank you for the quick turnaround. The op-node tag, the Grafana warning, the snapshot wording and the metrics hint are all fixed, and I re-checked each against the compose repo and the registries. One small thing is left, and it is about the new warning itself: it tells operators to do something that only works before Grafana's first start.

Status of earlier findings

  1. High, op-node tag stale: fixed. Both lines in network-config.mdx (:60 mainnet, :93 Celo Sepolia) now read celo-v2.2.3. Compose docker-compose.yml:70 still reads op-node:celo-v2.2.3 (compose main is 9634d6e4, nothing newer), the celo-org/optimism release celo-v2.2.3 is marked Latest, and the registry returns 200 for that tag.
  2. Medium, Grafana default credential: fixed in substance (a <Warning> was added), but see finding 1.
  3. Low, "default" wording for OP_RETH__SNAPSHOT: fixed. configuration.mdx:20 says true (default in the shipped .env files), and run-node.mdx:79 says "the shipped .env files set it to true". Both match mainnet.env:55 and celo-sepolia.env:45, and start-op-reth.sh only downloads when the value is exactly true.
  4. Low, where to read op-node metrics: fixed. monitoring.mdx:40 now points at Prometheus on http://localhost:9090 or Grafana and says 7300 is not published. Prometheus is published as ${PORT__PROMETHEUS:-9090}:9090, and op-node publishes only 9222 and 9545.

Findings

1. Medium · operate/operators/monitoring.mdx:27 · the Grafana warning's fix does nothing once Grafana has started once

The warning says to set GF_SECURITY_ADMIN_PASSWORD in .env "before the node is reachable from a network". Grafana's documentation for admin_password says "Set once on first-run", and compose keeps Grafana's state in the named volume grafana_data (docker-compose.yml:141), which survives restarts and docker compose down.

Failure scenario: an operator who has had monitoring running for weeks reads the warning, adds GF_SECURITY_ADMIN_PASSWORD=<new> to .env, restarts, and believes the default is gone. Grafana ignores the variable because the admin user already exists in the volume, so admin / celo still logs in on a port that is open to the internet.

Fix shape: say when the variable takes effect and what to do otherwise, for example "Set GF_SECURITY_ADMIN_PASSWORD in .env before the first start with monitoring enabled; Grafana reads it only when it creates the admin user. If Grafana has already started, change the password in Grafana (profile, Change password) or firewall port 3000." (I did not run an override or log in to a Grafana, so I have not tested either path. The first-run behaviour comes from the Grafana docs.)

Verified good

Checked against compose main (9634d6e4, 2026-10-06 08:46 UTC), the registries and release tags. Read through gh api; every file I needed was readable (mainnet.env, celo-sepolia.env, envs/common/grafana.env, scripts, docker/prometheus/prometheus.yml). The envs/*/op-reth.env files do not carry the snapshot variable; it lives in the root mainnet.env and celo-sepolia.env.

  • op-node celo-v2.2.3: compose literal at docker-compose.yml:70; manifest 200; the release says it is required for Glamsterdam on Celo Sepolia (6 October) and adds Amsterdam L1 header support. IMAGE_TAG__OP_NODE is only an empty line in the env templates and the compose image has no variable, so the maintenance sentence is accurate.
  • eigenda-proxy v2.6.0: compose literal at :32; ghcr manifest 200; Layr-Labs/eigenda has release v2.6.0 (v2.7.0 and v2.7.1 exist, compose has not adopted them).
  • op-reth celo-v1.0.5: compose default at :48; manifest 200. The tag list also has celo-v1.0.6, which compose has not adopted; compose is the source of truth, so leaving v1.0.5 is right. I could not read the op-reth source repo to see what v1.0.6 contains.
  • geth-all 1.8.9: compose :3; manifest 200.
  • Grafana admin / celo: envs/common/grafana.env holds GF_SECURITY_ADMIN_PASSWORD=celo, the user is Grafana's default admin, and the compose README lists the same login. Compose publishes ${PORT__GRAFANA:-3000}:3000 with no loopback binding, and .env is listed after grafana.env in env_file, so the precedence claim holds (by compose ordering; not run).
  • OP_RETH__SNAPSHOT=true in both env templates and in the compose README ("the default"); the script comment says the same. historical-proofs.mdx:123 already agrees.
  • Port 7300: the healthcheck owns host 7300 (:28); op-node serves metrics on 7300 inside the network (start-op-node.sh --metrics.port=7300) and Prometheus scrapes op-node:7300.
  • No new internal links, so no anchors to check. Frontmatter, heading case and page types are untouched. The diff is small (6 files).

What the PR got right

  • Every changed value traces to a compose line, and the PR text was honest about what was not verified.
  • Catching that localhost:7300 is the healthcheck, not op-node, was a real operator trap.
  • Replacing the IMAGE_TAG__OP_NODE pin example with the accurate statement about which tags are fixed in compose.
  • The review round was handled in one clean commit, and the snapshot wording now matches the other open PR on the same lines.

Merge-order hazards

  • #2345 (draft, other contributor) conflicts with this PR in archive-node.mdx, configuration.mdx and run-node.mdx (git merge-tree of the two heads); maintenance.mdx and network-config.mdx merge cleanly. The OP_RETH__SNAPSHOT wording is now essentially the same in both, so the conflict is mechanical. The second merger should keep #2345's pre-L2 wording (the snapshot holds the Celo L1 state at the migration, not the pre-L2 blocks) and must not reintroduce "it provides the pre-L2 (Celo L1) history" from this PR's context lines.
  • #2377 touches only operate/notices/** and operate/specification/upgrades/jello.mdx; git merge-tree of the two heads is clean. One ordering note: the #2377 notices now say "For current versions see Network config", so this PR should merge first or together, otherwise that link points at op-node v2.1.0 and eigenda-proxy v1.8.2.
  • Separate from this PR: main has no Glamsterdam notice even though op-node celo-v2.2.3 is required for it on Celo Sepolia today. A maintainer should decide whether to file an issue.

… on celo-org#2376)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eddiemessiah

Copy link
Copy Markdown
Contributor Author

Thanks Lena.

Grafana password (Medium): fixed in ed9184cf. The warning on Monitoring now says to set GF_SECURITY_ADMIN_PASSWORD in .env before the first start with monitoring enabled, because Grafana reads it only when it creates the admin user. If Grafana has already started, change the password in Grafana (user icon > Change Password). Either way, firewall port 3000.

  • Grafana docs, admin_password: "Set once on first-run." (same wording in the v9.3 docs, which match the compose default image grafana/grafana:9.3.0)
  • Compose main (9634d6e), docker-compose.yml:141: - grafana_data:/var/lib/grafana, so the admin user persists across restarts.

I left out a CLI reset command: the binary name differs between Grafana 9.3 (grafana-cli) and current releases (grafana cli), and I did not confirm which one the container has.

This builds on Paul's merge of main (3c20055b); that merge touched none of the six operator pages. op-node is still celo-v2.2.3 on compose main, so the tags are current.

Previewed /operate/operators/monitoring locally at 1280px and 412px: the callout renders cleanly, no overflow, 0 console errors.

On merge order: agreed. This should merge before or together with #2377, since its notices link to Network config.

@GigaHierz GigaHierz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the current head: the findings from the earlier review are resolved, the diff merges cleanly into current main, and the changed claims still check out.

@GigaHierz
GigaHierz merged commit aed5755 into celo-org:main Oct 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants