Repository navigation
docs(operators): sync image versions, Grafana password and snapshot default with the compose repo - #2376
Conversation
GigaHierz
left a comment
There was a problem hiding this comment.
head 214fac8 · API headRefOid == local refs/pull ref (re-fetched); 1 commit, base 50ed71d, 1 behind main (a12466b, the home/ to learn/ rename); merge-tree against origin/main is clean; mint broken-links --check-redirects and check-orphans.sh pass on the merge; CI green (Check for broken links, GitGuardian, StepSecurity); rendered monitoring and network-config at 1280 and 412, 0 console errors
Verdict: REQUEST-CHANGES
Thanks for the careful audit of the operator pages against the compose repo. Most of what you changed checks out against primary sources, and the evidence table made it quick to verify. One finding is urgent, because the compose repo moved on after you opened the PR. Everything else is small.
Findings
1. High · operate/operators/network-config.mdx:60 and :93 · op-node tag celo-v2.2.1 is already stale and would mislead operators on a day it matters
The compose repo's main is now 9634d6e4 ("Bump op-node image to celo-v2.2.3 (#107)", 2026-10-06 08:46 UTC, about 20 minutes after this PR was opened), and docker-compose.yml:70 reads op-node:celo-v2.2.3. The celo-org/optimism release for celo-v2.2.3 says it is "required for the Glamsterdam activation" and that Celo Sepolia operators "must upgrade to this version prior to the Glamsterdam hardfork on October 6th". Release celo-v2.2.2 is batcher-only. So v2.2.1 is not just older: it is the version that predates the Amsterdam L1 header support.
Failure scenario: an operator building outside the compose repo (the exact case this page serves, per the PR text) copies the linked op-node:celo-v2.2.1 tag and runs it against Ethereum Sepolia after Glamsterdam. op-node cannot handle the new L1 headers and the node stops following the chain.
Fix shape: change both lines (mainnet and Celo Sepolia) to celo-v2.2.3, and re-read compose docker-compose.yml on the day you push, since it can move again. Both tags (v2.2.1 and v2.2.3) exist in the registry; eigenda-proxy:v2.6.0, op-reth:celo-v1.0.5 and geth-all:1.8.9 also resolve and still match compose. Optionally, to stop this drifting a third time, replace the pinned image list with a sentence plus a link to docker-compose.yml as the source of truth (your PR text already names this risk). That is a judgement call for the maintainers, so either approach resolves the finding.
2. Medium · operate/operators/monitoring.mdx:24 · a documented default credential with no instruction to change it
The value is correct (envs/common/grafana.env contains GF_SECURITY_ADMIN_PASSWORD=celo, and the compose README says the same), and publishing a default credential is fine. But compose publishes Grafana as ${PORT__GRAFANA:-3000}:3000, which binds all host interfaces, and the page is written for people running production nodes.
Failure scenario: an operator runs monitoring on a VPS without a firewall rule for 3000, and anyone who reads this page can log in as admin.
Fix shape: add a short <Warning> under the credentials saying these are defaults, and to set GF_SECURITY_ADMIN_PASSWORD in .env (compose lists .env after grafana.env in env_file, so it overrides) or to firewall port 3000. Do not describe the override as tested unless you run it.
3. Low · operate/operators/configuration.mdx:20 and run-node.mdx:79 · "default" is true for the shipped env templates, not for the start script
scripts/start-op-reth.sh:29 only downloads a snapshot when OP_RETH__SNAPSHOT is exactly true; if the variable is missing from a .env (for example one copied before the variable existed) there is no snapshot. The default comes from mainnet.env:55 and celo-sepolia.env:45. The compose README calls true "the default", so your wording matches upstream, but the narrower phrasing is the precise one.
Fix shape: write true (default in the shipped .env files), false, and keep run-node as "set OP_RETH__SNAPSHOT=false (the shipped .env files set it to true)". This is also the wording in #2345, which avoids a second conflict (see merge-order hazards).
4. Low · operate/operators/monitoring.mdx:36 · the corrected op-node line says where metrics are not, but not where to read them
"scraped by Prometheus at op-node:7300 on the compose network; this port is not published to the host, where localhost:7300 is the healthcheck" is accurate (prometheus.yml target op-node:7300, start-op-node.sh:48-50, op-node publishes only 9222 and 9545), but a reader skimming a "Key metrics to watch" list still has no URL to open.
Fix shape: "From op-node (query in Prometheus at http://localhost:9090 or in Grafana; the op-node port 7300 is not published to the host, where localhost:7300 is the healthcheck)". Prometheus is published as ${PORT__PROMETHEUS:-9090}:9090 and the page already names it two paragraphs above.
Verified good
Checked against compose main (9634d6e4 at review time, and e90c955d which you cited), the registries and release tags:
- eigenda-proxy
v2.6.0: composedocker-compose.yml:32literal; manifest returns HTTP 200 from ghcr.io;Layr-Labs/eigendahas release v2.6.0. (The oldLayr-Labs/eigenda-proxyrepo is archived, which is consistent with the image now being published from the monorepo.) - op-reth
celo-v1.0.5unchanged and correct: compose line 48 default; tag present in the Artifact Registry tag list (v1.0.6 also exists but compose has not adopted it, so leaving v1.0.5 is right). - geth-all
1.8.9: compose lines 3; manifest returns 200. - Same images on both networks: the image lines take no network variable.
- Grafana user
admin/ passwordcelo:envs/common/grafana.env:1and README lines 262-263. OP_RETH__SNAPSHOT=trueinmainnet.env:55andcelo-sepolia.env:45; README "Snapshot modes" says "(the default)"; script comment atstart-op-reth.sh:26. Agrees with the new text in configuration, run-node, archive-node and the Celo Sepolia line of network-config.historical-proofs.mdx:123already says "the default", so it now agrees.- Healthcheck owns host port 7300 (
docker-compose.yml:27-28); op-node publishes only 9222/udp, 9222/tcp, 9545 (:81-83); op-node metrics on 7300 inside the network. IMAGE_TAG__OP_NODEhas no effect: the op-node image tag is a literal indocker-compose.yml, and the variable appears only as an empty line in the env templates. The maintenance.mdx sentence is accurate.- Not read: nothing was blocked. I read
envs/common/grafana.env,mainnet.envandcelo-sepolia.envthroughgh api. I did not run compose or log in to a Grafana, so the claim that.envoverridesgrafana.envrests on composeenv_fileordering, not on a run. - Anchors and links: the diff adds no internal links;
mint broken-links --check-redirectsandscripts/check-orphans.shpass on a merge into current main. Frontmatter, heading case and page structure are untouched; no promotional wording.
What the PR got right
- Scoped to values you checked, with file:line evidence and an honest residual-risk section. That is why the one stale value is easy to fix.
- Catching that
localhost:7300is the healthcheck and not op-node was a real operator trap. - Removing
IMAGE_TAG__OP_NODEas a pin example is correct, and raising the compose-repo gap as a question rather than papering over it is the right call. - Leaving archived notices alone is right; their versions are correct for their dates (but see the cross-PR note below).
- Line endings and diff size are tight (6 files, 11 insertions, 11 deletions).
Merge-order hazards
- #2345 (draft, other contributor) edits the same lines and conflicts with this PR in
archive-node.mdx(the snapshot paragraph, line 70),configuration.mdx(theOP_RETH__SNAPSHOTrow, line 20) andrun-node.mdx(the "From genesis" bullet, line 79).git merge-treeof the two heads reports exactly those three files;maintenance.mdxandnetwork-config.mdxmerge cleanly. Both PRs agree the default istrue, so there is no factual contradiction, but #2345 wording ("default in the shipped.envfiles") is the more precise one, so the second merger should keep it. - Both this PR and #2345 leave or restate "it provides the pre-L2 (Celo L1) history" for the mainnet snapshot in those three places; #2345 argues that claim is wrong (the snapshot holds only the Celo L1 state at the migration). Whichever merges first, the other should not reintroduce the old sentence when resolving the conflict.
- #2351 is already merged and this branch is based on it; no overlap remains.
- #2377 touches different files; the two merge in any order. #2367 (home/ to learn/) is already on main; this PR touches only
operate/, so a rebase is trivial. #2368 (build/ move) has not landed and does not touch these files. - Once the op-node tag is bumped, the archived notices still say "We recommend the latest released versions: ... op-node v2.1.0" (jello, fusaka, Celo Sepolia launch). That present-tense recommendation is now wrong; it is tracked in the #2377 review.
- Separate from this PR: docs on main have no Glamsterdam notice even though op-node
celo-v2.2.3is required for it on Celo Sepolia; a maintainer should decide whether to file an issue for that.
|
Previewed in a local live deployment (
Each page returns 200 and renders the new text, with no horizontal overflow on phone, no broken images and 0 console errors. |
…efault with the compose repo Align the operator pages with celo-l2-node-docker-compose main (e90c955): - network-config: op-node celo-v2.1.0 -> celo-v2.2.1 and eigenda-proxy v1.8.2 -> v2.6.0 for mainnet and Celo Sepolia (docker-compose.yml); Celo Sepolia snapshot is on by default - monitoring: Grafana admin password is celo (envs/common/grafana.env); op-node metrics on 7300 are scraped inside the compose network, not published to the host - configuration, run-node, archive-node: OP_RETH__SNAPSHOT defaults to true in both network .env files and scripts/start-op-reth.sh - maintenance: IMAGE_TAG__OP_NODE does not pin op-node; its tag is fixed in docker-compose.yml Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a warning, snapshot wording, metrics URL) - network-config: op-node celo-v2.2.1 -> celo-v2.2.3 on the mainnet and Celo Sepolia lines, matching compose main 9634d6e4 (2026-10-06); tag confirmed in the registry - monitoring: add a Warning under the Grafana credentials to set GF_SECURITY_ADMIN_PASSWORD in .env or firewall port 3000 - monitoring: the op-node metrics line now says where to read them (Prometheus at localhost:9090 or Grafana) - configuration, run-node: "default" is scoped to the shipped .env files, matching celo-org#2345 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
214fac8 to
db81e21
Compare
|
Thanks for the review. Head is now
Source for the tag: Checks on
Merge order: #2345 edits three of the same lines ( |
|
Follow-up on the one item I had left open: the local preview of All six operator pages return 200:
The new text is in the served HTML and renders: Screenshots of the four changed pages at both sizes are available if you want them attached here. |
GigaHierz
left a comment
There was a problem hiding this comment.
head db81e21 · API headRefOid == local refs/pull ref (re-fetched); 2 commits on base 7b8f97c = current origin/main (0 behind); merge-tree against origin/main clean; mint broken-links --check-redirects and check-orphans.sh pass on the head; compose main still 9634d6e4 (celo-v2.2.3); CI: Documentation Validation is action_required (waiting for a maintainer to approve the workflow run, not a failure; GitGuardian and StepSecurity pass); rendered monitoring on port 3621: warning present, password celo, no horizontal overflow, 0 console errors
Verdict: REQUEST-CHANGES
Thank you for the quick turnaround. The op-node tag, the Grafana warning, the snapshot wording and the metrics hint are all fixed, and I re-checked each against the compose repo and the registries. One small thing is left, and it is about the new warning itself: it tells operators to do something that only works before Grafana's first start.
Status of earlier findings
- High, op-node tag stale: fixed. Both lines in
network-config.mdx(:60 mainnet, :93 Celo Sepolia) now readcelo-v2.2.3. Composedocker-compose.yml:70still readsop-node:celo-v2.2.3(compose main is9634d6e4, nothing newer), thecelo-org/optimismreleasecelo-v2.2.3is marked Latest, and the registry returns 200 for that tag. - Medium, Grafana default credential: fixed in substance (a
<Warning>was added), but see finding 1. - Low, "default" wording for
OP_RETH__SNAPSHOT: fixed.configuration.mdx:20saystrue(default in the shipped.envfiles), andrun-node.mdx:79says "the shipped.envfiles set it totrue". Both matchmainnet.env:55andcelo-sepolia.env:45, andstart-op-reth.shonly downloads when the value is exactlytrue. - Low, where to read op-node metrics: fixed.
monitoring.mdx:40now points at Prometheus onhttp://localhost:9090or Grafana and says 7300 is not published. Prometheus is published as${PORT__PROMETHEUS:-9090}:9090, and op-node publishes only 9222 and 9545.
Findings
1. Medium · operate/operators/monitoring.mdx:27 · the Grafana warning's fix does nothing once Grafana has started once
The warning says to set GF_SECURITY_ADMIN_PASSWORD in .env "before the node is reachable from a network". Grafana's documentation for admin_password says "Set once on first-run", and compose keeps Grafana's state in the named volume grafana_data (docker-compose.yml:141), which survives restarts and docker compose down.
Failure scenario: an operator who has had monitoring running for weeks reads the warning, adds GF_SECURITY_ADMIN_PASSWORD=<new> to .env, restarts, and believes the default is gone. Grafana ignores the variable because the admin user already exists in the volume, so admin / celo still logs in on a port that is open to the internet.
Fix shape: say when the variable takes effect and what to do otherwise, for example "Set GF_SECURITY_ADMIN_PASSWORD in .env before the first start with monitoring enabled; Grafana reads it only when it creates the admin user. If Grafana has already started, change the password in Grafana (profile, Change password) or firewall port 3000." (I did not run an override or log in to a Grafana, so I have not tested either path. The first-run behaviour comes from the Grafana docs.)
Verified good
Checked against compose main (9634d6e4, 2026-10-06 08:46 UTC), the registries and release tags. Read through gh api; every file I needed was readable (mainnet.env, celo-sepolia.env, envs/common/grafana.env, scripts, docker/prometheus/prometheus.yml). The envs/*/op-reth.env files do not carry the snapshot variable; it lives in the root mainnet.env and celo-sepolia.env.
- op-node
celo-v2.2.3: compose literal atdocker-compose.yml:70; manifest 200; the release says it is required for Glamsterdam on Celo Sepolia (6 October) and adds Amsterdam L1 header support.IMAGE_TAG__OP_NODEis only an empty line in the env templates and the compose image has no variable, so the maintenance sentence is accurate. - eigenda-proxy
v2.6.0: compose literal at:32; ghcr manifest 200;Layr-Labs/eigendahas release v2.6.0 (v2.7.0 and v2.7.1 exist, compose has not adopted them). - op-reth
celo-v1.0.5: compose default at:48; manifest 200. The tag list also hascelo-v1.0.6, which compose has not adopted; compose is the source of truth, so leaving v1.0.5 is right. I could not read the op-reth source repo to see what v1.0.6 contains. - geth-all
1.8.9: compose:3; manifest 200. - Grafana
admin/celo:envs/common/grafana.envholdsGF_SECURITY_ADMIN_PASSWORD=celo, the user is Grafana's defaultadmin, and the compose README lists the same login. Compose publishes${PORT__GRAFANA:-3000}:3000with no loopback binding, and.envis listed aftergrafana.envinenv_file, so the precedence claim holds (by compose ordering; not run). OP_RETH__SNAPSHOT=truein both env templates and in the compose README ("the default"); the script comment says the same.historical-proofs.mdx:123already agrees.- Port 7300: the healthcheck owns host 7300 (
:28); op-node serves metrics on 7300 inside the network (start-op-node.sh --metrics.port=7300) and Prometheus scrapesop-node:7300. - No new internal links, so no anchors to check. Frontmatter, heading case and page types are untouched. The diff is small (6 files).
What the PR got right
- Every changed value traces to a compose line, and the PR text was honest about what was not verified.
- Catching that
localhost:7300is the healthcheck, not op-node, was a real operator trap. - Replacing the
IMAGE_TAG__OP_NODEpin example with the accurate statement about which tags are fixed in compose. - The review round was handled in one clean commit, and the snapshot wording now matches the other open PR on the same lines.
Merge-order hazards
- #2345 (draft, other contributor) conflicts with this PR in
archive-node.mdx,configuration.mdxandrun-node.mdx(git merge-treeof the two heads);maintenance.mdxandnetwork-config.mdxmerge cleanly. TheOP_RETH__SNAPSHOTwording is now essentially the same in both, so the conflict is mechanical. The second merger should keep #2345's pre-L2 wording (the snapshot holds the Celo L1 state at the migration, not the pre-L2 blocks) and must not reintroduce "it provides the pre-L2 (Celo L1) history" from this PR's context lines. - #2377 touches only
operate/notices/**andoperate/specification/upgrades/jello.mdx;git merge-treeof the two heads is clean. One ordering note: the #2377 notices now say "For current versions see Network config", so this PR should merge first or together, otherwise that link points at op-node v2.1.0 and eigenda-proxy v1.8.2. - Separate from this PR: main has no Glamsterdam notice even though op-node
celo-v2.2.3is required for it on Celo Sepolia today. A maintainer should decide whether to file an issue.
… on celo-org#2376) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks Lena. Grafana password (Medium): fixed in
I left out a CLI reset command: the binary name differs between Grafana 9.3 ( This builds on Paul's merge of Previewed On merge order: agreed. This should merge before or together with #2377, since its notices link to Network config. |
GigaHierz
left a comment
There was a problem hiding this comment.
Re-reviewed the current head: the findings from the earlier review are resolved, the diff merges cleanly into current main, and the changed claims still check out.
The hole, and the fix
The operator pages had drifted from celo-l2-node-docker-compose, the setup they document. An operator who followed them would pull the wrong op-node and eigenda-proxy tags if building outside the compose repo, fail to log in to Grafana, misread the snapshot default, try to pin op-node with a variable the compose file never reads, and open the healthcheck's metrics while expecting op-node's.
This PR changes only values checked against compose
mainat9634d6e465197a120e6bd7d80ba9f249b4a7a8d9("Bump op-node image to celo-v2.2.3 (#107)", 2026-10-06):network-config.mdx: op-nodecelo-v2.1.0->celo-v2.2.3and eigenda-proxyv1.8.2->v2.6.0, mainnet and Celo Sepolia. op-rethcelo-v1.0.5and the Celo L1 clientgeth-all:1.8.9already match and are unchanged. The snapshot line now says snapshots are on by default.monitoring.mdx: Grafana admin passwordoptimism->celo, plus a warning: compose publishes Grafana on every host interface,GF_SECURITY_ADMIN_PASSWORDmust be set in.envbefore the first start with monitoring enabled (Grafana reads it only when it creates the admin user, and the user persists in thegrafana_datavolume), after that the password is changed in Grafana, and port 3000 should be firewalled. The op-node metrics line no longer points athttp://localhost:7300/metrics; that host port belongs to the healthcheck container. op-node's port 7300 is scraped by Prometheus on the compose network and is not published to the host.configuration.mdx,run-node.mdx,archive-node.mdx:OP_RETH__SNAPSHOTistruein the shipped.envfiles, notfalse. "leaveOP_RETH__SNAPSHOT=false" becomes "set".maintenance.mdx: the pinning example is nowIMAGE_TAG__OP_RETHonly, with one sentence saying the op-node and eigenda-proxy tags are fixed indocker-compose.ymland change only ongit pull.IMAGE_TAG__OP_NODEhas no effect.What this does NOT do / residual risk
operate/notices/archive/*) still list op-nodev2.1.0and eigenda-proxyv1.8.2. That is correct for each notice's date, so they are unchanged.mainon 2026-10-06. They will drift again on the next compose bump.adminusername were not checked against the dashboard JSON or a running Grafana. They are unchanged.grafana-cli, the compose default image) and current releases (grafana cli), and I did not confirm which is on the path in the container.IMAGE_TAG__OP_NODEis still listed inmainnet.env/celo-sepolia.envin the compose repo even thoughdocker-compose.ymldoes not read it. That belongs in the compose repo and is not fixed here.Judgement calls
monitoring.mdxop-node metrics line: I corrected the endpoint instead of only changing the password, because the old URL returns healthcheck metrics. One-line revert if a maintainer would rather publish op-node's 7300 in compose.maintenance.mdx: I droppedIMAGE_TAG__OP_NODErather than opening a compose-repo issue to wire the variable up. Happy to drop this hunk; the rest stands without it.Issues
Refs: none filed. Found by an audit of the operator pages against the compose repo.
Stacking / conflicts
Based on current
mainthrough the maintainer's merge ofmain(3c20055b, parent418506f6). The merge touched no file in this PR. Touches onlyoperate/operators/{archive-node,configuration,maintenance,monitoring,network-config,run-node}.mdx. Should merge before or with #2377, whose notices link to Network config.Verification evidence
Source:
celo-org/celo-l2-node-docker-compose@9634d6e465197a120e6bd7d80ba9f249b4a7a8d9(main on 2026-10-08), fetched withgh api repos/celo-org/celo-l2-node-docker-compose/contents/<path>.celo-v2.2.3docker-compose.yml:70image: us-west1-docker.pkg.dev/devopsre/celo-blockchain-public/op-node:celo-v2.2.3v2.6.0docker-compose.yml:32image: ghcr.io/layr-labs/eigenda-proxy:v2.6.0celo-v1.0.5(unchanged)docker-compose.yml:48...op-reth}:${IMAGE_TAG__OP_RETH:-celo-v1.0.5}geth-all:1.8.9(unchanged)docker-compose.yml:3image: us-docker.pkg.dev/celo-org/us.gcr.io/geth-all:1.8.9celoenvs/common/grafana.env:1GF_SECURITY_ADMIN_PASSWORD=celo.envoverridesgrafana.envdocker-compose.yml:133-135env_file: - ./envs/common/grafana.env - .envdocker-compose.yml:141- grafana_data:/var/lib/grafanadocker-compose.yml:144- ${PORT__GRAFANA:-3000}:3000scripts/start-grafana.shif [ "$MONITORING_ENABLED" = "true" ]; then exec /run.shOP_RETH__SNAPSHOT=true(mainnet)mainnet.env:55OP_RETH__SNAPSHOT=trueOP_RETH__SNAPSHOT=true(Celo Sepolia)celo-sepolia.env:45OP_RETH__SNAPSHOT=truescripts/start-op-reth.sh:26# scratch. On by default (OP_RETH__SNAPSHOT=true); set it to false to sync fromdocker-compose.yml:27-28(healthcheck service)ports: - ${PORT__HEALTHCHECK_METRICS:-7300}:7300docker-compose.yml:80-83(op-node service)9222/udp,9222/tcp,9545scripts/start-op-node.sh:50;docker/prometheus/prometheus.yml:20--metrics.port=7300;targets: ["op-node:7300"]IMAGE_TAG__OP_NODEnot readdocker-compose.ymlgrep -c IMAGE_TAG__OP_NODE docker-compose.ymlreturns0Grafana: the
admin_passwordsetting is documented as "Set once on first-run" in both the current and v9.3 configuration docs (compose defaults tografana/grafana:9.3.0). The in-app path (user icon, then Change Password) is from the v9.3 user preferences docs.Line endings: the six files keep CRLF in the working tree. The PR diff against
main(418506f6) is 6 files changed, 15 insertions(+), 11 deletions(-).Preview
npx mint dev --port 3642on this branch:/operate/operators/monitoringreturns 200 with the new warning text. Checked in a browser at 1280px and 412px: the warning renders inside the callout, no horizontal overflow (scrollWidth412 at 412px), 0 console errors.mint broken-links(Windows):found 586 broken links in 125 fileson this branch and the same list, line for line, with the six files reverted tomain(418506f6). The report flags pages that exist (for example/operate/operators/monitoring), so it is a Windows path-resolution artifact. This PR adds no internal links. CI's Linux run is the authoritative check.Remaining ops steps
Checklist
mainlint/typecheck/testpass locally (n/a;mint broken-linksresult above)Closesissue (none closed)IMAGE_TAG__OP_NODEgap be fixed in the compose repo (wire the variable intodocker-compose.yml) rather than documented around?🤖 Generated with Claude Code