Repository navigation
FUG-128: CI observability — failure dashboarding (BEP → inline report + Grafana + columnar sinks) - #108
Open
issuefleet[bot] wants to merge 8 commits into
Open
FUG-128: CI observability — failure dashboarding (BEP → inline report + Grafana + columnar sinks)#108issuefleet[bot] wants to merge 8 commits into
issuefleet[bot] wants to merge 8 commits into
Conversation
Parse bazel's Build Event Protocol (--build_event_json_file) into normalized per-test-case records: follow each TestResult to its test.xml (JUnit) for per-case pass/fail + trace, capture build-failure targets from failed actions, bucket failures (disk/memory/timeout/network/build/assertion/other) and compute a scrubbed failure signature for trace-level aggregation. Flaky runs (pass after an earlier fail) surface as FLAKY. Emits NDJSON records, a self-contained static HTML report (CSS/SVG charts + collapsible per-target details), and a markdown step-summary. Pluggable sinks (Grafana Cloud Loki, ClickHouse-compatible columnar DB) that no-op without credentials. Stdlib-only; 18 unit tests green under bazel. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reusable composite action .github/actions/bep-report: parses a bazel
--build_event_json_file into the HTML report, uploads it as an artifact, links
it from the job summary + an ::notice:: at the end of the log, and pushes
per-test-case records to the sinks. Wired (always()) into every bazel-running
job: test/firmware/build-site (test.yaml), hitl_tests (hitl.yaml), test-macos
(macos.yaml), each tagged with its runner label (github:linux|macos, hitl).
Sink creds are read from repository secrets via a workflow-level env block so
every job — with or without a GitHub environment — inherits them; fork PRs get
empty values and the report/sinks no-op.
schema/ci_test_results.{sql,datasource}: the columnar table DDL for the
ClickHouse-compatible sink (Tinybird free Build plan / ClickHouse Cloud /
self-hosted), with the three dashboard reference queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
observability/ci/dashboards/ci-failures.json: Loki-backed dashboard with the three required views — failures by signature/trace (table), by category (bar), by runner/DUT (bar + timeseries), and a per-test-case failure heatmap (color- graded table) plus a recent-failures drill-down. runner/workflow template vars. grafana-dashboards.yaml now also syncs observability/ci/dashboards/ (reuses the existing 'Grafana' env creds). Full README covering the pipeline, the columnar DB decision (Tinybird free Build plan vs Loki default), and the single credential-setup step. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…status
Validated against real bazel output: bazel's synthesized test.xml for a FAILED
test keeps status="run" with no <failure> element (the failure lives only in
test.log), so trusting the XML alone under-reports failures. The BEP
TestResult.status is now authoritative: a failed attempt whose XML shows no
failing case falls back to a synthetic failing case, and a lone generic
failure ('exited with error code 1') is enriched with the test.log tail so the
real cause is visible in the HTML trace. Real per-case JUnit XML (pytest
--junitxml) is left untouched. +2 regression tests (20 total).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
Kevin picked Tinybird — but Tinybird Forward ingests via its Events API (POST /v0/events?name=<ds>), not a raw ClickHouse INSERT endpoint, so the existing clickhouse sink wouldn't reach it. Add a dedicated push_tinybird sink (TINYBIRD_API_URL/TOKEN/DATASOURCE, no-ops without creds) and wire the secrets into all three workflows. clickhouse sink stays for real ClickHouse/Cloud/ self-hosted. Rewrite the README config section into two explicit fully-free recipes: (A) Grafana Cloud Loki only — the shipped dashboard, least effort; (B) Tinybird columnar via the Events API, mapped from the Tinybird quick-start (only the ci_test_results datasource + an append token; skip the taxi sample, the pipe, and Tinybird Local). +2 sink tests (22 total). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Kevin confirmed the Grafana Cloud Loki write uses HTTP basic_auth (numeric instance id + Grafana.com API token) — which push_loki already does. Spell that out in the Recipe A table so it's unambiguous the same token/URL work unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address PR feedback (Kevin): 1. Zip link → inline report. GitHub only serves artifacts as .zip (no direct-open URL), so the full report is now rendered as GitHub-flavored markdown into $GITHUB_STEP_SUMMARY — it opens right on the run's Summary tab, no download. The styled HTML stays as an optional artifact download, relabeled as such. 2. Missing logs. Records now carry log_excerpt (the target's test.log tail) for EVERY target, pass or fail — so expanding a target always shows its output. Build failures with no attributed action fall back to the bazel console (captured from BEP progress events), so compiler output is shown instead of a bare 'build failed'. A 'Console output' section carries the full bazel log. log_excerpt is report-only (REPORT_ONLY_FIELDS) so it never bloats the sinks. 3. Grafana dashboard: confirmed — observability/ci/dashboards/ci-failures.json pulls from Loki and aggregates across all runners by default (runner=All), with explicit by-runner/DUT panels. (No change needed.) +4 tests (28 total): passing-target log capture, console capture, build-failure console fallback, own-stderr, inline markdown report. Validated against real bazel BEP (passing py_test + failing sh_test). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issuefleet
Bot
changed the title
The hitl_tests lane is the first job with LIVE Loki creds, so push_loki actually hits the network there. push_loki only caught HTTPError/URLError; an unforeseen error type (bare socket timeout, SSLError, ConnectionReset) would have escaped and failed the CI step. Two layers of defense: - push_all now wraps every sink in a broad try/except so a push can never propagate (with a per-sink test that a raising sink is swallowed). - the bep-report composite action runs the generator out from under set -e and swallows any failure (::warning:: + have_report=false), and the upload/link steps are continue-on-error. A report/telemetry hiccup can no longer fail a build. This also de-risks the additive change on every wired job. (If the hitl failure was instead a hardware flake, a re-run clears it; this makes the report step provably incapable of causing it.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI Observability: failure dashboarding (FUG-128)
Turns every Bazel-running CI job into queryable failure telemetry, a browsable
HTML report, and a Grafana dashboard — answering the three questions in the
issue: failures by trace/reason, by runner/DUT, and a heatmap by test
case. Built on the existing FUG-117 Grafana Cloud infrastructure so the only
manual step is credentials (most of which already exist).
What's here
tools/ci_observability/bep_report.py— a stdlib-only parser for Bazel'sBuild Event Protocol (
--build_event_json_file). It follows eachTestResultto its
test.xml(JUnit) for per-test-case pass/fail + trace, capturesbuild-failure targets from failed actions, buckets failures
(
disk/memory/timeout/network/build/assertion/other), and computes a scrubbedfailure signature so identical failures group across runs. The BEP
TestResultstatus is authoritative (bazel's synthesizedtest.xmlfor afailed test carries no
<failure>element — the failure is only intest.log, which enriches the trace). Validated against real bazel output forboth pass and fail paths; 20 unit tests green under bazel.
markdown into
$GITHUB_STEP_SUMMARYso it opens directly on the run'sSummary tab (no zip download), with a collapsible block per target showing
per-case results and the target's full
test.log/ build stderr (passingtargets too) plus the bazel console output. The same report is also uploaded
as a self-contained styled HTML artifact for offline viewing.
sinks.py) — Grafana Cloud Loki (default; reuses theFUG-117 credentials), a Tinybird sink (its Events API — the correct
ingestion path for Tinybird Forward), and a ClickHouse sink (native
INSERTfor ClickHouse Cloud / self-hosted); each no-ops without credentials,so it's safe on fork PRs.
.github/actions/bep-reportwired (always())into every bazel test/build job:
test/firmware/build-site(test.yaml),
hitl_tests(hitl.yaml),test-macos(macos.yaml) — each taggedwith its runner label (
github:linux/github:macos/hitl).observability/ci/dashboards/ci-failures.json(Loki-backed, three required views + drill-down).
grafana-dashboards.yamlnow syncs it too, reusing the existing "Grafana" environment.
schema/ci_test_results.{sql,datasource}: ClickHouse DDL +Tinybird datasource. Recommendation: Tinybird's free "Build" plan (managed
ClickHouse — free, fully-hosted, well-supported; ClickHouse Cloud is
trial-only). Loki is the zero-new-account default so the dashboard works the
moment the (already-existing) Loki secrets are present.
The only manual step: credentials
See
tools/ci_observability/README.md→ "Configuration — pick a recipe" (twofully-free options). Recipe A (simplest): copy the
GRAFANA_CLOUD_LOKI_*values (already in the FUG-117 "HITL" environment) to repository secrets and
add a Loki data source — the shipped dashboard lights up. Recipe B
(columnar):
tb deploytheci_test_resultsdata source and setTINYBIRD_API_URL/TINYBIRD_TOKEN. The dashboard sync reuses the existing"Grafana" environment. The HTML report + artifact link need no credentials at
all. Nothing else to stand up.
Testing
bazel test //tools/ci_observability:bep_report_test— 20 unit tests(BEP parsing, per-case granularity, categorization, signature scrubbing,
build failures, flaky detection, BEP-status reconciliation, HTML render, sink
no-op).
bazel test/sh_testBEP output (pass + fail).preklint suite.🤖 Generated with Claude Code
Closes-Linear: FUG-128 (https://linear.app/fughilli/issue/FUG-128/ci-observability-failure-dashboarding)