Skip to content

health-endpoints.mdx

Latest commit

 

History

History
274 lines (215 loc) · 12.5 KB

health-endpoints.mdx

File metadata and controls

274 lines (215 loc) · 12.5 KB
Raw
title Health Endpoints
description Two health endpoints World Monitor exposes for monitoring data pipeline integrity — Edge Runtime handlers querying Upstash Redis for system state.

/api/health

Primary health endpoint. Checks all Redis-backed data keys and seed freshness metadata in a single pipeline call.

Authentication: Compact health (?compact=1) is public for uptime and keyword monitors. Detailed health (/api/health without compact=1) and the operator history view (?history=1) require a valid operator/enterprise API key because they expose canonical Redis key names, record counts, and freshness thresholds. Browser origins must still pass the CORS allowlist in api/_cors.js; requests with no Origin header, such as server-side monitors, are allowed only for compact health unless they include an operator key. Health responses are never cached (Cache-Control: private, no-store, max-age=0 and CDN-Cache-Control: no-store).

HTTP Method: GET

Query Parameters

Parameter Values Description
compact 1 Omit per-key details; only return keys with problems

Response Status Codes

HTTP Status Overall Status Meaning
200 HEALTHY All checks OK, no warnings
200 WARNING Some keys stale or on-demand keys empty, but no critical failures
200 DEGRADED Critical keys empty, but ≤3% of all probed keys
200 UNHEALTHY Critical keys empty, >3% of all probed keys
401 - Detailed health or history requested without a valid operator/enterprise API key
503 REDIS_DOWN Could not connect to Redis — the only state that returns a non-200 code

The overall health verdict lives in the JSON status field, not the HTTP code. Every state except REDIS_DOWN returns 200 so warn-level seed jitter doesn't flap HTTP-status monitors (see PR #2699). REDIS_DOWN returns 503 because with Redis unreachable the endpoint can assess nothing, so a plain HTTP probe must see a failure.

Response Body

{
  "status": "HEALTHY | WARNING | DEGRADED | UNHEALTHY | REDIS_DOWN",
  "summary": {
    "total": 194,
    "ok": 180,
    "warn": 5,
    "onDemandWarn": 9,
    "staleContent": 0,
    "crit": 0
  },
  "checkedAt": "2026-03-11T14:00:00.000Z",
  "checks": {
    "earthquakes": {
      "status": "OK",
      "records": 142,
      "seedAgeMin": 8,
      "maxStaleMin": 30
    }
  }
}

summary fields: total is one entry per probed key (currently ~194 and growing as panels are added). warn excludes on-demand-empty keys — those are surfaced separately as onDemandWarn so they don't drive the overall verdict to WARNING. staleContent is a subset of warn (fresh seeder but the upstream feed stopped advancing). Only crit (EMPTY/EMPTY_DATA) drives DEGRADED/UNHEALTHY.

With ?compact=1, the checks object is replaced by problems containing only non-OK keys.

Key Classifications

Keys are grouped into three tiers that determine alert severity:

Tier Severity when empty Description
Bootstrap CRIT Seeded data required at startup. Empty means the dashboard is missing critical data
Standalone CRIT (seeded) / WARN (on-demand) Populated by seed loops or RPC handlers. On-demand keys are expected to be empty until first request
On-demand WARN Populated lazily by RPC calls. Empty is normal if nobody has requested the data yet

Per-Key Statuses

Status Severity Meaning
OK Green Data present, seed fresh
OK_CASCADE Green Key empty but a sibling in the cascade group has data (e.g., theater posture fallback chain)
NOT_CONFIGURED Green An optional source adapter this deployment never supplied a credential for (producer wrote sourceState: "unavailable", e.g. the Global Tenders SAM.gov adapter without SAM_GOV_API_KEY). Not a fault and not counted as a problem; flips to OK on the first run after the credential is added
STALE_SEED Warn Data present but seed-meta age exceeds maxStaleMin
STALE_CONTENT Warn Seeder is fresh but the upstream content stopped advancing (frozen feed); counted in summary.staleContent
COVERAGE_PARTIAL Warn Record count is below the key's declared minRecordCount (e.g. 10/13 chokepoints or 139/174 PortWatch port-activity countries)
SEED_ERROR Warn seed-meta reports status: "error" from the last seed run
REDIS_PARTIAL Warn A single per-command Redis error on this key's STRLEN/GET (not a full outage)
EMPTY_ON_DEMAND Warn On-demand key has no data yet (expected until first request); counted in summary.onDemandWarn
EMPTY Crit Bootstrap or seeded standalone key has no data
EMPTY_DATA Crit Key exists but contains zero records (and 0 is not a valid state for it)

China Coverage Projection

chinaCoverage projects the hourly Railway summary at health:china-coverage:v1. The evaluator checks each launched China contract for both a fresh producer heartbeat and fresh, substantive China content; a fresh seed cannot hide stale or missing source content. CHINA_DEGRADED is a warning projection for partial or stale coverage, while CHINA_UNAVAILABLE is critical when the summary is invalid or the launched content is unavailable. Blocked China contracts remain visible in the audit with their stable reason code but are excluded from the strict launched-entry health count.

Operators can obtain the same sanitized, read-only audit with node scripts/audit-china-coverage.mjs --json; add --strict to return a nonzero exit code unless every launched entry is healthy. The audit reads only the compact Redis contracts and emits status, age, and reason-code summaries— never credentials or raw upstream payloads.

Cascade Groups

Some keys use fallback chains. If any sibling has data, empty siblings report OK_CASCADE:

  • Theater Posture: theaterPostureLive -> theaterPosture (stale) -> theaterPostureBackup
  • Military Flights: militaryFlights -> militaryFlightsStale
  • Displacement: displacement (current UTC year) -> displacementPrev (prior year, covers the Jan-1 window before the new-year seed runs)

riskScores is intentionally stricter than a raw feed heartbeat. Its recordCount is realtime signal-density coverage: the count of score-relevant Tier-1 conflict, news, and cyber signal families present during the CII refresh. The conflict family is satisfied by either the ACLED path or the UCDP event feed, matching the CII v8 scorer. When those feeds are reachable but quiet, riskScores can still report COVERAGE_PARTIAL; underlying feed freshness is tracked by the source-specific health entries where those feeds publish seed metadata.

portwatchPortActivity also uses minRecordCount. A fresh seed-meta:supply_chain:portwatch-ports record below 174 countries reports COVERAGE_PARTIAL instead of OK; partial runs may still refresh per-country PortWatch cache entries, but the canonical country list and healthy seed-meta signal do not advance until full 174-country coverage returns.

Staleness Thresholds (maxStaleMin)

Selected thresholds from SEED_META:

Domain Max Stale (min) Notes
Market quotes, crypto, sectors, earthquakes, insights 30 High-frequency relay loops / critical event data
Military flights 30 Near-real-time tracking
Predictions, flight delays (FAA) 90 Polymarket / airport snapshots
Unrest 120 45min cron, 2h grace
Cyber threats 240 APT data updated less frequently
Wildfires 360 FIRMS NRT accumulates over hours
Climate anomalies 540 3h cron, 3× cadence
BIS extended, World Bank, IMF 2160-100800 Institutional data, weekly/monthly/annual

These are illustrative; SEED_META in api/health.js is the source of truth and each entry documents its own cadence rationale.

Example Requests

# Full health check (requires an operator API key)
curl -s https://api.worldmonitor.app/api/health \
  -H "X-WorldMonitor-Key: $WORLDMONITOR_API_KEY" | jq .

# Compact (problems only)
curl -s "https://api.worldmonitor.app/api/health?compact=1" | jq .

# UptimeRobot / monitoring: check HTTP status code
curl -o /dev/null -s -w "%{http_code}" "https://api.worldmonitor.app/api/health?compact=1"
# Returns 503 only when Redis is unreachable (REDIS_DOWN); 200 for every other
# state — the verdict (HEALTHY/WARNING/DEGRADED/UNHEALTHY) is in the body's `status`.

/api/seed-health

Focused endpoint for seed loop freshness. Checks only seed-meta:* keys without fetching actual data payloads.

Authentication: Requires valid API key or allowed origin.

HTTP Method: GET

Response Status Codes

HTTP Status Overall Status Meaning
200 healthy All seed loops reporting on time
200 warning Some seeds stale (age > 2x interval) or below a declared coverage floor
200 degraded Some seeds missing entirely
401 - Invalid or missing API key
503 - Redis unavailable

Response Body

{
  "overall": "healthy | warning | degraded",
  "checkedAt": 1710158400000,
  "seeds": {
    "seismology:earthquakes": {
      "status": "ok",
      "fetchedAt": 1710158100000,
      "recordCount": 142,
      "sourceVersion": null,
      "ageMinutes": 5,
      "stale": false
    },
    "market:stocks": {
      "status": "stale",
      "fetchedAt": 1710150000000,
      "recordCount": 85,
      "sourceVersion": null,
      "ageMinutes": 140,
      "stale": true
    },
    "supply_chain:portwatch-ports": {
      "status": "coverage_partial",
      "fetchedAt": 1710158100000,
      "recordCount": 139,
      "minRecordCount": 174,
      "sourceVersion": null,
      "ageMinutes": 5,
      "stale": true
    }
  }
}

Staleness Logic

A seed is considered stale when its age exceeds 2x the configured interval. This accounts for normal jitter in cron/relay timing. Seeds with an explicit minRecordCount also report coverage_partial and stale: true until their coverage floor is met.

Domain Interval (min) Stale After (min)
Predictions, military flights 8 16
Market quotes, earthquakes, unrest 15 30
ETF flows, stablecoins, chokepoints 30 60
Service statuses, spending, wildfires 60 120
Shipping rates, satellites 90-120 180-240
GPS jamming, displacement 360 720
Iran events, UCDP 210-5040 420-10080

Example Request

curl -s https://api.worldmonitor.app/api/seed-health \
  -H "Origin: https://worldmonitor.app" | jq .

Integration with Monitoring Tools

UptimeRobot

Use /api/health?compact=1 as the public monitor URL. The HTTP status code only distinguishes a total Redis outage from everything else:

  • 503 = REDIS_DOWN (Redis unreachable — a true hard outage)
  • 200 = every other state, including DEGRADED and UNHEALTHY

So an HTTP-status-only monitor catches a full backend outage but not degraded/unhealthy data. For those, add a keyword monitor.

Point the keyword monitor at https://api.worldmonitor.app/api/health?compact=1 and alert when the compact token "status":"HEALTHY" (no space after the colon) is absent from the response body. Compact mode serializes with no indentation, so this exact token is stable regardless of formatting.

The bare /api/health URL is now an operator view and returns 401 without an API key. Public monitoring should always use ?compact=1.

Custom Alerting

Parse the JSON response to build granular alerts:

# Alert on any critical keys
STATUS=$(curl -s "https://api.worldmonitor.app/api/health?compact=1")
CRIT=$(echo "$STATUS" | jq '.summary.crit')
if [ "$CRIT" -gt 0 ]; then
  echo "CRITICAL: $CRIT data keys empty"
  echo "$STATUS" | jq '.problems'
fi

Differences Between Endpoints

Aspect /api/health /api/seed-health
Scope Data keys + seed metadata Seed metadata only
Auth None (public) API key or allowed origin
Data fetched Full Redis values (to count records) Only seed-meta:* keys
HTTP 503 Only REDIS_DOWN (DEGRADED/UNHEALTHY return 200) No (always 200 unless Redis down)
Best for Uptime monitoring, dashboard health Debugging seed loop issues
Response size Larger (~194 keys with record counts) Smaller (seed-meta domains only)