Monitoring Guide

Practical recipes for setting up monitors that catch the right kinds of failures. Skip to the section that matches your endpoint.

The mental model

Every monitor is a scheduled HTTP request + a list of assertions. The probe is "up" only when every assertion passes. The monitor is "down" globally only when ≥ quorum regions report "up" failure within the last failureThreshold consecutive checks. (Tunable per monitor.)

When a monitor flips down, the IncidentCoordinatorDO opens an incident and dispatches to every attached alert channel. When it flips back up, you get a recovery notification. Incidents dedupe — you won't get spammed.

Recipe: REST API health endpoint

You have GET https://api.example.com/health returning { "ok": true, "version": "..." }.

Field	Value
URL	`https://api.example.com/health`
Method	`GET`
Interval	60s
Timeout	5000ms
Regions	All 5 (default)
Quorum	2
Failure threshold	2
Assertions	see below

status_code            eq        200
response_time_ms       lt        2000
header[content-type]   contains  application/json
json_path: $.ok        eq        true

The first three guard against the obvious failure modes (HTTP error, slowness, content-type drift). The json_path assertion turns "the server returned 200 OK with junk" into a real failure.

Recipe: Public JSON API (third-party check)

You depend on https://api.example.com/v1/products/123 and want to know if it goes down or starts returning the wrong shape.

status_code             eq        200
response_time_ms        lt        5000
json_path: $.id         eq        123
json_path: $.name       exists
json_path: $.price      gt        0

The exists operator is your friend for fields you require but don't know the exact value of. The gt 0 on price catches the "API returned an object but all fields are null" failure mode.

Recipe: Static site / landing page

You have https://example.com and want to know it serves your real HTML, not a CDN error page.

status_code             eq        200
response_time_ms        lt        3000
header[content-type]   contains   text/html
body_text               contains  Acme Corporation

The body_text contains of a known phrase from your hero copy is the cheapest way to detect "site got replaced with a 200-OK error page" or "deploy half-shipped a partially rendered template."

Recipe: Authenticated endpoint

You want to monitor an endpoint that requires a token. Don't put production secrets in monitor headers — create a dedicated read-only monitoring API token at your service.

URL: https://api.example.com/v1/me
Method: GET
Headers:
  Authorization: Bearer <monitoring-token>
  X-Probed-By: thump

Assertions:
  status_code           eq        200
  response_time_ms      lt        2000
  json_path: $.id       exists

If your API has rate limits, set the interval to ≥ 5 min to stay polite.

Recipe: GraphQL endpoint

GraphQL is POST to a single URL with a query in the body.

URL: https://api.example.com/graphql
Method: POST
Headers:
  Content-Type: application/json
Body:
  {"query":"{ health { status } }"}

Assertions:
  status_code                          eq        200
  json_path: $.data.health.status      eq        ok
  json_path: $.errors                  not_exists

The not_exists on $.errors catches GraphQL's "200 OK with errors in the body" pattern. Critical — GraphQL almost never returns non-200 for query failures.

Recipe: Cron job / background worker (heartbeats)

You have a backend job — a nightly backup, a queue worker, a cron — and you want to know when it stops running. A monitor can't help here: there's no URL to probe. Use a heartbeat (dead-man's switch) instead. Your job pings thump; silence opens an incident.

Create one under Heartbeats → New, then have the job ping its URL on success:

# at the end of your job
curl -fsS -m 10 -X POST https://thump.dev/ingest/heartbeats/<token>

Field	Meaning
Expected interval	How often the job should ping (e.g. 300s for a 5-minute cron)
Tolerance	Grace period past the interval before we alert — set it to cover a slow run, not a missed one

If no ping arrives within interval + tolerance, thump opens an incident and dispatches to the heartbeat's attached alert channels, exactly like a down monitor. The next ping resolves it.

Signed pings. Set a signing secret on the heartbeat and thump verifies an X-Hub-Signature-256 HMAC over the raw body on every ping — so a leaked URL alone can't be used to fake liveness. Signing is opt-in per heartbeat and works uniformly across the token, GitHub, and Cloudflare ingest paths.

Behind a firewall? The thump agent runs checks inside your network and reports out over the same heartbeat ingest — outbound-only, no inbound holes. See the private-network guide.

Recipe: the domain and certificate behind everything else

Two expiry dates can take your whole site down, and neither shows up in an HTTP check until it is already too late.

A TLS certificate expires and browsers refuse to connect. Create a monitor of type SSL certificate expiry, give it the hostname, and set Warn (days) to a window you can actually act in — 14 is a reasonable default, 30 if renewals need a human.

A domain registration expires and the outage is total: DNS stops resolving, so your HTTP check reports a DNS error indistinguishable from a hundred other causes, your TLS check never gets far enough to complain, and the certificate you were carefully monitoring is still perfectly valid. Nothing in your monitoring says "the domain lapsed". Create a monitor of type Domain registration expiry, paste the domain (a full URL or a www. prefix gets trimmed for you), and set a window that covers your renewal process — 30 days is the default because most registrars need a working card on file well before the redemption period starts.

Domain checks ask the registry directly over RDAP, so they also go down when the registry reports a hold or redemption status, whatever the expiry date says — a domain on client hold is already not resolving.

Field	Meaning
Domain	The registrable domain (`example.com`, `example.co.uk`)
Warn (days)	Go down when the registration expires within this many days

Check these daily, not every five minutes. Registration and certificate expiry move on the scale of months, and registries rate-limit RDAP. Picking Domain registration expiry sets the cadence to 24 hours for you.

Not every TLD publishes RDAP. Most do; a handful of country-code registries still don't. When one doesn't, the check says so by name rather than reporting a false expiry.

Recipe: one page for one outage

When a shared dependency fails — the database, the auth service, the CDN — every monitor behind it goes down in the same minute, and a monitoring tool cheerfully sends forty pages for one incident. They are all true and all useless: whoever is on call now has to work out which of the forty is the cause, at 3am, from a phone.

Set Depends on on a monitor to the thing it sits behind. When that dependency is down, this monitor still records an incident — history and the status page stay accurate — but it doesn't page, and its incident records what it's downstream of.

  Postgres          ← pages
    └─ API          ← suppressed, "caused by Postgres"
        ├─ Checkout ← suppressed, "caused by API"
        └─ Search   ← suppressed, "caused by API"

Dependencies chain, so a database outage behind an API behind six pages is still one alert. Each incident attributes to its own dependency, which reads as a chain back to the cause.

If the dependency recovers and the dependent is still down, it pages then. "The database came back but the API is still wedged" is exactly the thing you want to hear about, and it's the case a blanket mute would swallow.

Cycles are refused when you set them, not silently ignored later — and a monitor can't depend on itself.

Pricing-style assertions you might want

response_time_ms < 1000 — your SLA target. If your p95 is 800ms and you alert at 1000, you'll be the first to know when the curve shifts.
status_code lt 400 — accepts 2xx and 3xx. Useful for endpoints that legitimately return 301/302.
body_size_bytes gt 100 — catches "API returned an empty payload" without locking the exact size.
header[cf-cache-status] eq HIT — confirms your CDN is doing its job. Combine with a region filter to alert if a specific edge starts missing.

Picking interval + quorum

Start with 60s interval + 5 regions + quorum 2. That gives you:

A check from 5 places every minute
Incidents only when 2+ regions agree something's wrong (filters out single-region flakes)
Roughly 5 region-checks/min/monitor — well within thump's free tier

Tune down to 30s + quorum 3 for production-critical endpoints. Tune up to 5min + quorum 1 for non-critical or rate-limited targets.

Setting up alerts

Once your monitor is created, add an alert channel under /app/channels:

Discord — paste your channel webhook URL (https://discord.com/api/webhooks/...)
Slack — paste an incoming webhook URL
Webhook — your own endpoint that receives the canonical incident payload (see apps/api/src/notifications/types.ts)
Email — your inbox (requires the operator to configure Resend; not available on hosted thump until that ships)

Then go back to the monitor and attach the channel under "Alert channels" — or attach the channel to N monitors at once from the channel detail page.

Testing before saving

The monitor-create form has a "Test now" button that runs a one-off probe with your current draft (URL + assertions) and shows pass/fail per assertion + the captured actual value. Use it to tune your json_path expressions before saving — they're notoriously easy to typo.

Public playground

Anyone can probe a URL at https://thump.dev — paste it into the "Try it right now" widget on the landing page. Rate-limited to 20 probes / 5 min per IP. No signup. Useful for one-off "is this URL up?" checks.