fix(ci): the site check tells a Cloudflare edge block from an outage #4
No reviewers
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Toilville/toilville-static-sites!4
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/health-check-edge-block"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
site-health-checkhas failed every scheduled run for a week, reporting:toilville.dev is not down. The check is reporting a measurement it did not make.
Scope
One step: how
check()classifies a non-200..forgejo/workflows/site-health-check.yml, +88/−5. No new tooling, no new dependency, samealpine/curlcontainer, same four domains, same schedule.Measured 2026-10-17 from the forge host, every request identical but for the address family:
403 Your request was blocked.— every user agent tried200— every user agent tried200The 403 is Cloudflare's configured block response for the zone: 403,
server: cloudflare, a 25-bytetext/plainbody, origin never reached. The toilville.dev zone runs Super Bot Fight Mode and allow-lists machine-reachable hosts one at a time in forge-opsterraform/bot_allowlist.tf; the apex is not among them, and this runner's IPv4 egress is a Hetzner datacenter address that SBFM scores accordingly. The job runs in a container on the docker bridge, which has no global IPv6 address at all, so it cannot route around it either.Three changes, all inside
check():BLKD, listed under "Not measured", reported every run, and not counted as a failure. The detector requires all three of403+server: cloudflare+ that exact body, so an origin 403 proxied through Cloudflare still fails: it carries the origin's own body and headers.truechiptilldeath.com's intermittent "expected 200 got 302" was.-4. The container has no IPv6, so the address family was never actually a choice; stating it makes the comment above true rather than lucky.This is not the repair, and the commit says so. The repair is in Cloudflare: add the forge egress IPs to a skip rule covering the apex, as
api.toilville.devalready has (api_schema_bot_allow). That zone is at the 5-rulehttp_request_firewall_customcap, so it means extending an existing rule's expression rather than adding a sixth — a production WAF change, deliberately not bundled into a workflow PR. Once it lands, theBLKDbranch simply stops firing and nothing here needs reverting.Out of scope: the Cloudflare change above; the staging-vs-live question in this workflow's own header comment; the missing Mailgun alerting.
Testing
Verified against a deterministic local origin, not just the happy path — a throwaway HTTP server returning each shape on demand, reached from the same
alpine/curlimage the job uses:200503403+Server: Apache403+server: cloudflare+ the exact body302 -> 503302 -> 302The third row is the one that matters: the new branch must not become a way for a real 403 to pass. It does not.
Plus the four live domains, from a laptop whose egress Cloudflare blocks on different zones than the forge's:
That inversion is the point — same four domains, same user agent, different source address, opposite results. It reproduces the SBFM-scores-by-source behaviour from the other side, and it exercised the
BLKDpath end to end.Also verified:
sh -non the extracted step, and the redirect branch against a real hop (http://toilville.dev→301 -> 200 via https://toilville.dev/).What this does not prove. The
BLKDdetector is a fingerprint of the zone's current custom block response. If Cloudflare's block body or headers change, a blocked request falls through toFAIL— which is the safe direction (a false alarm, not a missed outage), but it will need the fingerprint updated.Impact Statement
build-deploy.ymlis untouched.curl, still no Mailgun.000, and both land in theFAILbranch. The only thing that stops failing is a 403 that Cloudflare itself generated before reaching the origin — which was never evidence about the origin.What it buys. A daily check that stopped meaning anything a week ago starts meaning something again. A false alarm every morning is worse than no check: it trains the reader to ignore the lane that is supposed to catch the real outage.
Docs
No public contract changed. The measurement, the cause and the actual repair are recorded in a comment on the branch that implements the behaviour, so the next reader does not have to re-run the IPv4/IPv6 comparison to find out why a 403 is tolerated here.
site-health-check has failed every scheduled run for a week, reporting FAIL toilville.dev — expected 200 got 403 toilville.dev is not down. Measured 2026-10-17 from the forge host, every request identical but for the address family: IPv4 -> 403 Your request was blocked. (every user agent tried) IPv6 -> 200 (every user agent tried) and from a residential address, IPv4 -> 200. The 403 is Cloudflare's configured block response for the zone: 403, server: cloudflare, a 25-byte text/plain body, origin never reached. The toilville.dev zone runs Super Bot Fight Mode and allow-lists machine-reachable hosts one at a time in forge-ops terraform/bot_allowlist.tf; the apex is not among them, and this runner's IPv4 egress is a Hetzner datacenter address that SBFM scores accordingly. The job runs in a container on the docker bridge, which has no global IPv6 address, so it cannot route around it either. So the check was reporting a measurement it had not made. A daily false alarm is worse than no check: it trains the reader to ignore the lane that is supposed to catch a real outage. Three changes, all in the check(): - An edge block is its own outcome. Reported every run, named BLKD, listed under "Not measured", and NOT counted as a failure. The detector requires all three of 403 + server: cloudflare + that exact body, so an origin 403 proxied through Cloudflare still fails -- it carries the origin's own body and headers. - A redirect is followed and the destination judged. An apex -> www or http -> https hop landing on 200 is the site working. The hop is printed so a new redirect stays visible. This is also what truechiptilldeath.com's intermittent "expected 200 got 302" was. - curl is pinned to -4. The container has no IPv6 at all, so the address family was never really a choice; saying so makes the comment above true rather than lucky. A real outage still fails: Cloudflare answers a down origin with 52x, and a DNS or TLS failure gives 000. Verified against a deterministic local origin, not just the happy path: healthy 200 -> OK origin 5xx 503 -> FAIL origin 403 403 + Server: Apache -> FAIL (not swallowed) edge-403 shape 403 + cloudflare -> BLKD redirect to 5xx 302 -> 503 -> FAIL redirect loop 302 -> 302 -> FAIL plus the live four, where the laptop's own egress is blocked on two different zones than the forge's -- which is the SBFM-scores-by-source behaviour this commit is about, reproduced from the other side. THIS IS NOT THE REPAIR, it is the check telling the truth about what it can see. The repair is in Cloudflare: add the forge egress IPs to a skip rule covering the apex, as api.toilville.dev already has (api_schema_bot_allow). That zone is at the 5-rule http_request_firewall_custom cap, so it means extending an existing rule's expression rather than adding a sixth. Once that lands, the BLKD branch simply stops firing.