Forge Operator Guide

Reference for forge operators (Peter + future sysadmin). Covers infrastructure internals that teammates do not need to know.

Compose Stack Ownership

See deploy/README.md for the full authoritative stack list and layering.

Quick summary: - core.yml — always running: postgres, redis, redpanda, openbao - ritual.yml — always running: ritual-server, ritual-consumer, ceremony-warden - platform.yml — always running: keycloak, forgejo, matrix - observability.yml — always running: grafana, prometheus, dozzle - apps.yml — always running: forge-bridge, slack-bridge, audit-events, lore-drift, git-poller

Legacy stacks (docker-compose.yml, docker-compose.prod-full.yml) are deprecated — do not use for new deployments.

Drift Check

bin/forge-deploy-check           # report drift between tracked files and live host
bin/forge-deploy-check --sync    # apply tracked files to host

Kafka Broker Policy

Context	Broker	Protocol
External/TLS clients (daemons, scripts, hooks)	`spel.toilville.dev:19092`	`SASL_SSL`
Internal Docker containers (on `forge-network`)	`redpanda:9092`	plaintext

Never use 10.10.0.1:19092 for TLS clients. The Redpanda TLS cert is issued for spel.toilville.dev. IP addresses break hostname validation.

The host firewall must also allow inbound 19092/tcp on toilville-forge. That rule is persisted via netfilter-persistent and is required for the public Kafka listener to survive reboots.

SASL credentials: secret/forge/kafka-scram in OpenBao.

Validation:

bin/forge-kafka-check   # fail if any TLS client uses an IP broker

Vault / OpenBao

Internal URL (Docker gateway): http://172.22.0.1:8200
WireGuard URL (host access): http://10.10.0.1:8200
Public URL: https://vault.toilville.dev (via nginx proxy → 10.10.0.1)
Systemd service: openbao on toilville-forge

Secret Paths

The canonical mapping of service → vault path is in forge-manifest.spel under infrastructure.secret_paths. Key paths:

Service	Vault Path
kafka SCRAM credentials	`secret/forge/kafka-scram`
ritual-server	`secret/forge/ritual-server`
audit-events	`secret/forge/audit-events`
lore-guards	`secret/forge/lore-guards`
forge-bridge	`secret/forge/forge-bridge`
git-poller	`secret/forge/git-poller`
Woodpecker CI	`secret/forge/woodpecker`

Vault-Agent Sidecars

Each service has a sidecar in deploy/vault-agent/<service>/: - agent.hcl — AppRole auth + template sink - secrets.env.tpl — renders secrets to /vault-secrets/secrets.env

Vault address in sidecars is http://172.22.0.1:8200 (Docker gateway IP).

Hook System

Hook Lifecycle

Claude hooks are registered in ~/.claude/settings.json: - PreToolUse: bin/claude_pre_tool — Bastia grant gate + audit event - UserPromptSubmit: generated/hooks/claude_pre_prompt.sh — context injection - SessionStart: generated/hooks/claude_session_init.sh — env injection + permission sync

Regenerating Hooks

The generated/hooks/ files are generated from bin/templates/hook_wrapper.sh.tmpl. Do not edit generated files by hand — edit the template and regenerate:

bin/forge-regen-hooks
git add generated/hooks/
git commit -m "regen: hooks from canonical template"

Hook Behavior

All hooks fail open (exit 0) by default. The only blocking behavior is the Bastia grant gate in claude_pre_tool for Z2/Z3 operations (vault writes, psql mutations, remote writes, paladin frame ops, ssh/scp to production).

Local-only mode (set at session init via RITUAL_LOCAL_FIRST=1): - Pre-prompt hook skips Kafka/server and serves from cr-sqlite cache - Cache freshness is emitted to stderr: [forge-local] cache age: Xs, tier: N

Bastia Grant Scopes

Managed in core.bastia_grant_scopes and core.bastia_grant_policies.

Scope	Policy	Auto-Approve	Max TTL
`vault:read`	bastia-grant-vault-read	Yes	60s
`vault:write`	bastia-grant-vault-write	No (SLA)	60s
`db:query`	bastia-grant-db-query	Yes	120s
`paladin:exec`	bastia-grant-paladin-exec	No (SLA)	120s
`remote:write`	bastia-grant-remote-write	No (SLA)	60s
`service:restart`	bastia-grant-service-restart	No (SLA)	60s
`secret:patch`	bastia-grant-secret-patch	No (SLA)	60s
`forgejo:credential-rotate`	bastia-grant-forgejo-credential-rotate	Yes	120s
`kafka:delegated`	bastia-grant-kafka-delegated	—	900s

Narrow helpers in bin/forge-privileged-op wrap the common operations. Do not add raw bao, ssh, or scp to Claude’s allowlist — use bounded helpers instead.

Privilege Operations

bin/forge-privileged-op vault-read <path>
bin/forge-privileged-op vault-write <path> <json>
bin/forge-privileged-op remote-copy <local> <host:path>
bin/forge-privileged-op remote-restart <service>
bin/forge-privileged-op service-restart <service>
bin/forge-privileged-op secret-patch <path> <field> <value>
bin/forge-privileged-op db-exec <sql-file>
bin/forge-privileged-op rotate-forge-ci-password

Each operation: requests grant → executes → revokes. Fail-closed if Bastia denies.

Local Cache

cr-sqlite cache lives at ~/.ritual/schema.db. Freshness tiers:

Tier	Condition	Behavior
0	< 5 min old	Fresh — Kafka not needed for reads
1	5 min – 1 hr	Stale — reads may lag
2	> 1 hr	Very stale — consider sync
3	Missing	No local cache — server required

bin/forge-local-cache-check          # human-readable report
bin/forge-local-cache-check --json   # machine-readable

System Health Check

bin/ritual-system-doctor   # run all checks
bin/forge-kafka-check      # Kafka policy validation
bin/forge-deploy-check     # deploy drift check
bin/forge-local-cache-check # cache freshness

Working with Toilville — Teammate Guide

You don’t need to learn Spelwork, ritual, Kafka, or Docker to contribute. This guide explains the human workflows.

How to submit a request or idea

Send an email to the Toilville forge inbox. Write your request in plain English in the subject line. The forge reads it, creates a work item, and routes it for review.

Examples of things you can send: - Review the latest design mockup - Can you approve the Q2 budget estimate? - Update the sprint goal for this week - Help — the integration is broken again

You’ll receive a reply if clarification is needed. Decisions and outcomes are recorded automatically.

How to contribute code or documentation

Open a pull request or issue in the Forgejo repository at git.toilville.dev. The forge monitors all repos in the toilville organization and creates work items automatically when you:

Open a new issue
Submit a pull request
Push a commit to a tracked branch
Leave a review comment

You don’t need to mention the forge or use any special syntax. Just work normally in git.

How decisions get made

When a request requires a group decision, you may receive a decision parlay — a structured prompt asking you to vote, object, or add context.

You can respond by email reply or by commenting in the relevant git thread.
Votes, objections, and clarifications are recorded in the lineage trail.
A decision is recorded once the threshold is met (unanimous, majority, or steward call).
You can review past decisions by asking: What did we decide about X?

How to request a service operation

If you need something restarted, redeployed, or updated on the forge infrastructure, send an email or open an issue describing what you need and why. The request is routed for approval before anything runs.

Do not ask for raw access to production servers. Bounded operation requests are the correct path.

What you’ll never need to know

These are implementation details — you do not need to understand them to contribute:

Kafka topics or broker addresses
OpenBao / Vault secret paths
Docker compose files
Claude hook internals
Bastia grant scopes

If you find yourself needing these, ask an operator — it likely means there’s a missing bounded helper or form that should exist.

Getting help

Send an email with your question. Or open a Forgejo issue in toilville/spelwork tagged help.

SPELWork API & Architecture Docs