note

Canonical source: docs/ops-runbook.md in the repo root. This page mirrors it for browsing here.

Ops runbook

One page. Who to page, how bad it is, and what to actually type. Every command below assumes you're in apps/api/ with wrangler logged into the account that owns tee-api, tee-db, and the tee-backups R2 bucket.

Severity levels

	Meaning	Examples	Response
SEV1	Every branch is down — no one can take an order	API 5xx across the board, D1 unreachable, JWT_SECRET rotated/lost	Page immediately, fix or roll back within minutes, don't wait for root cause
SEV2	One branch or one feature is down, others unaffected	One branch's KDS not receiving pushes, payment webhook failing for one PSP, a single integration erroring	Fix same business day, no need to wake anyone at 2am
SEV3	Degraded but everyone can still work	Slow report queries, a cosmetic UI bug, an analytics tag not firing	Normal backlog, fix when convenient

First 5 minutes of any incident

1. Check the obvious: npx wrangler tail on tee-api — is it even receiving requests? Is it 5xx-ing, or timing out, or just quiet?
2. Check Cloudflare status (cloudflarestatus.com) — if Cloudflare itself is degraded, nothing below will help; just wait it out.
3. Check what changed — git log --oneline -10 and npx wrangler deployments list (in apps/api). Most incidents are "we just deployed something." If the timing lines up, that's step 1, not step 5.

SEV1 playbook

Symptom: API is 5xx or unreachable everywhere.

• npx wrangler tail to see the actual error.
• If it's a bad deploy: npx wrangler deployments list, then npx wrangler rollback <previous-version-id>. This is instant — no rebuild, no redeploy.
• If it's D1 (Error: D1_ERROR or similar): check npx wrangler d1 info tee-db for status; D1 outages are rare and Cloudflare-side, there's nothing to do but wait and communicate.

Symptom: JWT_SECRET was rotated/lost and every login now fails.

• Every staff PIN login and every issued token breaks at once — this is the single most disruptive secret to lose. There is no recovery except setting it back: npx wrangler secret put JWT_SECRET with the original value. If the original value is truly gone, every currently-logged-in user must re-login, but that's it — PINs themselves are unaffected (they're hashed independently in D1, not derived from JWT_SECRET).

Symptom: Data looks wrong / a bad migration or bad write corrupted something.

• Don't panic-write more fixes into prod. First: npx wrangler r2 object get tee-backups/d1-backup/<date>.json for the most recent good nightly backup (or trigger one immediately via Super Admin → 💾 Backups → Run backup now, if the corruption is isolated and the rest of today's data is still worth keeping).
• Restoring means re-inserting specific rows from that JSON via wrangler d1 execute --command, table by table — there's no one-shot "restore" button by design, so a restore is deliberate and scoped instead of nuking today's legitimate orders along with the bad rows.

SEV2 playbook

One branch's KDS isn't updating in realtime.

• KDS pushes go through a Durable Object per branch (KdsRoom). Confirm the branch's branch_id is right in the KDS app's URL/config. A stuck DO is rare; if suspected, that branch's orders still work via POS/HQ polling — this degrades UX, it doesn't block anyone from selling.

A delivery webhook (GrabFood/Nham24) is failing.

• Check npx wrangler tail for POST /delivery/grab/webhook or .../nham24/webhook — likely causes: HMAC signature mismatch (partner rotated their secret — update it in Super Admin → 🧩 Integrations), or an unmapped SKU (422 SKU_NOT_MAPPED — add the mapping, the order is safely rejected rather than silently mispriced, not lost — the partner's system will retry or show a failure to re-place).

A Super Admin is locked out (forgot PIN, lost 2FA device).

• Forgot PIN: use the "PIN ភ្លេចទេ?" link on HQ's login screen (/auth/forgot-pin) — works for any account with a registered email, SA or not.
• Lost the TOTP device: there's no self-service TOTP recovery by design (that would be a backdoor). Fall back to the emailed OTP path instead — it's still accepted even with TOTP enabled if totp isn't sent in the verify call; only the client prioritizes TOTP when it's on. If email is also unreachable, disable TOTP directly in D1: UPDATE users SET totp_secret = NULL WHERE email = '<email>'.

SEV3 / general maintenance

Rotating a secret (BAKONG_WEBHOOK_SECRET, RESEND_API_KEY, SECRET_STORE_KEY, JWT_SECRET): npx wrangler secret put <NAME>, then npx wrangler deploy to make sure it's picked up. SECRET_STORE_KEY rotation invalidates every credential saved in the SA Integrations console (they were encrypted with the old key) — re-save each one after rotating it.

Checking backup health: Super Admin → 💾 Backups shows the last 30 days. If the list is stale (no entry for last night), the nightly cron silently failed — check npx wrangler tail around 00:00 ICT (17:00 UTC) the next time it runs, or trigger one manually to confirm the R2 binding and D1 access still work.

Deploying a fix: standard flow is npx tsc --noEmit clean, npx vitest run green, then npx wrangler deploy (API) and/or npm run build && npx wrangler pages deploy <app>/dist --project-name=<project> (frontends). See Deployment for the full per-app list.

Who to page

Fill this in for your team — there's no on-call rotation configured in this codebase, only the technical recovery steps above.