Incidents and maintenance

An incident is active harm or material loss of expected service, safety, privacy, security, or data integrity. Urgency changes the order of work, not the need for an accurate record.

Open the record

Create an urgent Wildbase Work task as soon as doing so does not delay immediate protection. Name an incident lead for the next decision and use one Discord thread for live coordination. Keep durable facts, decisions, commands, and outcomes on the task.

Start with:

Started or first observed:
Affected product and environment:
User-visible symptom:
Known scope:
Current revision or last change:
Immediate risk:
Incident lead:
Next check:

Do not declare a volunteer on-call merely because they are online. Ask for the specific access or knowledge needed, then continue with the people who accept the work.

Locate the failing boundary

Preserve the exact failing identifier and timestamp. Redact credentials and personal data; do not paste a complete environment or unreviewed import log.

Maintenance mode

The Packbase server exposes maintenance state through the MAINTENANCE runtime variable. /context returns a maintenance error and /server/describeServer reports the message; the web application turns that into its outage screen.

The implementation checks whether the variable is a non-empty string. Set it to the message people should see; disable it by unsetting it or making it empty. The literal value false is still non-empty and therefore still enables maintenance.

Maintenance mode is not a global pause or kill switch. It makes authentication resolve as anonymous, blocks /context and feed reads, and drives the web outage surface, but other public API routes and startup or background workers can continue. It does not drain traffic, stop queues, or make a data operation safe.

Changing a production runtime variable requires explicit environment access. Record the exact message, activation time, owner, verification, and removal condition. Maintenance can reduce exposure to a broken path; it is not evidence that recovery is complete.

The server’s /health route reports process liveness and basic process metrics. It does not check PostgreSQL or Clerk, and its connection count is currently a placeholder. Do not use it alone as readiness proof.

Mitigate

Reduce harm first: pause a broken path, narrow access, activate maintenance, use an authorised documented rollback, or prepare the smallest safe fix. Code still requires a signed commit, pull request, approval, and required CI; if no reviewer is available, use a non-code mitigation or wait rather than bypassing the default branch.

Before retrying a background job or migration, inspect partial effects. Before rolling back code, check whether data or configuration has already moved beyond the old version’s expectations.

For a suspected vulnerability, keep exploitable detail and secrets out of current Wild HQ channels. A redacted incident record may name the affected product and impact without publishing the exploit.

Recover and close

Recovery needs an affected behaviour check, not only a green process. Verify the user path, data invariant, background completion, and absence of the original error. Remove maintenance deliberately and verify again.

Then record:

• impact and duration;
• the technical and organisational conditions that allowed it;
• what detected it and what did not;
• mitigation and final repair;
• verification evidence;
• remaining risk; and
• concrete follow-up tasks with owners.

Good-faith mistakes belong to the team. Improve the guardrail, test, context, rollout, or runbook that would make the next response safer.

Wild HQ Housekeeping Mode is a separate governance process for foundational disagreement. Do not activate it merely to label a product incident.