Several Packbase side effects continue after the request that accepted them. Before changing one, determine whether its state survives a process restart.
Current background workers
| Work | State and durability | Operational consequence |
|---|---|---|
| Howl creation and media processing | In-memory job queue; the client polls a status route. Completed status is short-lived and failed status is retained only briefly. | Process loss can lose active status or work. A failed late step may follow a successful post write. |
| Gossip propagation and related graph work | In-memory fire-and-forget queue with short-lived failure records. | Retry and restart are not durable by default. |
| External-service import | Progress, cursor, and status are stored in PostgreSQL and in-progress work resumes at startup. | Treat cursor correctness and repeatability as part of the data contract. |
| Imported-asset retry | Pending work, attempts, leases, errors, and completion are stored in PostgreSQL; a bounded worker claims due records. | Retries survive restarts, but permanent failure after the attempt limit still needs inspection or repair. |
| Expired pack-pin cleanup | In-process periodic loop. | Deletion must also invalidate affected feed caches. |
| Feed-cache cleanup | In-process periodic loop. | Cache age and invalidation can explain stale results without a database defect. |
| Last-online writes | Debounced during runtime and flushed during shutdown. | Abrupt termination may lose the latest presence update. |
Follow a job through every side effect
For howl creation, acceptance, asset movement, video conversion, post creation, gossip seeding, points or rating updates, pack activity, and cache work do not happen as one database transaction. If a job reports failure, inspect the job status, database post, object storage, and later bookkeeping before retrying.
Never assume “failed” means “rolled back.” A retry without inspection can create a duplicate post or repeat an external side effect.
Design a safe job
State these properties in code and tests:
- a stable job or object identity;
- which state is durable and which is process memory;
- when the caller receives acceptance or completion;
- whether each step is idempotent;
- the retry limit and backoff, if any;
- how partial completion is detected and repaired;
- how status expires;
- what happens on SIGTERM or SIGINT; and
- which cache or downstream view is invalidated.
Do not add a queue abstraction merely to call a function later. If the work matters after a restart, give it durable state or make the loss an explicit accepted limitation.
Preserve shutdown order
The server stops accepting new HTTP work and drains active requests. It stops the feed-cache cleanup timer, drains pack-pin cleanup, then drains howl jobs before gossip jobs because howl completion may enqueue gossip. It next drains imported-asset retries, import jobs, and last-online writes before disconnecting Prisma. Keep this dependency order when adding a worker.
The deploy platform must allow enough termination grace for that bounded drain. A process receiving a shorter hard kill can lose in-memory work even when the application shutdown code is correct.
Debug without leaking data
Use the job or howl identifier, exact time window, runtime revision, first error, and current state of each side effect. The server supports DEBUG namespaces, but logs may include account identifiers, source asset URLs, and cursors. Review and redact logs before moving them into Discord, Wildbase Work, or a pull request.
For an active failure, use the incident procedure.