Background jobs

The API runs background jobs with pg-boss on the same Postgres, in its own schema, pgboss. Today there is one kind of job: sending an email. This page covers:

1. How it works (enqueue, the worker, retries, retention).
2. Why a job holds no readable secret.
3. How to try it locally and in the tests.
4. What production needs.
5. Operations: inspecting failed jobs and re-queuing them.

Related docs:

• Authentication, Invitations: the flows whose mail goes through the queue.

---

1. How it works

The flow

POST /api/auth/register (or invite, resend invitation; forgot-password and resend-verification right after their 202)
   |
   +-> one Prisma transaction
         |- write the token / invitation (+ audit event)
         '- INSERT the mail.send job  (JobsService.enqueue(tx, ...), encrypted)
   |
   <-- 202 / 201 / 204  (never waits for SMTP)

MailWorker (in the API process, polls every 0.5 s)
   |- decrypt -> render (el/en) -> SMTP
   |- ok      -> job completed (row deleted 5 min later)
   '- failure -> job retried (5 retries, exponential backoff from 30 s)
                after the 6th failed attempt -> error log + copy in mail.send.failed (kept 7 days)

• No mail without a commit, no commit without a queued mail. The job is inserted through the caller’s Prisma transaction (pg-boss’s db option). If the transaction rolls back, for example on an audit failure, a 409 or a failed insert, neither the token nor the mail exists.
• Forgot-password and resend-verification answer first. They return 202 before looking the email up. The same token-plus-job transaction then runs detached (DetachedWork in api-auth), so their response time reveals nothing about the address. Shutdown drains the detached work that is already running (beforeApplicationShutdown) before pg-boss stops, but that drain runs before the HTTP server closes. A request that completes while the server is closing starts work nothing waits for, so its mail can be lost; the user can ask again. Every other mail is queued before its response.
• Responses don’t depend on SMTP. A mail server outage delays mail; it does not fail requests or lose mail.
• One queue, mail.send. The payload names its template: verification, password-reset or invitation. A job with a template this build does not know fails like any other failure: it is retried, then dead-lettered. It is never silently dropped.

The pieces

What	Where
Job runner: start/stop, enqueue, workers	libs/core/api/jobs/ (@core/api-jobs, JobsService)
Payload encryption	libs/core/api/security/src/lib/utils/payload-crypto.ts
Mail queue and worker	libs/core/api/mail/src/lib/{utils/mail-job,services/mail-queue,workers/mail.worker}.ts
Callers	EmailVerificationService, PasswordResetService, InvitationsService
E2E	apps/keelmint/api-e2e/src/api/jobs.spec.ts, support/database.ts

pg-boss is ESM-only, so it is imported in one place only (pg-boss.loader.ts, a dynamic import('pg-boss')). Every other file uses import type.

Lifecycle

• pg-boss starts when the application bootstraps, before the API listens. On first start it creates and migrates the pgboss schema itself, under an advisory lock, so several instances can start together.
• On SIGTERM/SIGINT (app.enableShutdownHooks()), pg-boss stops in onApplicationShutdown. Nest 11 closes in this order: onModuleDestroy (Prisma disconnects), beforeApplicationShutdown (the detached-work drain), the HTTP server close, then onApplicationShutdown. So a forgot-password or resend-verification request that completes during the HTTP close starts detached work that is not drained and can race pg-boss stopping; its enqueue then fails, is logged, and the mail is lost (the user can ask again). pg-boss waits up to 10 s for a mail being sent, then closes its connection pool.
• Each instance runs one mail worker, which takes one job at a time in FIFO order, as long as no job is retried. During an SMTP outage a retried mail waits for its backoff, so one with a superseded link can arrive after a newer, valid one. The older link then shows “expired or already used”: use the newest mail. pg-boss sleeps the polling interval (0.5 s) after every fetch, so one instance sends about 2 mails per second. For more, raise localConcurrency in JobsService (mails to one address may then arrive out of order) or run more instances.

Retries and what you see in the logs

The queue is created with retryLimit 5, retryBackoff true and retryDelay 30 s, so a mail is tried at most 6 times over roughly half an hour. The settings are re-applied on every boot.

Attempt fails	Log	Job
attempts 1-5	warn Mail job <id> (<template>) failed on attempt <n>; retrying	retry, runs again after backoff
attempt 6 (the last)	error Mail job <id> (<template>) failed on its last attempt + reason	failed, copied to mail.send.failed

The reason is only the transport code and SMTP code, e.g. Sending verification mail failed (EENVELOPE, SMTP 451). Logs never contain the address. Alert on the error line (see section 4).

Errors that pg-boss itself emits (connection loss, maintenance) are logged as Job runner error. They pass through describeJobError, which keeps only the message, stack and queue name, because a database error can carry row values.

Other error lines worth knowing:

• Mail job <id> has an unknown template: the worker cannot render the payload (for example one queued by a newer deploy). The job fails and dead-letters after its retries.
• Job <id> on mail.send has an unreadable payload: wrong or rotated key, or a tampered row (see section 2).

Retention

pg-boss deletes its own rows, so there is no cleanup job:

• mail.send rows, completed or failed, are deleted 300 s (5 minutes) after they finish.
• A mail that failed for good is copied, still encrypted, with its last error, to mail.send.failed. Nothing works that queue, and its rows are deleted after 7 days.

2. Secrets in job rows

The links in these mails carry raw tokens, and the rule everywhere else is that only a token’s sha256 is stored. So every job payload is encrypted before it is inserted:

• AES-256-GCM, a random 12-byte IV and a 16-byte tag. Stored as { "v": 1, "iv", "tag", "data" }, all base64.
• The key is JOBS_ENCRYPTION_KEY: 32 random bytes in canonical base64 (44 characters, ending in =). Generate it with openssl rand -base64 32. The API refuses to boot without a valid one.
• The queue name is bound into the ciphertext as additional authenticated data (AAD), so a payload moved to another queue does not decrypt. To read a row in mail.send.failed, decrypt it with its original queue name (mail.send).
• A failed attempt’s stored error contains no address. The worker replaces the transport error with one that holds only the codes.

Changing the key makes jobs that are already queued unreadable. They fail with Job <id> on mail.send has an unreadable payload and end in mail.send.failed. Rotate when the queue is empty, and resend what failed.

3. Local setup and tests

• docker compose up -d db mailpit. Add JOBS_ENCRYPTION_KEY to .env (copy it from .env.example, or generate one with openssl rand -base64 32).
• pnpm nx serve keelmint-api, register, and watch the mail arrive in Mailpit (http://localhost:8025) within about a second.
• Look at the queue: docker exec -it keelmint-db psql -U postgres -d keelmint -c "select name, state, retry_count, created_on from pgboss.job order by created_on desc limit 10".
• Simulate an SMTP outage: Mailpit runs with chaos triggers enabled (MP_ENABLE_CHAOS in docker-compose.yml, and in CI). curl -X PUT localhost:8025/api/v1/chaos -d '{"Recipient":{"ErrorCode":451,"Probability":100}}' makes it refuse every recipient; -d '{}' turns that off.
• E2E: PORT=3001 pnpm nx e2e keelmint-api-e2e. jobs.spec.ts checks that a refused mail is retried and delivered, and that a job row holds neither the address nor the token. truncateAll clears pgboss.job between tests.

4. Production

• Build and deploy: the build inlines pg-boss into a lazy chunk, dist/apps/keelmint/api/dist-*.cjs, which main.cjs loads with require when the runner starts. Only pg stays external (it is the shared driver, and the pruned package.json lists it). Deploy the whole dist/apps/keelmint/api directory, not just main.cjs: without the chunk the API fails at boot.
• Database rights: on first start pg-boss runs CREATE SCHEMA pgboss and its migrations. Either the app’s DB user may create schemas, or a DBA creates pgboss owned by that user beforehand. Prisma manages only public and ignores pgboss.
• Connections: pg-boss has its own pool (up to 10 connections) next to Prisma’s. Count both against your Postgres max_connections, per instance.
• Startup: the API needs Postgres to boot (pg-boss starts before it listens). If the database is down, the process exits; let your process manager restart it.
• Key: set JOBS_ENCRYPTION_KEY from your secret store. Keep it stable across deploys and identical on every instance.
• Second key: TWO_FACTOR_ENCRYPTION_KEY encrypts TOTP secrets, not jobs. Never reuse one key for both (Two-factor authentication, “Key”).
• Storage: helpdesk images are stored before the job-carrying transaction (Helpdesk). Set STORAGE_DRIVER/STORAGE_DIR (or the S3_* keys) on every instance; with more than one instance, use s3.
• Monitoring: alert on Mail job ... failed on its last attempt and Job runner error. A growing mail.send.failed means mail is not reaching people.
• Shutdown: give the container about 20 s between SIGTERM and SIGKILL (docker stop -t 20, stop_grace_period: 20s, or terminationGracePeriodSeconds: 20). The detached-work drain and the HTTP close come first, then pg-boss waits up to 10 s for a mail mid-send; Docker’s default of 10 s is too short. If it is killed anyway, the job is retried after its expiry (15 min), so a mail may rarely be sent twice.

5. Operations

Inspect failed jobs

Dead-lettered mail sits in mail.send.failed for 7 days. The payload is encrypted, so SQL shows state, timing and the sanitized error, not the address:

select id, state, created_on, completed_on,
       source_retry_count, source_output
from pgboss.job
where name = 'mail.send.failed'
order by created_on desc;

-- how many mails are waiting, retrying or failed right now
select name, state, count(*) from pgboss.job group by name, state;

In a dead-letter row, output and retry_count are not filled in: the original job’s error is in source_output and its attempt count in source_retry_count. For example source_output holds Sending verification mail failed (ECONNECTION, SMTP n/a).

Re-queue

Do not copy a failed row back into mail.send by hand: the payload is bound to its queue name by the encryption, and a job inserted outside the application skips the transaction guarantee. Use the application flows instead:

• Verification and reset mail: the user presses “resend” or “forgot password” again (a new token, a new job).
• Invitation: the platform admin presses Resend on the pending invitation.

If many jobs failed because of an outage that is now over, find the affected users from the mail.send.failed rows’ created_on window and the audit log, then trigger the same flows. To discard failed rows early, delete them from pgboss.job where name = 'mail.send.failed'. They are pg-boss bookkeeping, not application data, and pg-boss prunes them itself after 7 days.