Operations

Running Wayscribe for a team. It is deliberately small (one API, one web application, one PostgreSQL database), so most of this is short.

1. What holds state

Everything is in PostgreSQL. There is no second datastore, no object storage, and no external service (ADR-012). Back up the database and you have backed up Wayscribe.

The demo profile adds ElasticMQ, which holds nothing worth keeping.

Bring your own database

DATABASE_URL is the whole coupling. Point it at the PostgreSQL your team already runs, the one somebody backs up, monitors, and can restore, and Wayscribe needs nothing else from you.

export COMPOSE_FILE=compose.published.yaml
export WAYSCRIBE_VERSION=vX.Y.Z   # the release to run; releases are 0.x
export DATABASE_URL=postgresql://user:password@db.internal:5432/wayscribe
docker compose up -d

COMPOSE_FILE names the files every docker compose command in the shell reads. The commands for the published images in this document are written without -f and rely on it, so they always see the stack you started. WAYSCRIBE_VERSION names the release, as its git tag, and is required: compose.published.yaml has no latest fallback, because its migrate service applies the schema of whatever image it pulls, and an unpinned pull could move the database across a minor release, which before 1.0 may change the API. Export both again in a new shell.

It needs PostgreSQL 15 or later, an ordinary database, and an ordinary role: USAGE, CREATE, SELECT, INSERT, UPDATE, DELETE on its own schema. It does not need CREATE on the database: it installs no extensions and touches nothing outside the tables its migrations create, so an existing database with other tables in it is fine.

CI runs the whole integration suite, migrations included, on PostgreSQL 15, 17 and 18. 16 is not run; it lies between two releases that are, and doctor passes it. The README’s Supported versions table lists the Node, Compose and platform versions as well.

The bundled overlay runs PostgreSQL in a container instead and sets DATABASE_URL for you:

export COMPOSE_FILE=compose.published.yaml:compose.bundled.yaml
docker compose up -d

A command that leaves the overlay out reports the PostgreSQL container as an orphan and suggests --remove-orphans; do not take that advice, because it removes the database container.

The overlay is for evaluation and for local work. Nothing about it is unsuitable for production except that it is invisible to whoever is responsible for your data: no backup schedule, no monitoring, and a docker compose down -v away from gone.

Ports, URLs, and a second stack

API_PORT and WEB_PORT move the published ports, and APP_URL and API_URL say where the stack is reached from outside. The API builds links with the last two, so set them alongside the ports rather than only moving the ports:

export API_PORT=8085 WEB_PORT=3005
export API_URL=http://localhost:8085 APP_URL=http://localhost:3005
docker compose -p wayscribe-staging up -d

Both default to http://localhost:8080 and http://localhost:3000, which is what a single stack on the documented ports wants. Behind a reverse proxy they are the names a browser uses, not the container’s own.

What up --wait waits for

docker compose up -d --wait returns when every container is healthy. Both images declare a health check: the API answers GET /health on its own port, and the web app answers GET /health on its port, which loads its configuration and answers 503 when that fails. A passing probe therefore means the server answered a request, not that a process started, and for the web app that it is configured.

Both servers also refuse to start on a configuration they cannot load, such as ADMIN_TOKEN and ADMIN_TOKEN_FILE given together or a token file that is not there. The process writes the reason to its log, naming the setting and never its value, and exits 1, so up --wait fails at once with container <name> exited (1) rather than reporting it Healthy. Earlier images of the web app started anyway, passed their health check on /login, which reads no configuration, and failed the first sign-in with a 500.

Each check is probed every two seconds during its start period, so --wait returns as soon as the stack is actually serving. Afterwards the cadence depends on which health check is in force: the images declare thirty seconds, which is what compose.published.yaml runs on, while compose.yaml, the source stack, overrides the API’s with its own ten second one. Either way the start period is the same and --wait behaves identically. On the bundled overlay, a cold start with no volume and no containers, measured on a laptop, takes about nine seconds end to end: PostgreSQL initialising, migrate applying the schema, then both servers answering.

An older release’s images have no health check on web and none of these timings, so a stack running one reports web as Healthy on nothing more than its process having started.

Schema changes

The migrate service applies migrations on boot, against your database, and it is the only thing that writes schema. If your team applies migrations through its own process, leave that service out and run the same command when you choose to:

docker compose run --rm --entrypoint node api \
  packages/database/dist/cli.js migrate

The API reports /ready 503 migrations_pending until they are applied, so it will not serve reads against a schema it does not recognise.

Building the images yourself

The published images say what they are: the API in /ready, the web app on the line under every signed-in page. Both read it from two build arguments, WAYSCRIBE_BUILD_VERSION and WAYSCRIBE_BUILD_COMMIT, which apps/api/Dockerfile and apps/web/Dockerfile both declare and the release passes to both. An image you build yourself, from a checkout or a git archive, needs the same two on both builds:

VERSION=local-$(git rev-parse --short HEAD)
COMMIT=$(git rev-parse HEAD)
docker build --file apps/api/Dockerfile \
  --build-arg WAYSCRIBE_BUILD_VERSION="$VERSION" \
  --build-arg WAYSCRIBE_BUILD_COMMIT="$COMMIT" \
  --tag wayscribe-api:"$VERSION" .
docker build --file apps/web/Dockerfile \
  --build-arg WAYSCRIBE_BUILD_VERSION="$VERSION" \
  --build-arg WAYSCRIBE_BUILD_COMMIT="$COMMIT" \
  --tag wayscribe-web:"$VERSION" .

An image built without them reports 0.0.0, “not a release build”, whatever commit it came from. With one image built each way, the version line cannot compare them and says so, rather than calling them different builds (F-051). infrastructure/compose.yaml builds both without them, so the source stack says “not a release build” on both halves.

2. Backup

From a source checkout

The backup helpers run from a source checkout, not from the published images.

1. Check out the release the installation runs, so its migrations match the database, and install dependencies with pnpm install --frozen-lockfile.
2. Install PostgreSQL 18 or newer client tools (pg_dump and pg_restore) on the machine running the CLI and put them on PATH. They are not included in the API serving image. pg_dump --version should print 18 or newer.
3. Set DATABASE_URL in the environment or the checkout’s .env. For backup:verify, also set ENCRYPTION_KEY, and ENCRYPTION_KEY_PREVIOUS during a rotation, to the values the API runs with.
4. Take the backup, verify it, and restore it when needed:

pnpm run backup:create --output wayscribe.dump
pnpm run backup:verify --input wayscribe.dump
pnpm run backup:restore --input wayscribe.dump --database restored_copy

Each command prints one line on success (backup_created bytes=..., backup_verified {...}, backup_restored database=...) and exits 0. A failure prints backup_failed <code> on stderr and exits 1. --help on any of them lists its options.

The helper makes a whole-database custom-format archive, including unrelated tables when the source is shared. It creates a private 0600 temporary file, streams the dump, checks that it starts with the custom-format PGDMP header, flushes it to disk, and publishes it only after success. The directory entry is flushed too before backup_created is printed. Existing files and symlinks are never replaced, including a destination created during the dump. A failed or canceled dump removes its owned partial file. Output contains safe codes and byte counts, never child diagnostics, paths, payloads or credentials. If the backup is published but a temporary file or trust directory cannot be removed afterwards, the command still reports backup_created and exits 0, and prints backup_cleanup_incomplete on stderr naming what to remove by hand.

All three commands accept --timeout-ms: a positive integer in milliseconds, default 600000 (ten minutes), maximum 3600000 (one hour). The deadline includes connection, database statements and child execution. Cancellation and timeout terminate and reap the child; all database shutdown and cleanup share at most five additional seconds, rather than restarting the operation timer.

Use a TCP postgresql://user:password@host:port/database URL with an explicit, nonempty user, password and database. Passwordless, Unix socket, service profile, multi-host and arbitrary query parameters are refused. Inherited PG* settings are ignored. pg_dump and pg_restore receive only PATH, HOME, LANG, LC_* and TZ from the caller’s environment, plus the connection settings the helper sets; ENCRYPTION_KEY, ENCRYPTION_KEY_PREVIOUS, ADMIN_TOKEN, DATABASE_URL and every other variable are withheld from them. TLS modes supported by both clients are:

• No sslmode, or sslmode=disable: no TLS. Use a trusted local/private endpoint.
• sslmode=verify-full&sslrootcert=/absolute/path/ca.pem: verify the certificate chain and the URL’s hostname with that explicit PEM CA bundle (at most 1 MiB), and require TLS 1.2 or newer. Both clients use the same snapshot of CA bytes.

Other TLS modes, implicit/system roots, client certificates and keys, CRL settings and target/service overrides are rejected before archive/database creation. The tools disable GSS encryption, implicit client certificates and implicit CRL-file discovery. An absent CRL filename inside an owned private directory is used; this subset does not perform certificate revocation checking. Never weaken a TLS URL to make the helper accept it; use the manual workflow for unsupported setups.

The dump client must also be at least as new as the source server major. Restore to the same or a newer server major; restoring into an older server is not a supported promise. backup:create works on every supported server (15, 17, 18). backup:verify and restore need a PostgreSQL 17 or newer server: pg_restore 17+ sends SET transaction_timeout, which 15 and 16 reject, so the helpers refuse those servers with backup_failed server_version before creating any database. To move a 15 installation’s data, create the backup on 15 and restore it onto a 17 or 18 server. CI tests the helpers with client 18 against PostgreSQL 15 (refusal), 17 and 18. See the PostgreSQL pg_dump documentation and TLS documentation.

Manual backup

The existing manual container workflow remains available:

On your own database, Wayscribe’s tables are ordinary tables in it: back them up the way that database is already backed up.

With the bundled overlay, the data is in the postgres-data volume. With COMPOSE_FILE set as in §1:

docker compose exec -T postgres \
  pg_dump -U wayscribe -d wayscribe --format=custom > wayscribe-$(date +%F).dump

The stack built from source runs the same PostgreSQL service. Name its file instead of relying on COMPOSE_FILE:

docker compose -f infrastructure/compose.yaml exec -T postgres \
  pg_dump -U wayscribe -d wayscribe --format=custom > wayscribe-$(date +%F).dump

Treat the dump as if it contained your customers’ request bodies, because it does. Payloads are stored as jsonb in the clear; redaction, not encryption, is what keeps secrets out of them. A dump is readable by anyone who holds it.

Entity identifiers and alias values are encrypted, with a key derived from ENCRYPTION_KEY, which is not in the dump. So a backup taken without that key restores a database whose journeys cannot be searched or attributed to a customer, while their payloads remain readable. Store the key separately, and store it somewhere you will still have it when you need the backup.

3. Restore

The checkout helper pnpm run backup:restore --input wayscribe.dump --database restored_copy accepts only trusted operator-held custom-format archives: a dump can execute database definitions. The destination name must start with a lowercase ASCII letter, contain only lowercase letters, digits and underscores, and be at most 63 bytes. The source name and postgres, template0, and template1 are refused. DATABASE_URL is the maintenance connection on the destination cluster; its role needs CREATEDB.

The helper claims a new database with CREATE DATABASE ... TEMPLATE template0. It restores atomically with no original ownership or privileges, and never uses --clean or --create. No migrations or application traffic run automatically. Successful restores keep the new database. A failed restore drops only its acknowledged owned database. Existing databases are never dropped.

A connection failure during CREATE can leave the result uncertain. The create_outcome_unknown error names the requested database for manual inspection; the helper does not infer ownership from its existence or drop it. If cleanup of an acknowledged database fails, the original error is retained and a separate instruction names the owned database requiring manual cleanup.

Verifying a backup

pnpm run backup:verify --input wayscribe.dump checks that an archive restores and that its data is usable with the keys you hold, without touching the source database or keeping a copy. Run it after every scheduled backup and before you depend on one.

It restores into a generated wayscribe_restore_check_<random> database on the DATABASE_URL cluster (so the role needs CREATEDB), inspects it in a read-only transaction with statement time capped at ten minutes, and always drops it, including on failure, timeout and cancellation. The same uncertain-CREATE and manual-cleanup messages as restore apply. It checks, in order:

• Schema. The archive’s applied migrations must match this checkout’s exactly. A missing or extra migration reports schema_mismatch and nothing else is read, so verify with the release that took the backup.
• Integrity. Every project, environment, journey, event, destination and replay reference resolves, including the project/environment scope reads rely on, and every alias an event stated belongs to that event’s journey. A broken reference reports integrity_failed.
• Encrypted values. Every entity identifier, alias value and replay header set decrypts under ENCRYPTION_KEY or ENCRYPTION_KEY_PREVIOUS. Any that does not reports encrypted_values_unreadable.
• Search tokens. For each decrypted entity identifier and alias value, the stored search token must be the current or previous key’s token for that value. A mismatch reports search_tokens_mismatch: the record restores but search cannot find it.

The result is one JSON line: row counts per application table, the number of encrypted values examined, pending/unknown migration counts, and the failure codes. It names no rows, values or keys. backup_verified exits 0; backup_verification_failed exits 1. Missing or malformed keys fail before any database is created with backup_failed keys_required.

Manual restore

The manual commands below restore into an existing database and intentionally use --clean; select that destination carefully.

Name the dump you are restoring. With the bundled overlay and COMPOSE_FILE set as in §1:

docker compose exec -T postgres \
  pg_restore -U wayscribe -d wayscribe --clean --if-exists < wayscribe-2026-09-15.dump

On the stack built from source:

docker compose -f infrastructure/compose.yaml exec -T postgres \
  pg_restore -U wayscribe -d wayscribe --clean --if-exists < wayscribe-2026-09-15.dump

Restore against the same ENCRYPTION_KEY. A restore under a different key leaves every encrypted field undecryptable and every search token unmatchable. The interface degrades one field at a time rather than failing, so this looks like missing data rather than an error. Check the key first when a restored installation shows blank identifiers. A dump taken before a key rotation needs the old key back as ENCRYPTION_KEY_PREVIOUS; see §6.

4. Upgrading

Migrate, then deploy. In that order.

pnpm db:migrate

/ready returns 503 with migrations_pending until the schema matches what the running build expects. That is the designed answer, not a fault: a process serving against an older schema must not take traffic. A rolling deploy that starts new containers before migrating will report unready until the migration runs, which is the correct behaviour and not a reason to skip the ordering.

/ready also reports the running version: version, commit when the build recorded one, and source, which says whether those came from the image’s own build (build) or only from the package (package, meaning a source run or a hand-built image). All three are on the 503 answers too, so the first question after an unready container is already answered (docs/API_SPEC.md section 14).

Every release is gated on an upgrade test (scripts/upgrade-test.mjs, the upgrade-test CI job). It builds the previous release (until there is one, a commit from before v1’s storage format changes), records journeys, aliases, a transformation diff, an error and a replay destination through it, then starts the new build against the same database volume and checks that every search and detail reads back unchanged, that the journey list reads the old journeys under every filter (with no label or last step, since nothing is backfilled), that existing API keys still authenticate, and that rotate:reencrypt upgrades the stored formats. From a source checkout, node scripts/upgrade-test.mjs runs it with Docker and nothing else, and UPGRADE_BASELINE_REF chooses the version to upgrade from.

Migrations are plain ESM JavaScript at the package root (ADR-027). Do not rename a migration file after it has been applied anywhere. knex records the filename, so a rename makes the directory look corrupt to every database that already ran it.

From a published image, without a source checkout:

docker run --rm --network wayscribe_default \
  -e DATABASE_URL=postgresql://wayscribe:wayscribe@postgres:5432/wayscribe \
  --entrypoint node wayscribe-api packages/database/dist/cli.js migrate

Migration 017 adds a column to entity_aliases

017_alias_displayable.js adds displayable boolean not null default false (ADR-053). On PostgreSQL 11 and later that is a catalogue change, not a table rewrite, so it finishes at once on any size of table. It does need a moment of exclusive lock, and to get it the ALTER waits for every transaction already using the table while new ingestion queues behind it. The migration therefore sets lock_timeout to five seconds: behind a long-running transaction it fails with canceling statement due to lock timeout and changes nothing, and running migrate again retries it. If it keeps failing, look for the transaction holding the table:

select pid, state, xact_start, query from pg_stat_activity
where pid in (select pid from pg_locks where relation = 'entity_aliases'::regclass);

The previous API keeps working while the column exists and it does not know about it: its inserts get false, which is how every alias read before.

Migration 020 records which aliases each event stated

020_event_stated_aliases.js adds stated_alias_ids uuid[] to journey_events, nullable and with no default (F-042). Like 017 it is a catalogue change with no table rewrite, holds its exclusive lock for an instant, and sets lock_timeout to five seconds, so behind a long-running transaction it fails with canceling statement due to lock timeout, changes nothing, and succeeds when migrate runs again; check 'journey_events'::regclass in the query above.

There is no backfill, because nothing recorded which event stated which alias before. Events stored before the migration, and events the previous API stores between migrate and deploy, read aliases: null from GET /v1/events/:eventId, which means “not recorded”, not “stated none”. Their aliases are still on the journey, GET /v1/journeys/:journeyId.

While the previous API and this one both run, the same event sent to one of each at once can deadlock, because the previous build writes the event row before it locks the journey: one delivery gets a 500 and the SDK retries it. It happens only during the deploy, and only for a duplicate of an event that states aliases.

Migration 021 records the step that failed a journey

021_journey_failed_step.js adds four columns to journeys, failed_step, failed_step_at, failed_step_received_at and failed_step_event_id, all nullable with no default and no index (ADR-063, F-047). Like 020 it is a catalogue change with no table rewrite, holds its exclusive lock for an instant, and sets lock_timeout to five seconds, so behind a long-running transaction it fails with canceling statement due to lock timeout, changes nothing, and succeeds when migrate runs again; check 'journeys'::regclass in the query above. Rolling it back drops the four columns under the same timeout.

There is no backfill. A journey failed before the migration, or failed by the previous API between migrate and deploy, reads failedStep: null until its next failure, and the web app shows its last step for it, as before.

The previous API does not know the columns, so when it clears or completes a failed journey it leaves them as they were. The reads show failedStep only while the status is failed, which hides that stale value while the journey is out of failed. If the previous API then fails the journey again, the read can name the earlier, cleared step until a failure applied by this build and stamped later replaces it; a failure this build applies while the journey is not failed replaces it however it is stamped. This happens only while the previous API still writes, so it ends with the deploy, and it only ever names a step that did fail in that journey.

Upgrading to the journey browsing release (migrations 018 and 019)

This release adds journey labels, last steps and partial text matching on public values (ADR-054). Two migrations run, and neither rewrites a table.

Migration 018 (018_journey_browse.js) adds six nullable columns to journeys (label, last_step, and the timestamp and event id that decide which event set each), display_value to entity_aliases, the check constraint entity_aliases_display_value_only_when_displayable, and the trigger entity_aliases_clear_masked_display_value with its plpgsql function of the same name. None of the columns has a default, so each is a catalogue change with no table rewrite. It runs in two transactions of its own:

1. The ALTERs, with the constraint added not valid, which is also a catalogue change, and the trigger, created under the lock the ALTERs already hold. The lock handling is the same as 017: five seconds of lock_timeout per lock request, and running migrate again retries after canceling statement due to lock timeout. The query above finds the transaction in the way; check 'journeys'::regclass as well.
2. VALIDATE CONSTRAINT, which reads every row of entity_aliases but takes only a SHARE UPDATE EXCLUSIVE lock, so reads and ingestion carry on while it runs. The column is null in every existing row, so validation always passes. If it gives up behind a lock, the columns, the trigger and the unvalidated constraint stay, the migration is not recorded, and the next migrate skips the first transaction and validates.

The timeout applies to each lock request, and the first transaction takes two locks. The ALTER on journeys can wait up to five seconds, then holds that lock while the ALTER on entity_aliases waits up to five more, so in the worst case writes to journeys stall for about ten seconds before the migration either finishes or gives up. A rollback (down) takes the same two locks in the same order, so it can stall writes to journeys for as long. Ingestion also locks journeys before entity_aliases, so a deadlock with ingestion is not expected in either direction. Rolling back 018 on its own (knex’s migrate:down --name 018_journey_browse.js) drops display_value, and PostgreSQL drops 019’s entity_aliases_displayable_idx with it, while 019 stays recorded as applied; roll back the whole batch (db:rollback), or 019 and then 018. Do not roll back 017 alone while 018 is applied either: its column is dropped, and 018’s trigger then fails every alias write because it reads displayable.

Migration 019 (019_journey_browse_indexes.js) builds journeys_project_recent_idx and entity_aliases_displayable_idx with CREATE INDEX CONCURRENTLY, so it does not block ingestion, but it can wait: up to 10 minutes for its lock and for every transaction that started before the build, such as a long retention batch, an admin deletion or a nightly pg_dump. It then fails with canceling statement due to lock timeout, and running migrate again drops the invalid index the attempt left and builds it afresh. Run it against PostgreSQL directly, not through a transaction-pooling PgBouncer. On Helm, allow for the wait with helm upgrade --timeout 30m; the migrate Job now prints migration failed ... see the error above rather than database not ready for a failure that is not a connection failure, retries such a failure once rather than 30 times, and stops after migrations.activeDeadlineSeconds (30 minutes by default). Indexes in section 10 has the query that shows which transaction the build is waiting on, and what to do if the migrate process was killed partway.

There is no backfill. A journey recorded before the upgrade shows no last step until its next event, and no label until an event that carries a label arrives; until then the Journeys page shows it by entity type and identifier, as the Recent page did. Only aliases stated displayable after the upgrade get a plain-text copy, so q does not find an older displayable alias until an event states it again. The previous API, still running between migrate and deploy or alongside the new one during a rolling upgrade, writes rows without these columns, and they read null in the same way, which the constraint allows. It also masks aliases the new API has already given a copy, and it lowers the flag without clearing the copy, because it does not know the column exists; its key rotation (rotate:reencrypt) folds duplicates the same way. The constraint would refuse those statements, failing the event with a 500. The trigger clears the copy of any row written masked before the constraint is checked, so the previous build’s statements succeed and the copy goes with the flag. The upgrade test checks that journeys the previous build recorded list with label and lastStep null.

Migration 015 rewrites every replay run’s headers

Replays used to store the headers they sent, including the destination’s decrypted configured headers, in replay_runs.request_headers. Migration 015_redact_replay_run_headers.js replaces every value in that column with [REDACTED] and keeps the header names. An old row does not say which headers came from the destination, so the ones Wayscribe set itself, such as user-agent, are redacted too.

It is one UPDATE of every replay_runs row that has headers, in one transaction. Measured on 2026-09-15 on PostgreSQL 17, on the development machine (commit 8e948a4, which does not name it), with 100,000 runs carrying payloads of about 1 KB: the migration took about 2 seconds. An update to an existing run while it ran, such as the previous API finishing a replay, waited until it committed, about 2.3 seconds. Inserts of new runs were not blocked. The table doubled in size, because every row is rewritten, until vacuum reclaimed the old versions.

Migrate, then deploy, still applies, and it leaves a window: between the migration committing and the new API taking traffic, the previous API is still the one serving, and a replay it sends is stored the old way. Replays are manual, so the simplest course is not to send one during the upgrade.

If one was sent, redact again once the new API is serving, limited to runs created from an hour before migration 015 was applied. Count them first:

select count(*)
from replay_runs
where request_headers is not null
  and created_at >= (
    select migration_time from knex_migrations
    where name = '015_redact_replay_run_headers.js'
  ) - interval '1 hour';

Then rewrite the same rows:

update replay_runs
set request_headers = case
  when jsonb_typeof(request_headers) = 'object' then (
    select coalesce(jsonb_object_agg(key, to_jsonb('[REDACTED]'::text)), '{}'::jsonb)
    from jsonb_each(request_headers)
  )
  else null
end
where request_headers is not null
  and created_at >= (
    select migration_time from knex_migrations
    where name = '015_redact_replay_run_headers.js'
  ) - interval '1 hour';

Runs the new API wrote in that window are rewritten too, and lose the values it kept on purpose, such as user-agent and x-wayscribe-replay. The header names stay. Nothing secret is lost, only what those rows could show about the non-secret headers.

Its down migration does nothing: the values cannot be restored, and restoring them would be the defect. The migration does not reach copies. A dump, WAL archive, or replica snapshot taken before it still holds the header values, and so do the destination’s credentials inside them. If a destination header was a credential that matters, rotate it at the destination as well.

5. Projects and keys

A new installation has no projects, and a key belongs to one. Create the project first; the environment is created for you by key:create.

docker compose run --rm --entrypoint node api \
  packages/database/dist/cli.js project:create acme "Acme Payments"

docker compose run --rm --entrypoint node api \
  packages/database/dist/cli.js key:create acme production checkout-worker

project:list, key:list, and key:revoke do what they say. A key is printed once and is not recoverable: issue another rather than hunting for it.

Every command prints its arguments and options with --help, as in packages/database/dist/cli.js key:create --help, and --help alone lists the commands and how to run them in the image and in a checkout. Neither needs DATABASE_URL, and a request for help never runs the command: --help or -h anywhere before a -- that follows an argument prints the help, including after the -- that pnpm run <script> -- --help adds. After such a --, where every argument is otherwise a value, either one is refused and nothing runs. help <command> prints the same help, and so does a bare help, in any letter case, before a -- on every command except project:create, key:create and key:list, where it is read as a slug or a name. On a command that deletes or revokes, help is always a request for help, even where a journey id, an identifier, an environment or a destination could be called help; a value that really is help goes after --, as in delete:journey acme -- help, as a value beginning with a dash does.

An argument containing a dash other than the ASCII hyphen is refused before a --, since it is most likely a -- that macOS, Slack, a word processor or a full-width keyboard turned into an en or em dash, a non-breaking hyphen, a minus sign or a full-width hyphen-minus: rollback --help with its -- turned into an em dash used to roll back. Type -- with two hyphens. A name or value that genuinely contains such a dash, such as a customer name with an en dash between two surnames, goes after --. An argument beyond those a command declares is refused rather than ignored; project:create and key:create are the exception, because their name takes the rest of the arguments, as in project:create acme Acme Payments.

A project or key name that begins with a dash goes after --, as for the deletion commands: project:create beta -- -Beta names the project -Beta. Before a --, anything beginning with a dash is read as a flag, and one the command does not have is refused.

key:create and key:revoke each write an audit row, api_key.created or api_key.revoked, naming the key by its prefix (SECURITY.md section 13).

For a script, key:create … --json prints one JSON object on one line and nothing else, so nothing has to be parsed by position:

docker compose run --rm --entrypoint node api \
  packages/database/dist/cli.js key:create acme production checkout-worker --json

{"apiKey":"wsk_…","keyPrefix":"wsk_…","projectSlug":"acme","environmentName":"production"}

Those four fields are all it prints, and key:create --help names them. The key’s database id is left out.

The flag may appear anywhere in the arguments. Without it the human form above is unchanged. Either way the key reaches stdout, so redirect it into the place it belongs rather than leaving it in a terminal’s scrollback or a CI job’s log.

A slug is lowercase letters, digits and hyphens, because it reaches project selection, the CLI, and the interface. It cannot be changed afterwards without touching everywhere an operator has written it down.

6. Key rotation

Changing ENCRYPTION_KEY outright loses access to what is already stored. Four things derive from it by HKDF: field encryption, search tokens, the API-key pepper, and event content hashes. Swap the value and recreate the API containers, and every stored entity identifier, alias value, and replay destination header stops decrypting, every existing journey stops being findable by identifier, and every issued API key answers 401.

Rotation avoids that with a grace period (ADR-044). The key being replaced stays configured as ENCRYPTION_KEY_PREVIOUS beside the new one. The API writes under the new key and reads under both, API keys move to the new key as they authenticate, and rotate:reencrypt moves the stored data across. Only when rotate:status says nothing is left under the old key does the old key go.

Where the keys are read from

Set both variables in the one place your stack reads them from. A key set anywhere else never arrives, and nothing says so.

Stack	Read from
infrastructure/compose.yaml, with or without the demo overlay	infrastructure/defaults.env, then the repository-root .env, which wins. Shell exports do not reach these services. Edit .env.
compose.published.yaml	the shell, or a .env beside compose.published.yaml. A shell export wins over that file.
Helm	secrets.encryptionKey and secrets.encryptionKeyPrevious, or the ENCRYPTION_KEY and ENCRYPTION_KEY_PREVIOUS keys of your existingSecret.
pnpm commands in a source checkout	the repository-root .env. A variable exported in the shell wins over it.
Either Compose stack, from files	ENCRYPTION_KEY_FILE, ENCRYPTION_KEY_PREVIOUS_FILE, ADMIN_TOKEN_FILE and READ_TOKEN_FILE, each naming a file the value is read from at startup. See “Secrets the container’s environment does not hold” below.

Surrounding whitespace is trimmed from both keys and from ADMIN_TOKEN, so a trailing newline from a secrets file, or a space either side of a pasted value, does not make a different key or a token nobody can type at the login form. An empty ENCRYPTION_KEY_PREVIOUS means no rotation is in progress. The same value in both variables stops the API at boot with a message saying so, rather than starting a rotation that rotates nothing.

Secrets the container’s environment does not hold

With the Compose paths above, ENCRYPTION_KEY and ADMIN_TOKEN are ordinary container environment variables. They are part of the container’s Config.Env, so anyone who can run docker inspect or docker compose config on the host can read them, and that is exactly the access starting the stack requires. Holding Docker access to that host is therefore equivalent to holding the key that decrypts every stored payload and the token that signs every admin session. That is an ordinary trade-off rather than a defect, and it is worth knowing before deciding who gets that access. A Helm install already avoids it with existingSecret, a Kubernetes Secret rather than a container environment variable.

ENCRYPTION_KEY_FILE, ENCRYPTION_KEY_PREVIOUS_FILE, ADMIN_TOKEN_FILE and READ_TOKEN_FILE are the Compose equivalent. When one is set, the value is read from that file at startup by the API, the web app, doctor, and the key rotation commands. The plain variables stay supported and unchanged.

READ_TOKEN is optional. Set, it lets a reader principal search and read one project, payloads included, and nothing else (ADR-070); it must differ from ADMIN_TOKEN. The web app’s WEB_ANONYMOUS_READ_ONLY=true signs every visitor in as that reader with no login; it needs READ_TOKEN, refuses to start with ADMIN_TOKEN set, and is meant for a public demo of generated data only (docs/DEMO_HOSTING.md). doctor warns while it is on.

infrastructure/compose.secret-files.yaml is an overlay that wires this up with Docker’s own secrets mechanism:

export COMPOSE_FILE=compose.published.yaml:compose.secret-files.yaml
export ENCRYPTION_KEY_PATH=/etc/wayscribe/encryption-key
export ADMIN_TOKEN_PATH=/etc/wayscribe/admin-token
docker compose up -d

Each file holds the value and nothing else. Whitespace around the value is ignored, so a file ending in a newline is fine: reading the file drops what is at the end, and the setting itself is trimmed at both ends afterwards, exactly as it is when it comes from a variable. A file and a variable holding the same value therefore give the same key or token, and the length is measured on the trimmed value. An empty file is refused rather than read as an unset setting, which would otherwise start the stack on a published default. Setting both a variable and its _FILE is refused at startup, by the API and the web app alike, naming the setting and printing no value, because nothing on a running container would say which had won. doctor reports the same refusal as a failed Secrets from files check.

This keeps the values out of docker inspect. It does not hide them from anything that can read the files or enter the running container, which Docker access to the host also allows.

A rotation on this path is started and ended by the setting, not by the file. To start one, add ENCRYPTION_KEY_PREVIOUS_FILE pointing at a file holding the outgoing key, beside ENCRYPTION_KEY_FILE holding the new one. To end it, at step 6 of the procedure below, unset ENCRYPTION_KEY_PREVIOUS_FILE and recreate the containers. Emptying the file it names does not end the rotation: an empty file is refused, by design, so the API would stop instead of starting without the previous key. With the overlay, remove the encryption_key_previous secret and its two lines rather than truncating the file the host holds.

Running the commands

rotate:reencrypt and rotate:status read the keys the way the API does, so run them with the same two variables the API has.

# Published images, with COMPOSE_FILE set as in §1
docker compose run --rm --entrypoint node api \
  packages/database/dist/cli.js rotate:status

# infrastructure/compose.yaml, which reads .env for the one-off container too
docker compose -f infrastructure/compose.yaml run --rm --entrypoint node api \
  packages/database/dist/cli.js rotate:status

# A source checkout
pnpm rotate:status

Replace rotate:status with rotate:reencrypt for the other command. On Helm, add ENCRYPTION_KEY_PREVIOUS to the kubectl run that deploy/helm/README.md uses for project:create. key:create during a rotation needs both keys as well, so a key issued mid-rotation verifies against the API beside it. key:revoke reads no key.

For the published images, COMPOSE_FILE (§1) keeps every command on the files the stack was started with. For a stack built from source, give every docker compose command in this section the same -f files the stack was started with. For the demo that means adding -f infrastructure/compose.demo.yaml after -f infrastructure/compose.yaml. Leave it out and Compose warns about orphan containers and suggests --remove-orphans; do not take that advice, because it removes the demo services.

The procedure

1. Generate the new key: openssl rand -hex 32. Keep the old one. It is the only way back if something goes wrong, and a backup taken before the rotation needs it (see below).
2. Where your stack reads its keys (above), set ENCRYPTION_KEY to the new key and ENCRYPTION_KEY_PREVIOUS to the old one.
3. Recreate every API container with the new keys: docker compose -f infrastructure/compose.yaml up -d (or docker compose up -d for the published images, with COMPOSE_FILE set as in §1), or kubectl rollout restart deployment/<release>-wayscribe-api after the Helm upgrade, since the chart does not restart pods when its secret changes. Not docker compose restart: it restarts the container with the environment it was created with, and does not re-read env_file, so the new keys never arrive. Every replica has to be on the new keys before the next step, because one still on the old key keeps writing under it.
4. Run rotate:reencrypt. It prints a line per batch, then one summary line per table. If it is interrupted, run it again; it resumes where the committed batches left off.
5. Run rotate:status. It exits 0 when every row and every unrevoked API key is under the new key. Until then, work through what it lists (below) and run it again.
6. Remove ENCRYPTION_KEY_PREVIOUS and recreate the API containers again, the same way as in step 3. On the file path, unset ENCRYPTION_KEY_PREVIOUS_FILE rather than emptying the file it names: an empty file is refused, so the API would stop instead of starting without the previous key (“Secrets the container’s environment does not hold”, above).
7. Run rotate:status once more. It should report Previous key: not set and Complete, and the API’s boot log should carry no warning about unreadable data.

From step 3 on, nothing is interrupted: new events record, old journeys search and open, and every API key still authenticates.

Event content hashes are not moved by any step, because a hash can only be recomputed from the event it describes. After step 6, a resend of an event recorded under the old key, which only a duplicate delivery produces, is answered 409 event_id_conflict rather than recognised as a duplicate. The SDK treats that as permanent and drops the resend; the event stored the first time is unchanged (ADR-048).

A script can wait on step 5, since the exit code is the answer:

until pnpm rotate:status > /dev/null; do sleep 300; done

Use the form of the command your stack runs (above) in place of pnpm. The exit code is also 1 when the command cannot run at all, such as with DATABASE_URL unset or the database unreachable, so a loop like this one waits forever on a misconfiguration; its error still reaches stderr every pass.

What rotate:status lists

• API keys not yet under the current key. Each moves the next time it authenticates, because the presented key is the only thing a verifier can be recomputed from. Wait for the services holding them to send events. A key that will not be used again should be revoked with key:revoke and the prefix the listing shows (key:revoke wsk_AbCdEfGh), and reissued with key:create if something still needs one. Revoked keys are not counted.
• API keys whose key id is not recorded. Issued before key ids were stored. They record one the next time they authenticate. During a rotation they may be under either key, so they are treated like the keys above and keep the exit code at 1. With no previous key configured they are listed as key id not recorded yet; recorded on next use and do not affect the exit code: they can only be under the one key there is.
• API keys under a key that is not configured. Their key id is neither the current nor the previous key, so they cannot authenticate and will not move by themselves. Set ENCRYPTION_KEY_PREVIOUS to the key the listing names to let them authenticate during a grace period, or revoke them. They keep the exit code at 1.
• Rows under the previous key. Run rotate:reencrypt again. A row that ingestion changed while the command was reading it is left for the next run, and the command says how many there were.
• Rows with no stored value. Reported, and they do not hold the rotation open: there is nothing in them to rewrite. Their search tokens, though, can only be recomputed from a value, so they stay under the old key and stop matching search once ENCRYPTION_KEY_PREVIOUS is removed.
• Rows under an unknown key, malformed rows, and legacy rows no key opens. These keep the exit code at 1 until the key that wrote them is configured again or the rows are deleted. rotate:reencrypt counts them as unrecoverable and leaves them as they are. An unknown key is named by its id, which is a fingerprint of the key and safe to paste into a ticket.

What rotate:reencrypt costs

It walks journeys, entity_aliases, and replay_destinations by primary key in batches of 500, one transaction per batch, and spends one to three queries on each row. A large table takes a long time; run it when you can leave it. It is safe under live traffic: each rewrite is conditional on the value it read, so a concurrent write is never overwritten.

Without ENCRYPTION_KEY_PREVIOUS it upgrades instead of rotating. It rewrites legacy values the current key opens into the fr1 format under that same key, leaves search tokens as they are, and counts anything under another key as unrecoverable. Its first line names the mode: Upgrading legacy values under key … or Re-encrypting under key …, reading values under … and legacy values. If the first line says Upgrading when you meant to rotate, step 2 did not reach this process; nothing under the old key was touched.

It exits 1 when another run holds the rotation lock, and changes nothing. The retention sweep exits 0 in the same situation, and the difference is deliberate: a sweep runs hourly in every API replica, so a held lock means another replica is doing the same work. A re-encryption is started once, by someone, and a script waiting on it has to know this run did nothing.

It needs two database connections, one holding the lock for the whole run and one doing the work. The retention sweep is built the same way. A connection pooler or a role CONNECTION LIMIT that allows this process one connection will stall it.

A lost lock stops the run. The connection holding the lock sits in an idle transaction while the batches run on another, so a server idle_in_transaction_session_timeout shorter than one batch, a pooler, or an administrator can end it. The command checks that connection before every batch, which also resets that timeout, so only a single batch that outlasts it drops the lock. When it is lost the command stops after the batch in progress, prints that it lost the rotation lock, and exits 1. Nothing needs undoing, since every rewrite is conditional on the value it read: run rotate:reencrypt again. Check the setting with SHOW idle_in_transaction_session_timeout;. The retention sweep checks its lock the same way; when it loses it, the API logs retention sweep lost its lock and stopped early and the next sweep continues.

API keys are not touched. See above for why they move on use instead.

If the previous key was removed too early

The API still starts, because refusing would stop ingestion over a read problem. Instead:

• after it starts listening, it logs one warning with a count per table of the data the configured keys cannot read, and names rotate:status;
• a read that meets a value under a key it lacks logs one warning per key id;
• a replay whose destination headers cannot be decrypted is refused and recorded as blocked, rather than sent without the destination’s credentials;
• API keys that had not moved yet answer 401.

Put the old key back as ENCRYPTION_KEY_PREVIOUS, recreate the API containers, and continue from step 4. Nothing stored was lost; it was only unreadable. Events a service sent with a key that answered 401 in the meantime were not stored, and the SDK does not retry a 4xx.

Backups across a rotation

A dump holds values under whichever key was current when it was taken. To restore one taken before a rotation finished, configure the key it was taken under as ENCRYPTION_KEY_PREVIOUS beside the current key, restore, and run the procedure from step 3. Keep an old key for as long as you keep backups made under it.

Installations from before key ids

Values written before this release carry no key id. They read normally, and rotate:status counts them as legacy and exits 1 until they are rewritten. A rotation rewrites them; so does one run of rotate:reencrypt with only ENCRYPTION_KEY set, which upgrades them into the new format under the key they are already under. rotate:status then exits 0. API keys issued before this release are listed as key id not recorded yet; recorded on next use, which does not hold the exit code at 1 while no rotation is under way.

ADMIN_TOKEN and API keys

ADMIN_TOKEN rotation is safe and cheap. It invalidates every web session, because the session signing key is derived from it, which is the correct behaviour after a suspected leak, not an inconvenience.

API keys rotate individually and without side effects:

pnpm key:create local production new-worker-key
pnpm key:revoke wsk_AbCdEfGh    # the old key's prefix, from key:list

7. Retention

Each environment has its own retention_days. A sweep runs hourly inside the API process, guarded by a PostgreSQL advisory lock so replicas do not delete concurrently (ADR-026), and deletes in bounded batches.

Deleting a journey removes its events, aliases, and replay runs by cascade.

Retention stops while the API is down and resumes on the next start. Each sweep that deleted anything writes one line:

{ "journeysDeleted": 1420, "batches": 2, "environmentsExamined": 3, "durationMs": 91 }

A sweep that stops completing writes nothing at all, which is why its outcome is also a metric (§13).

Retention is not the only way data leaves: §8 deletes a journey, an identifier, or a time window on demand.

DEFAULT_RETENTION_DAYS applies to environments created by db:seed and key:create. Changing it does not alter environments that already exist: update environments.retention_days for those. No command or route changes an environment’s retention or capture settings, so a change made in SQL writes no audit row (SECURITY.md section 13).

8. Deleting data

Retention removes data by age. Three commands remove it on demand: one journey, every journey matching an identifier, and an environment’s journeys in a time window. They cover what retention cannot wait for: a redaction miss that stored something it should not have, a customer asking to be forgotten, and a noisy window such as a load test. A fourth command removes a replay destination (ADR-045).

Every deletion is a hard delete. A journey’s events, its aliases, and the replay runs of those events go with it. Each deletion writes a row to audit_events in the same transaction as the delete, so nothing is deleted without a record.

Dry run first

delete:identifier and delete:range take --dry-run, which lists exactly what the same command would delete and deletes nothing. Run it, read the table, then run the command again without the flag.

pnpm delete:identifier local CLI-CUST-1 --dry-run

Dry run: 2 journeys match in local (all environments). Nothing was deleted.

ID         ENVIRONMENT  ENTITY TYPE  EVENTS  LAST ACTIVITY
jrn_cli_b  development  customer     1       2026-08-11 10:00:00
jrn_cli_a  development  customer     2       2026-08-10 10:00:05

pnpm delete:identifier local CLI-CUST-1

  batch 1: 2 journeys deleted so far
Deleted 2 journeys and 3 events in local (all environments). The audit log records the search token, not the value.
Journeys recorded after the run started were left alone; run it again with --dry-run to check for any.

The commands

Command	Deletes
pnpm delete:journey <project> <journey-id>	one journey
pnpm delete:identifier <project> <value> [--environment <name>] [--dry-run]	every journey whose entity id or any alias is the value, which is what search finds for it
pnpm delete:range <project> <environment> --before <date> [--after <date>] [--dry-run]	the environment’s journeys whose last event falls in [after, before)
pnpm delete:destination <project> <destination-id>	a replay destination and every replay run sent to it

Erasure finds entity ids and aliases, not payloads. delete:identifier, and POST /v1/erasures, match the value against journeys’ entity ids and aliases, which is exactly what search finds. A value that appears only inside a payload is not matched: an email address a service put in input but never recorded with identify() or as an alias. An erasure request for such a value deletes nothing, and the dry run shows zero journeys. An erasure that scans payloads is not built. Until it is, find those journeys another way (the entity id or an alias the customer is also known by, the time window and service of their activity, or your own application’s records), confirm each on its journey page, and delete each with delete:journey <project> <journey-id> or the journey page’s delete action. Recording the identifiers a request may name as aliases is what makes erasure find them.

Each exits 1 whenever what was asked did not fully happen: an unknown project, environment, journey, or destination, a value that is only whitespace, an invalid date, a held lock, or a run that stopped part way. A script can rely on the exit code.

A date is an ISO-8601 date, read as midnight UTC, or a timestamp with an explicit offset such as 2026-09-01T00:00:00Z. A timestamp without an offset is refused, because it would delete a different window depending on the server’s time zone. --after defaults to the beginning of time and must be earlier than --before.

delete:identifier needs ENCRYPTION_KEY, and ENCRYPTION_KEY_PREVIOUS during a rotation, because the value is matched by its search tokens under both keys. It never prints the value, and nor does a refusal of its arguments: one it does not accept is named by its position, such as Unexpected argument 3, as on every deletion command.

The value you type is still recorded outside Wayscribe. It stays in your shell’s history, and anyone who can list processes on that host sees it in ps while the command runs. The same is true of the API key given to doctor --api-key (§12), which is a credential rather than an identifier. In bash with HISTCONTROL=ignorespace (or zsh with setopt HIST_IGNORE_SPACE), start the command with a space and it is not saved; otherwise remove the line afterwards (history -d <number> in bash). Run it on a host whose process list only operators can read.

A value or id that begins with a dash goes after --, so it is not read as an option: pnpm delete:identifier acme -- -A1. So does a value that is the word help, which before a -- asks for help. The one exception is --help or -h, which the CLI refuses as a value, so that asking for help can never delete anything; erase such a value through the admin API (POST /v1/erasures, docs/API_SPEC.md).

delete:range takes the retention sweep’s advisory lock, so a range deletion and a sweep never run at once. While the sweep holds it the command deletes nothing, says so, and exits 1; run it again when the sweep has finished.

From the published image, without a checkout:

docker compose run --rm --entrypoint node api \
  packages/database/dist/cli.js delete:identifier acme customer-42@example.com --dry-run

The admin API deletes a journey, an identifier, and a destination (docs/API_SPEC.md §17 to §19), and a journey’s page links to a confirmation page that deletes it. Range deletion is in the CLI only.

Late arrivals

An erasure or a range deletion deletes the journeys that existed when it started. A journey created while it runs, because the customer is still being ingested or the environment is live, is left for the next run; without that limit a busy environment could keep a run going, and holding the retention lock, indefinitely. Run the dry run again afterwards, and if it lists anything, run the deletion again.

What the audit row holds

Action	Records
journey.deleted	the journey id, its environment, its event count
erasure.completed	the current key’s search token for the value, the environment if one was named, journeys and events deleted, complete
range.deleted	the environment id, after, before, journeys deleted, complete
replay_destination.deleted	the destination id, its name, replay runs deleted

An erasure’s row never holds the value. A later erasure of the same value records the same token under the same key, so the two can be matched.

A destination’s audit rows, at creation and at deletion, hold its name and not its base URL. Rows written by replay_destination.created before that changed do hold the base URL, and deleting the destination leaves them. If a URL carried something that must go, strip it and keep the rest of the record:

update audit_events
   set metadata = metadata - 'baseUrl'
 where action = 'replay_destination.created';

Erasure and range deletion commit in batches, each its own transaction. The row is written with the first batch and updated with every batch after it, and the last batch, which finds nothing left, sets complete: true. complete: false means the run stopped before it finished: the process died, the database connection failed, or a range deletion lost its lock, in which case the row also has lockLost: true. Its counts are exactly what was deleted. Run the same command again to delete the rest; that run writes a row of its own.

What deletion does not remove

• The bytes, until vacuum. A deleted row stays in the table’s files as a dead row until autovacuum processes the table, and even then the space is marked reusable rather than overwritten. VACUUM on journeys, journey_events, and entity_aliases hastens the first; VACUUM FULL rewrites the files, and locks the tables while it does.
• Backups. Every dump and every WAL archive taken before the deletion still holds the data. Deleting from the database does not reach them; how long they are kept is your backup retention, and an erasure request covers them too.
• What your services logged. Deletion removes what Wayscribe stored.

9. Exposure

Every published port binds to 127.0.0.1. A docker compose up on a cloud host does not expose the stack to the internet, and that is the only thing standing between the default configuration and an open admin interface.

If you put Wayscribe behind a reverse proxy, terminate TLS there and do not republish the container ports on 0.0.0.0. The web app sets its own Content-Security-Policy, with a fresh nonce per response, and X-Frame-Options, Referrer-Policy and X-Content-Type-Options (docs/SECURITY.md §2). Let them through: a proxy that replaces the policy with a fixed one blocks the interface’s scripts, and one that caches pages would serve a stale nonce. Pages are sent Cache-Control: no-store already.

Pass the original Host header through to the web app, or set X-Forwarded-Host, and set X-Forwarded-Proto. The web app refuses a form post whose Origin names a different host from the one the request was sent to, so a proxy that rewrites Host to the container’s name without X-Forwarded-Host refuses every sign-in. Redirects after a form post are path-only (Location: /login?error=invalid), so they follow whatever scheme and host the browser is on and do not depend on X-Forwarded-Proto; set it anyway, because anything behind the proxy that builds an absolute URL does.

location / {
  proxy_pass http://127.0.0.1:3000;
  proxy_set_header Host $host;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

The admin token grants project-wide read of every recorded payload. It is a single shared secret with no user accounts and no audit of who used it. Treat it as an operator credential, not a login.

Replay destinations

Replay makes the API send a recorded payload to a destination an admin configured. REPLAY_ALLOWED_HOSTS is the control that keeps that from being a request forgery tool inside your network: a destination whose host is not in the list is refused before anything is sent (ADR-033). It is load-bearing, because private addresses are deliberately allowed, since every development destination lives in one. Nothing else stands between a leaked admin token and a request to any service the API can reach.

• A host matches exactly, on any port. There are no wildcards.
• localhost is the API’s own container or pod, including the API port and METRICS_PORT.
• host.docker.internal reaches every service listening on the Docker host, not only the one under development: databases, other applications, and anything bound to the host’s loopback interface.
• A Compose or Kubernetes service name reaches that service from inside the network, whether or not it is published.

compose.published.yaml and the Helm chart default to localhost alone. infrastructure/compose.yaml, the development and demo stack, allows host.docker.internal and demo-integration so the demo and a service running on your machine can be replayed to, and deploy/helm/values-local.yaml allows host.docker.internal for a kind cluster. A production installation should set the list explicitly to the few development hosts it replays to, and to localhost or an unused name when it replays to nothing:

REPLAY_ALLOWED_HOSTS=billing-dev.internal,orders-staging.internal

An installation that relied on the previous default of localhost,host.docker.internal in compose.published.yaml or the chart must now set it.

Guessing the admin token

Both places that accept it throttle failures per source address: five failures within a minute lock that address out for five minutes. The web login redirects a locked address to /login?error=throttled. The API counts a refused credential on any route that takes the admin token (every route but ingestion and the health checks), answers it with the same 401 as before, and then answers every request from that address that presents credentials with 429 too_many_attempts and a Retry-After header, the right token included, until the lock expires. A request with no credentials is not counted, a database error while a key is looked up is not counted, and ingestion is never throttled.

The API counts refusals, not attempts in flight. Guesses sent one after another get exactly five 401s. Guesses sent all at once can get more: every request that passed the lock check before the fifth refusal was recorded still has its credential checked. On the admin-only routes that is only what was already between the check and the comparison, which has no I/O between them; on the read routes an API key is looked up in the database first, so a burst of N concurrent bad keys from one address can see up to N 401s, and the request after it is 429. That is a deliberate trade. The admin token is at least 32 characters and an API key carries 192 random bits, so extra guesses change nothing about the odds, and each costs one indexed lookup; holding the count exact would mean reserving and queueing requests in flight on the authentication path. Reads with a valid key are never held back. The web login reads the form and then checks the lock and compares with nothing in between, so it stays at exactly five under any concurrency.

An IPv6 address counts as its /64, the block one host is usually given, and an IPv6 address carrying an IPv4 one, as a dual-stack socket reports an IPv4 client (::ffff:203.0.113.5, in any spelling), as that IPv4 address. A port, brackets, and an interface zone are ignored, so [2001:db8::1]:443 and 2001:db8::1 are one address. Both counts are held in memory, per process: they reset on restart, N replicas allow N times the attempts, and each remembers at most 50,000 addresses, forgetting the one that failed least recently beyond that.

The source address is the socket’s by default. X-Forwarded-For is ignored, because any client can write it and a new value per guess would otherwise make every guess the first. Behind a reverse proxy, every request arrives from the proxy, so every client shares one count, and a guesser can lock the operator out. Set TRUSTED_PROXY_COUNT, on the API and the web app, to the number of proxies that append to X-Forwarded-For; the address used is then the entry that many hops from the right, which is what the outermost of them saw. Set it only when nothing reaches the container except through those proxies: a client that connects directly can write as many hops as the count and choose its own address.

10. Sizing

Event volume drives everything. One journey is one row plus one row per event, plus a row per alias. Payloads are stored inline as JSONB.

Measured disk per event

scripts/measure-storage.mjs records journeys of the demo’s shape (ten events and two aliases each, with small Salesforce and customer payloads averaging 120 bytes of JSON input and output per event) through the real ingestion code, in each capture mode, and reports what journeys, journey_events, and entity_aliases occupy with their indexes and TOAST.

Current smaller-scale run. Measured on 2026-09-19 at source 9da8b37, with 10,000 journeys and 100,000 events per capture mode. PostgreSQL 17.11 ran as postgres:17-alpine on aarch64 in a container limited to 2 CPUs and 3 GiB, with a 2 GiB tmpfs, on a shared Apple M3 Pro host running Node 24.19.0. Demo capture and site verification could overlap. Tmpfs is not physical storage I/O, and this is not a production capacity or throughput measurement.

Capture mode	100,000 events	Compacted, per event	Per journey
metadata-only	116.1 MiB (1,217 B)	944 B	11.9 KiB
allowlisted-fields	140.8 MiB (1,476 B)	1,190 B	14.4 KiB
redacted-payload	157.8 MiB (1,655 B)	1,377 B	16.2 KiB
full-payload	159.2 MiB (1,669 B)	1,377 B	16.3 KiB

The run completed all four modes with exit 0 and did not trigger its storage guard. In each mode, deleting 5,000 journeys and running plain VACUUM left the compacted files essentially the same size. Refilling the same number of journeys reused space: metadata-only, for example, went from 116.1 MiB as ingested to 90.1 MiB compacted, 90.2 MiB after the sweep and VACUUM, then 112.8 MiB after refill.

Historical million-event run. Measured on 2026-09-15 on an Apple M3 Pro, PostgreSQL 17.11 (postgres:17-alpine, default configuration) in Docker Desktop with 12 CPUs and 7.75 GiB:

Capture mode	100,000 events	1,000,000 events	Compacted, per event	Per journey
metadata-only	113.5 MiB (1,190 B)	1,001 MiB (1,049 B)	873 B	10.2 KiB
allowlisted-fields	134.8 MiB (1,414 B)	1.19 GiB (1,273 B)	1,097 B	12.4 KiB
redacted-payload	154.2 MiB (1,617 B)	1.39 GiB (1,494 B)	1,320 B	14.6 KiB
full-payload	151.3 MiB (1,586 B)	1.39 GiB (1,496 B)	1,320 B	14.6 KiB

The historical run’s first two columns are as ingested, after a plain VACUUM; the per-event figure is the total divided by events, so each event carries its share of its journey and aliases. Compacted is after VACUUM FULL. The difference is mostly the journeys table: every event updates its journey, the updates cannot be HOT because indexed columns change, and at a million events the table and its indexes were 121 MiB as ingested against 77 MiB compacted. A running installation sits between the two.

What the historical million-event numbers say:

• Indexes are 39 to 56 percent of the disk. At a million events in metadata-only, the three tables held 559 MiB of indexes out of 1,001 MiB; in redacted-payload, 556 MiB out of 1.39 GiB.
• Payload capture costs about four times the payload’s JSON size. redacted-payload added 445 bytes per event over metadata-only for 120 bytes of JSON: JSONB is larger than JSON text for small objects, and a step with both an input and an output also stores their diff. full-payload and redacted-payload match here because the demo payloads hold no secrets.
• Retention does not shrink the files. Starting from the compacted size after VACUUM FULL (832.7 MiB, metadata-only), deleting half the journeys with the retention sweep and running a plain VACUUM left it at 832.9 MiB. Ingesting as many journeys again brought it to 982 MiB, below the 1,001 MiB the same volume took the first time: the freed space was reused over one delete-and-refill cycle. Longer runs were not measured. A sweep or a §8 deletion is not a way to get space back. VACUUM FULL returns it, but holds an exclusive lock that stops ingestion for as long as it runs.

A formula

disk ≈ events per day × retention days × bytes per event × 1.5

Take bytes per event from the 1,000,000-event column for your capture mode, and for payloads larger than the demo’s add four times their average JSON size (input plus output). The margin of 1.5 covers what the measurement does not: autovacuum falling behind a burst, journeys that keep receiving events and so outlive the window, and index growth beyond what a million events shows. It does not cover WAL (max_wal_size, 1 GB by default), backups, or the database’s other tenants.

For example, a million events a day kept 30 days in redacted-payload: 1,000,000 × 30 × 1,494 × 1.5 is 67 GB, about 63 GiB.

To measure your own shape, against a scratch database (it refuses one that already holds journeys, and works in schemas of its own that it drops after):

pnpm --filter "@wayscribe/api..." build
node scripts/measure-storage.mjs --database-url postgresql://… --journeys 10000

Indexes

Search looks a value up in one index per kind of identifier: journeys_pkey for a journey id, journeys_entity_value_idx and entity_aliases_value_idx for entity and alias values, and one index each on journey_events for trace, span, message, and correlation ids. It takes about 0.1 ms at a million journeys for a value that matches a few journeys. A value that matches thousands (a shared correlation id, say) is joined with a sequential scan of the project’s journeys, so its cost grows with the journey count: 13 ms for 2,400 matches among 120,000 journeys and 64 ms for 20,000 among a million, for an API key scoped to one environment. These search figures were measured on 2026-09-15 with EXPLAIN (ANALYZE, BUFFERS) on PostgreSQL 17, on the development machine; commits 5ffc08e and 33f8f1e record them and do not name the machine, and no script in the repository repeats them. journeys_recent_idx serves retention selection. The recent-journeys list adds journeys_status_recent_idx and journey_events_service_idx; the second costs one more index write on every event insert, and the span id index adds one on every event that carries a span id. replay_runs_journey_event_idx (migration 016) serves deletion rather than reads: every event a deleted journey takes with it looks up its replay runs through that foreign key, and without the index each one scanned the project’s replay runs (§13, Statement timeout). The journey list over every environment and its text filter use journeys_project_recent_idx and entity_aliases_displayable_idx (migration 019, Listing journeys below). Migrations 013, 014, 016, and 019 build their indexes with CREATE INDEX CONCURRENTLY, so on a large installation they take longer than the other migrations but do not block ingestion while they run (014 took 3 seconds over 3 million events, as its own comment records, measured 2026-09-15).

Run migrate against PostgreSQL directly, not through a PgBouncer in transaction pooling mode. 019 sets lock_timeout on its connection and then builds on it, and behind transaction pooling the setting and the build can land on different server connections.

A concurrent build does not block ingestion, but it waits for transactions that started before it, in any table of the database, to end: a long retention batch, an admin deletion, a nightly pg_dump. While it waits, the migrate step simply takes longer. 019 waits up to 10 minutes for any one of them, or for its own lock, and then gives up. migrate prints:

migration file "019_journey_browse_indexes.js" failed
migration failed with error: canceling statement due to lock timeout

followed by the stack, and exits 1. Nothing is blocked meanwhile, and nothing is lost: run migrate again once the transaction ends, and it drops the index the interrupted build left invalid and builds it afresh. To see what the build is waiting for, list the transactions older than it:

select pid, usename, application_name, state, xact_start,
       now() - xact_start as running_for, left(query, 80) as query
  from pg_stat_activity
 where datname = current_database()
   and xact_start < (select xact_start from pg_stat_activity
                      where query ilike 'create index concurrently%'
                      order by xact_start limit 1)
 order by xact_start;

An idle in transaction row there is a session someone left open; ending it (select pg_terminate_backend(<pid>)) lets the build finish. A pg_dump is best left to finish.

On Helm, the migrate Job retries migrate up to 30 times while the error reads as a connection failure, printing database not ready. Any other failure, such as this lock timeout, prints migration failed ... see the error above and is retried once only, so read the error printed above that line. The Job as a whole is stopped after migrations.activeDeadlineSeconds, 30 minutes by default, so the worst case is bounded; a build it stops leaves an invalid index that the next migrate rebuilds. Each attempt at 019 can wait 10 minutes, and Helm waits for the Job only as long as --timeout (5 minutes by default), so on an installation where a backup may be running, upgrade with helm upgrade --timeout 30m, matching the deadline, or expect Helm to report a timeout while the Job carries on. Before this release the Job printed database not ready for every failure, whatever the cause.

If one of those builds stops partway, what to do depends on how it stopped:

• The build failed (an error was reported, or the connection dropped) and migrate exited. Run migrate again. It drops the index the failed build left invalid and builds it afresh.

• The migrate process was killed (kill -9, an evicted pod, a stopped container). Because 013, 014, 016, and 019 run outside a transaction, the migration lock is still set and every migrate after it fails with a message that the migration table is locked; the Helm Job’s retries fail the same way and /ready stays migrations_pending. First make sure no migrate is still running anywhere, then release the lock and migrate:

  node packages/database/dist/cli.js migrate:unlock
  node packages/database/dist/cli.js migrate

  (pnpm db:migrate:unlock from a checkout.) Releasing the lock while another migrate is running lets two run at once, so check first.

Listing journeys

scripts/measure-journey-list.mjs records journeys through the real ingestion code and times GET /v1/journeys through the real API, in process (Fastify’s inject: authentication, validation, the query and the response are in the figure, the network is not). Each case is three warm-up requests and then 40 timed ones, page of 25, and the slowest cases are run again under EXPLAIN (ANALYZE, BUFFERS) with the SQL and parameters the API sent.

Current run. Measured on 2026-09-19 at source 9da8b37 on an Apple M3 Pro shared development host with Node 24.19.0. PostgreSQL 17.11 ran in an aarch64 postgres:17-alpine container limited to 2 CPUs and 3 GiB, with a 2 GiB tmpfs, 128 MB shared_buffers and jit on. The cache was warm. Demo capture and site verification could overlap. Tmpfs is not physical storage I/O, and these are development measurements rather than production latency or throughput claims.

The run recorded 120,000 journeys, 360,000 events and 360,000 aliases with four concurrent producers. It found 2,937 journeys in the last 24 hours and 89,845 in 30 days. The table reports p50 / p95 in milliseconds from 40 samples, with every index present:

Case	24 h	30 d
No filter, admin (every environment)	6.7 / 14.1	5.1 / 7.4
No filter, admin, second page	5.6 / 7.2	5.8 / 9.4
No filter, API key (one environment)	5.6 / 6.9	6.0 / 9.2
status=failed, admin	5.1 / 6.9	6.1 / 9.0
q=sync (matches many), admin	6.9 / 10.6	7.5 / 8.9
q=sync, admin, second page	7.2 / 12.0	7.9 / 10.3
q matching none, admin	29.1 / 51.2	483.3 / 594.2
q matching none, API key	26.9 / 37.8	268.8 / 432.2
q=acme and entityType=invoice, admin		5.9 / 6.6

The four EXPLAIN cases confirmed index-only alias scans with zero heap fetches. The 30-day no-match plans inspected 89,845 admin journeys or 71,904 API-key-scoped journeys; the 24-hour plans inspected 2,937 or 2,340. This completed run used only the every-index query, four EXPLAIN and ingestion stages. It did not remove indexes for a comparison. The script covers the existing browse, text, admin, API-key and second-page cases; it does not cover the newer timing filters, whose separate 20,000-journey evidence follows below.

Historical migration 019 comparison. The 2026-09-16 run used an Apple M3 Pro and PostgreSQL 17.11 in postgres:17-alpine with its default configuration, including 128 MB shared_buffers and jit on, with the cache warm.

The data: 120,000 journeys of three events each (360,000 events) and three aliases each, about 650 MiB in the three tables. Four environments hold 80, 12, 6 and 2 percent of the journeys. 85 percent completed, 5 percent failed, 10 percent still active. Entity types customer, order, invoice and subscription at 50, 30, 15 and 5 percent. Three journeys in four have a label, mostly distinct (Sync order ORD-0001234 for Acme 271), built from eight verbs and twenty company names so that some words recur: sync is in 11,136 labels, acme in 4,515. 85 percent of journeys show two of their aliases (204,442 displayable aliases in all); every journey has one masked alias, and the rest have only masked ones. Last activity is spread evenly over 40 days, so a 24-hour window holds about 2,940 journeys and a 30-day window about 89,860. The API key is scoped to the busy environment. q matching none is the worst case: nothing lets the query stop early, so every journey in the window is tested.

Migration 019 added two indexes for this list, journeys_project_recent_idx on journeys (project_id, last_event_at, id) and entity_aliases_displayable_idx on entity_aliases (project_id, journey_id) include (display_value) where displayable. The historical “before” stage used the same rows with both dropped; p50 / p95 in milliseconds:

Case	24 h before	24 h after	30 d before	30 d after
No filter, admin (every environment)	13.2 / 15.0	1.6 / 2.8	30.7 / 32.3	1.5 / 3.4
No filter, admin, second page	12.8 / 13.3	1.7 / 1.8	31.4 / 32.3	1.7 / 2.0
No filter, API key (one environment)	2.6 / 2.9	2.3 / 3.4	2.9 / 3.8	1.9 / 2.1
status=failed, admin	1.6 / 5.2	2.6 / 3.3	2.5 / 3.9	2.1 / 3.5
q=sync (matches many), admin	16.3 / 17.2	4.8 / 7.5	770.1 / 844.6	4.3 / 5.4
q=sync, admin, second page	15.2 / 17.5	2.6 / 3.8	772.3 / 799.2	4.2 / 5.4
q matching none, admin	14.9 / 15.6	10.1 / 11.7	805.7 / 842.7	261.6 / 315.6
q matching none, API key	10.6 / 11.1	8.4 / 9.1	517.6 / 538.0	207.5 / 214.1
q=acme and entityType=invoice, admin			113.6 / 117.0	4.5 / 5.4

Before the indexes, four runs on the same rows put the worst case, q matching none for an admin over 30 days, between 840 and 930 ms at p95: under the one second this list was measured against, but too close to it on a fast machine with a warm cache. With the journeys index alone, the two q matching none cases over 30 days were 731 and 614 ms at p95; the alias index is what brings them to about 300 and 200.

What the plans show:

• Before, every environment with any status read the whole window. No index led with (project_id, last_event_at): journeys_recent_idx has the environment second and journeys_status_recent_idx the status, so the planner scanned journeys_recent_idx over the window, merged it with the environments, and sorted. Without q that sort was cheap (a top-25 heapsort of 89,864 rows, 31 ms). With q every journey in the window had its label and aliases tested before the sort, so a q matching many journeys cost as much as one matching none: 89,864 journeys and 89,864 alias probes, about 545,000 buffers, about 800 ms. Now the list walks journeys_project_recent_idx backwards in order and stops after a page, so q=sync reads a few hundred journeys. The API key’s list, one environment, already walked journeys_recent_idx that way and is unchanged.
• q matching none still cannot stop early, whatever the index: it tests all 89,864 journeys for an admin over 30 days, 71,918 for the API key, 2,944 over 24 hours. What changed is the price of each test. Each alias probe is now an index-only scan of entity_aliases_displayable_idx with no heap fetch (363,699 buffers for the admin case, all cached, against 548,398 before, a quarter of them read from disk), and the whole plan is cheap enough that PostgreSQL no longer JIT-compiles it. The cost still grows with the journeys in the window, about 3 ms per thousand here. These figures were taken right after VACUUM (ANALYZE), when every page is marked all-visible. On a live system the most recently written pages are not, and an index-only scan still reads the table for those, so the gain on text matching nothing will be smaller for the newest journeys, which are the ones a 24-hour window holds, until autovacuum reaches them.
• JIT compilation was about 170 ms of the old plans. Their estimated cost passed jit_above_cost, and compiling took 171 to 176 ms of the 786 to 848 ms they executed in; with jit = off on the database, the three slowest cases measured 626, 618 and 704 ms at p95. The new plans are estimated below the threshold and do not compile, but a larger window or table can bring it back.

The timing filters were measured separately on 2026-09-18 in an isolated PostgreSQL 17 test database with 20,000 journeys and 60,000 events, after VACUUM ANALYZE, using the repository’s actual page-of-25 SQL and EXPLAIN (ANALYZE, BUFFERS). The synthetic journeys covered zero and 1-4,999 ms event-start spans, 20 percent active status, nullable event durations, and duration values on two of three events. With every row inside the required since window:

• minDurationMs=1000 walked journeys_project_recent_idx backward, removed one row at the filter before filling the page, touched 57 shared buffers including the returned summary fields, and executed in 0.157 ms.
• minStepDurationMs=2000 used a nested-loop semi-join: the same journey index supplied rows in page order and journey_events_timeline_idx probed the project-and-journey-scoped events. It touched 187 shared buffers including the returned summary fields and executed in 0.227 ms.
• inactiveBefore with a four-hour cutoff used an index-only backward scan of journeys_status_recent_idx, including project, active status, since, and cutoff in the index condition. It touched 62 shared buffers including the returned summary fields and executed in 0.086 ms.

These are warm-cache development measurements on a modest representative data set, not production latency promises. They show the predicates use the existing bounded-window and per-journey indexes, so no duration index was added. Measure with your own event density and window before adding one: a step-duration filter that rejects many journeys will perform more per-journey probes before it fills a page.

In that historical run, at 120,000 journeys, the journeys index was 11 MB and the alias index 19 MB.

Current ingestion measurement. Every event updates its journey’s last_event_at, which is now written to one more index, and every displayable alias is written to the alias index. In the current 2026-09-19 run, timed through the API with every index present, 300 samples of a batch of 100 events had a p50 / p95 of 383.1 / 450.3 ms. The batch created ten journeys of ten events, each with a label and three aliases. A single event starting a journey, also over 300 samples, measured 6.0 / 8.2 ms.

The historical 2026-09-16 ingestion comparison made the three index sets take turns, round after round, on the same tables, so drift as the tables grew was shared. p50 / p95 in milliseconds:

	Both indexes	Journeys index only	Neither
Batch of 100, run 1 (400 each)	230.6 / 279.6	237.3 / 288.5	234.3 / 289.1
Single event, run 1 (400 each)	4.3 / 6.5	4.0 / 5.8	4.6 / 7.1
Batch of 100, run 2 (800 each)	248.3 / 294.4	248.9 / 299.6	243.1 / 294.0
Single event, run 2 (800 each)	4.7 / 7.0	4.9 / 6.7	4.6 / 6.5

The difference is inside the noise. A batch, which is how the SDK sends, costs the same with either index set. A single event’s p95 moves by less than a millisecond in either direction and not consistently: in run 1 both indexes were 12 percent slower than the journeys index alone but faster than neither, and in run 2 both were 4 percent slower than the journeys index alone and 8 percent slower than neither. Run 2 started from 133,860 journeys, after run 1’s ingestion.

What to expect, then. In the current run, returned-page queries were at or below 14.1 ms p95 in every measured case. Text matching nothing still read every journey in the window: 51.2 ms p95 for 2,937 admin-visible journeys over 24 hours and 594.2 ms for 89,845 over 30 days. The API-key-scoped equivalents were 37.8 and 432.2 ms. This cost grows with the journeys in the window. An installation recording tens of thousands of journeys a day should search text over a day or a week rather than a month. If a plan of this list shows JIT in EXPLAIN ANALYZE on your data, ALTER ROLE … SET jit = off for the API’s role takes that share off.

To measure your own shape, against a scratch database (it refuses one that already holds journeys, and works in a schema of its own that it drops after):

pnpm --filter "@wayscribe/api..." build
node scripts/measure-journey-list.mjs --database-url postgresql://… --journeys 120000

--keep leaves the schema for a second run with --reuse, which applies any new migration first. Each --drop-index <name> adds a stage without that index and the ones named before it, so indexes can be compared on the same rows; --no-list measures ingestion only.

11. Security scanning

Three scanning jobs run in the security stage, and all three block. A fourth, sbom, blocks too, but judges nothing about the images: it fails only when an SBOM cannot be generated, because a release would then fail at the same step (see Verifying a published image below).

Job	Tool	What it gates
audit	pnpm audit	dependency advisories at high and above
secrets	gitleaks	credentials anywhere in the history
container-scan	Trivy	HIGH and CRITICAL CVEs in both images

GitLab’s own Dependency Scanning and Container Scanning templates are Ultimate-tier. On a Free project they produce an empty report, which looks exactly like a scanner that works, so these run the underlying tools directly. Each tool’s image is pinned by version and digest (gitleaks zricethezav/gitleaks:v8.30.1, and Trivy, Syft and cosign likewise), so a new release of a scanner changes the gate only when somebody moves the pin.

Create a pipeline schedule. Under Build → Pipeline schedules, a daily or weekly run on the default branch. This is the part that matters: an advisory is published against a dependency that has not changed, and a base-image CVE appears without anybody committing anything. A push-only gate reports yesterday’s answer indefinitely.

container-scan is deliberately not on every push: it builds two images, and the same reasoning that keeps demo and e2e manual applies. It runs on the default branch, on tags, and on the schedule.

When one of them fails

audit. Prefer fixing over ignoring. A transitive advisory can usually be pinned forward with an entry in overrides in pnpm-workspace.yaml, which removes the finding rather than hiding it. Each entry says what it is for and should be dropped once the parent ships a version that resolves it.

secrets. Assume it is real until you have read the line it matched. If it is genuinely a fixture, add an allowance to .gitleaks.toml, written against the value rather than the path, so it cannot hide whatever lands in that file next. If it is real, the credential is already published: rotate it first, and treat removing it from history as cleanup rather than as the fix.

container-scan. Check whether the package is ours before reaching for an ignore. The first run, on 2026-08-10, found seven CVEs in npm and corepack, which the base image ships and the runtime never uses; both Dockerfiles now delete them, which is a smaller attack surface as well as a clean scan.

Publishing the SDK to npm

@wayscribe/node is published by the manual publish-sdk job on a vMAJOR.MINOR.PATCH tag, with npm trusted publishing and provenance. No npm token exists anywhere in the project: the job’s GitLab OIDC token, with the audience npm:registry.npmjs.org, is exchanged by npm for a short-lived publish token, and a second token with the audience sigstore signs the provenance statement that npmjs.com shows beside the version. npm stopped issuing classic and Automation tokens in November 2025, and the granular tokens left expire in 90 days at most (npm’s announcements: creation of classic tokens stopped on 2025-11-05; checked 2026-08-10).

The one-time setup used for 0.1.0 must remain in place:

1. Keep maintainer access to the existing @wayscribe/node package. Its deprecated 0.0.1-placeholder.0 version originally reserved the name; the released SDK is published only through the job below.

2. Open npmjs.com → @wayscribe/node → Settings → Trusted publishing, choose GitLab CI/CD, and enter exactly:

   • Namespace: jojithedev
   • Project name: wayscribe
   • Top-level CI file path: .gitlab-ci.yml
   • Environment: leave empty
   • Allowed actions: enable direct npm publish

   npm stage publish is always allowed. Trusted-publisher configurations created since 2026-09-03 default to staging only, but this repository’s scripts/publish-sdk.sh calls npm publish, so the direct action must also be allowed. Changing to staged publishing is a separate release-workflow change and would add a human approval step.

3. In the same settings page, under Publishing access, choose Require two-factor authentication and disallow bypass 2FA tokens. This setting is compatible with trusted publishing and the direct npm publish action above. It disallows tokens that bypass two-factor authentication; it does not make OIDC the only possible publishing route, because an authorized maintainer can still publish interactively with a one-time password. The release job remains this repository’s intended publication route.

4. Protect v* tags (below). Anyone who can create one can run the job.

A field that does not match fails during npm publish with an authentication error, not at the start of the job. The job needs npm 11.5.1 or later, which the node:24-alpine image has; scripts/publish-sdk.sh checks the version and says so rather than failing inside publish. To rehearse a release anywhere, run DRY_RUN=1 scripts/publish-sdk.sh vX.Y.Z, which builds, packs, checks the packed manifest, and runs npm publish --dry-run --force. The force flag is limited to this no-write rehearsal so a version already present on npm can be checked again; the credentialed publication keeps npm’s normal version guards.

To verify a published version’s provenance, run npm audit signatures in a project that depends on it, or read the provenance panel on the package page.

Publishing the Python SDK to PyPI

wayscribe is published by two manual jobs on a packages/sdk-python/v<version> tag, where the version is the one in packages/sdk-python/pyproject.toml (PEP 440, so 0.1.0a1 or 0.2.0): publish-python-testpypi uploads to TestPyPI and publish-python to PyPI. As with npm, no PyPI token exists anywhere in the project. The job’s GitLab OIDC token (audience pypi, or testpypi) is exchanged by twine for a short-lived upload token, and a second token with the audience sigstore signs the PEP 740 attestations that PyPI shows beside each file (ADR-067).

One-time setup, done on 2026-09-25:

1. On pypi.org, Your account, Publishing, a pending GitLab trusted publisher: PyPI project wayscribe, namespace jojithedev, project wayscribe, top-level pipeline file .gitlab-ci.yml, environment pypi. The first upload creates the project under that account.
2. The same on test.pypi.org, with environment testpypi.
3. Protect packages/sdk-python/v* tags, as v* is. Anyone who can create one can run the jobs.

Rehearse on TestPyPI first, then install from it in a clean container: pip install --index-url https://test.pypi.org/simple/ wayscribe==<version> (add --pre for an alpha). To check a release anywhere without uploading, run DRY_RUN=1 scripts/publish-python.sh packages/sdk-python/v<version>, which builds the sdist and wheel, checks the wheel’s file list, and runs twine check --strict; the python-publish-dry-run job runs the same on main.

Verifying a published image

Every released api and web image is signed, and carries a CycloneDX software bill of materials (SBOM) for each platform, attached as a signed attestation. Both are made in the publish-images job with keyless signing: GitLab gives the job an OIDC token, and Sigstore’s certificate authority issues a short-lived certificate naming the pipeline file and the tag it ran for. There is no signing key for anyone to steal, and the signature is recorded in Sigstore’s public transparency log. Releases v0.1.0, v0.2.1 and v0.2.2 completed and verified this path; their exact digests and public SBOM downloads are in the release records for v0.1.0 and v0.2.1 and v0.2.2.

The job pushes each image by digest with no tag, attaches the SBOMs to that digest, signs it, verifies the signature and attestations as below, and only then points the version tag and latest at it. A release whose signing failed has no version tag, so an unsigned image cannot be pulled by its version.

The version tag is the git tag, v included: release 0.1.0 publishes api:v0.1.0 and web:v0.1.0. WAYSCRIBE_VERSION in compose.published.yaml and image.tag in the Helm chart take that form, and the chart’s default is v plus its appVersion. scripts/check-chart-image-tag.sh renders the chart in CI and fails if its default is anything else.

What a signature proves depends on tag protection. The certificate says a pipeline ran .gitlab-ci.yml at refs/tags/vX.Y.Z in this project. That is worth something only if nobody but a maintainer can create a v* tag. Before release v0.1.0, the project owner added v* as a protected tag with Allowed to create set to Maintainers (Settings → Repository → Protected tags). Keep that protection in place. The pipeline also runs release jobs only for tags of the exact form vMAJOR.MINOR.PATCH, so phase-*, usable-v0 and pre-release tags never publish, but that pattern is a guard against mistakes, not against someone who can push tags.

Before enabling a registry cleanup policy, make sure it keeps tags matching sha256-.*. Where the registry does not serve the OCI referrers API, cosign stores signatures and attestations under tags of that form beside the image, and a cleanup policy that deletes them leaves every release unverifiable.

Use cosign 3.x, the major version the pipeline signs with, plus jq and Docker’s buildx for the SBOM steps.

In the commands below, vX.Y.Z stands for the release you run; releases are 0.x, beginning with v0.1.0.

The signature. For a version you have chosen, name its tag exactly:

cosign verify registry.gitlab.com/jojithedev/wayscribe/api:vX.Y.Z \
  --certificate-identity 'https://gitlab.com/jojithedev/wayscribe//.gitlab-ci.yml@refs/tags/vX.Y.Z' \
  --certificate-oidc-issuer https://gitlab.com

The double slash before .gitlab-ci.yml is part of GitLab’s identity format, not a typo. To accept any release tag, for example in an admission policy that checks every image a cluster pulls:

cosign verify registry.gitlab.com/jojithedev/wayscribe/api:vX.Y.Z \
  --certificate-identity-regexp '^https://gitlab\.com/jojithedev/wayscribe//\.gitlab-ci\.yml@refs/tags/v[0-9]+\.[0-9]+\.[0-9]+$' \
  --certificate-oidc-issuer https://gitlab.com

Replace api with web for the web image. A pass prints the checks cosign performed and exits 0. Any other result means the image was not produced by this project’s release pipeline: do not run it, and report it as described in SECURITY.md.

latest verifies only with the regexp form. Its certificate names the version tag the image was released under, not latest, so the exact identity above would need you to know that version already. Pin a version tag or a digest in anything you deploy.

The SBOM. An SBOM lists the files of one platform, so it is attached to that platform’s manifest rather than to the multi-platform tag. Find the digest for your platform, verify the attestation on it, and extract the document:

IMAGE=registry.gitlab.com/jojithedev/wayscribe/api
TAG=vX.Y.Z
ARCH=amd64   # or arm64

DIGEST=$(docker buildx imagetools inspect "$IMAGE:$TAG" --format '{{json .Manifest}}' \
  | jq -r --arg arch "$ARCH" \
    '.manifests[] | select(.platform.os == "linux" and .platform.architecture == $arch) | .digest')

cosign verify-attestation "$IMAGE@$DIGEST" --type cyclonedx \
  --certificate-identity "https://gitlab.com/jojithedev/wayscribe//.gitlab-ci.yml@refs/tags/$TAG" \
  --certificate-oidc-issuer https://gitlab.com \
  | head -n 1 | jq -r '.payload' | base64 -d | jq '.predicate' > "api-$TAG-$ARCH.cdx.json"

cosign verify-attestation prints one JSON document per verified attestation. A release attaches one SBOM per platform digest, so there is one line; head -n 1 keeps the extraction to a single document if a digest ever carries more.

The result is a CycloneDX 1.6 JSON document that Grype, Trivy (trivy sbom api-vX.Y.Z-amd64.cdx.json), and Dependency-Track read directly. The release job keeps the same files as artifacts, and the sbom job produces SBOMs for the images built from each default-branch pipeline, which is what to look at between releases.

Rehearsing the publish step

scripts/publish-image.sh with DRY_RUN=1 runs everything publish-images does except the cosign calls, which it prints: the multi-platform build, the push by digest, the SBOMs from the pushed manifests, and the tags. Point it at a throwaway registry rather than the real one:

docker run -d --name rehearsal-registry -p 127.0.0.1:5055:5000 registry:2
printf '[registry."127.0.0.1:5055"]\n  http = true\n' > /tmp/buildkitd.toml
docker buildx create --name rehearsal --driver-opt network=host \
  --buildkitd-config /tmp/buildkitd.toml --use

DRY_RUN=1 SBOM_DIR=/tmp/sboms \
  SYFT_DOCKER_ARGS='--network host -e SYFT_REGISTRY_INSECURE_USE_HTTP=true' \
  scripts/publish-image.sh v0.0.0-rehearsal \
    apps/api/Dockerfile=127.0.0.1:5055/wayscribe/api \
    apps/web/Dockerfile=127.0.0.1:5055/wayscribe/web

docker buildx rm rehearsal && docker rm -f rehearsal-registry

Signing itself cannot be rehearsed without publishing: the signature is pushed beside the image and written to the public transparency log.

12. Checking an installation

doctor checks an installation end to end and says what to fix. Run it after key:create, after an upgrade, and whenever something looks wrong. It is in the API image, like the other commands:

docker compose run --rm --entrypoint node api \
  packages/database/dist/cli.js doctor --api-url http://api:8080 --api-key wsk_…

From a checkout, pnpm run doctor --api-url http://localhost:8080 --api-key wsk_… reads the repository-root .env.

A key passed as --api-key is part of the command line, so it is visible to anything that can read the process list, which inside a container is anything with Docker access to the host. doctor reads WAYSCRIBE_API_KEY instead when the flag is absent, which is the safer route:

docker compose run --rm --entrypoint node \
  -e WAYSCRIBE_API_KEY api packages/database/dist/cli.js doctor --api-url http://api:8080

The value is trimmed, so a key read from a file that ends in a newline works. --api-key wins when both are given, for checking one key while the environment holds another.

A variable that is set but empty is reported rather than left out, so the summary counts it as skipped and says the check did not happen:

SKIP  API key                 Not checked: WAYSCRIBE_API_KEY is set but empty.

That is what a Compose file’s WAYSCRIBE_API_KEY: ${WAYSCRIBE_API_KEY} passes on a host without the variable, and what -e WAYSCRIBE_API_KEY passes from a shell that has it set empty. Like every SKIP, it does not change the exit code. With the variable unset and no flag, which is what -e passes from a shell without it, doctor checks no key and prints no line for one.

Run it with the API’s environment, because that is what it checks: the same DATABASE_URL, ENCRYPTION_KEY, ADMIN_TOKEN, and DATABASE_STATEMENT_TIMEOUT_MS. docker compose run … api gives it exactly that. Inside Compose the API is http://api:8080, not localhost.

The statement timeout row reads DATABASE_STATEMENT_TIMEOUT_MS from doctor’s own environment and never asks the running API, so it says so, and it can pass while API reachable fails:

PASS  Statement timeout       DATABASE_STATEMENT_TIMEOUT_MS here is 4000 ms, so an API started with this environment cancels a statement after 4000 ms.

Each check prints one line, and anything that did not pass prints its fix beneath it:

Check	Fails when	Warns when
Database reachable	the connection is refused, the host does not resolve, or authentication fails
PostgreSQL version	below 15	newer than 18, the newest release CI tests (CI tests 15, 17 and 18)
Migrations	any are pending, the database has one this build does not, or the role has no USAGE on the schema holding them (the fix names the GRANT)
ENCRYPTION_KEY, ADMIN_TOKEN, ENCRYPTION_KEY_PREVIOUS	one is a published development default, or ADMIN_TOKEN is too short to start the API	ADMIN_TOKEN is not set where doctor runs
Keys readable	stored data or API keys are under a key that is not configured (the boot check’s count)	a rotation is in progress
Projects and keys		no project, no unrevoked API key, or the published demo key (wsk_demo0000) is unrevoked
Journey environments	an event was written by another environment’s API key than its journey’s own, which ingestion now refuses (ADR-038, amendment) and earlier builds did not
Secret-looking names		a recently stored payload holds a plain value under a key name that looks like a secret, or the sample did not finish within 5 seconds or could not run
API key (--api-key or WAYSCRIBE_API_KEY)	the key is unknown, revoked, belongs to a removed project, or does not verify under the configured keys
API reachable (--api-url)	GET /ready does not answer 200; its reason is printed
Statement timeout	the value is invalid	it is 0

A check that depends on one that failed prints SKIP rather than failing a second time: with migrations pending, nothing that reads the tables runs. A check that cannot run at all, such as one refused by a role without grants, prints FAIL with PostgreSQL’s SQLSTATE (42501 for a missing grant) and a fix where doctor knows one, and the remaining checks still run. The exit code is 1 when anything failed and 0 otherwise, so warnings do not break a script.

The API key is checked locally, with the keyring, and never sent anywhere. The output names its prefix and the project and environment it belongs to, which is the environment every event it sends must name. Doctor never prints the database password, the admin token, either encryption key, or more of a key than its prefix.

It changes nothing. It reads applied migrations from knex_migrations only when that table exists, so on a database nobody has migrated it does not create knex’s tables the way migrate and /ready do; it does not record a key as used; and it does not move a verifier during a rotation. An applied migration this build does not have, left by a newer build, is a FAIL of its own.

Events written across environments

Before the amendment to ADR-038 an API key for one environment could write events and aliases into a journey another environment created. Journey environments counts the events that were. It prints counts and no journey ids, because a journey id can carry a business identifier. List them with:

select p.slug as project, j.id as journey_id, je.name as journey_environment,
       ee.name as event_environment, count(*) as events
  from journey_events e
  join journeys j on j.project_id = e.project_id and j.id = e.journey_id
  join projects p on p.id = j.project_id
  join environments je on je.id = j.environment_id
  join environments ee on ee.id = e.environment_id
 where e.environment_id <> j.environment_id
 group by p.slug, j.id, je.name, ee.name;

Each row is a journey that holds another environment’s events. Its aliases cannot be told apart: entity_aliases records no environment, so an alias on such a journey may have come from either side, and a status of failed may be the other environment’s. Read the journey, then delete it with pnpm delete:journey <project> <journey-id> (§8). Deleting only the foreign events would leave their aliases behind. The query and the check scan journey_events; on a large installation run it off-peak, and if doctor’s check is cancelled by the statement timeout, run the query directly.

Secret-looking names stored in plain text

Redaction matches names, so a credential under a name no rule covers is stored in the clear (SECURITY.md section 4, ADR-055). The Node SDK warns about these as it records; Secret-looking names finds them for every sender, from what was stored:

WARN  Secret-looking names    2 key names that look like secrets hold plain values in the 2,000 most recent events sampled: authToken (in 12), sessionCredential (in 3).
                              Fix: If a name holds a secret, add "**.<name>" to the SDK's redact option, or to the environment's redaction_paths for another sender; values already stored stay until deleted (docs/OPERATIONS.md §8). If it does not, the warning can be ignored; it never fails doctor.

It prints key names and counts, never a value, at most ten names, each cut to 64 characters with credential shapes masked and control characters replaced. It is a warning at most, so a false positive never changes the exit code, and a sample that cannot finish within 5 seconds, or cannot run at all (a missing grant, say), is a warning with the reason rather than a failure. For a name that is not a secret, the SDK’s knownSafeNames silences the SDK’s line; doctor has no such list.

What it reads. The cap of 2,000 events is shared evenly between environments, in every project. Each environment gives the 5 latest events of each of its 100 most recently active journeys, newest first, up to its share, so one busy environment cannot fill the sample; with more than 2,000 environments, those after the first 2,000 in project order get none. From each event’s input_payload, output_payload and custom_metadata it reads every object at any depth and keeps the keys whose value could be a credential by the SDK’s value rule (a number, or a string that is not empty, not [REDACTED], not a setting word such as none, and at least 8 characters under a name ending in auth), and whose folded name ends in the last three characters of a term. Only those names and their counts leave the database; doctor applies the full name rule to them. error, runtime_metadata, deployment_metadata, payload_diff and replay runs are not read.

What it costs. The journeys come from journeys_recent_idx and the events from journey_events_timeline_idx as an index-only scan, then each sampled event is read once by primary key, so the cost depends on the sample and not on the size of the table. It runs in a read-only transaction with statement_timeout at 5 seconds. Measured on 2026-09-16 with EXPLAIN (ANALYZE, BUFFERS) on PostgreSQL 17, on the development machine (commit 190564b, which does not name it), with 600,000 events (626 MB of journey_events, about 5 KB of payload each) in four environments: 1,600 events sampled (four shares of 400), 7,698 shared buffers, about 155 ms warm and 168 ms on the first run, most of it reading keys with jsonb_each.

Looking wider. A name used once, long ago, is outside the sample. To look over a period, run this off-peak, with the window you want; it prints names and counts only:

set statement_timeout = '60s';
select k.key as name, count(distinct (e.project_id, e.id)) as events
  from journey_events e
  cross join lateral (values (e.input_payload), (e.output_payload), (e.custom_metadata)) d(doc)
  cross join lateral jsonb_path_query(d.doc, 'strict $.**') o(value)
  cross join lateral jsonb_each(o.value) k(key, value)
 where e.received_at > now() - interval '7 days'
   and d.doc is not null
   and jsonb_typeof(o.value) = 'object'
   and jsonb_typeof(k.value) in ('string', 'number')
   and k.value not in ('""'::jsonb, '"[REDACTED]"'::jsonb)
   and k.key ~* '(token|secret|passw(or)?d|credentials?|auth(orization)?|cookies?|signature|api_?key|private_?key|connection_?string|database_?url|dsn)$'
 group by k.key
 order by events desc;

That pattern is a coarse version of the rule in SDK_SPEC.md section 13, and it scans every event in the window, since no index covers received_at.

13. Monitoring

Metrics

Set METRICS_PORT and the API serves Prometheus metrics at /metrics on that port, and only there: /metrics on the API port is 404, so publishing ingestion never publishes metrics by accident. Unset, which is the default, nothing listens (ADR-047).

No Compose file in this repository publishes the port. A Prometheus on the same Compose network scrapes api:9464; one on the host needs a loopback mapping in an override file:

compose.override.yaml

services:
  api:
    ports:
      - "127.0.0.1:9464:9464"

With Helm, api.metricsPort adds a container port named metrics that no Service carries; scrape the pods.

The endpoint has no authentication. It carries no request values, but it does describe traffic, so keep it on a network only your monitoring reaches.

Metric	Type	Labels
wayscribe_http_requests_total	counter	method, route, status
wayscribe_http_request_duration_seconds	histogram	method, route
wayscribe_events_total	counter	result: accepted, duplicate, rejected
wayscribe_query_timeouts_total	counter	route
wayscribe_db_pool_connections	gauge	state: used, free, pending
wayscribe_retention_sweep_runs_total	counter	outcome: completed, locked, stopped_early, failed
wayscribe_retention_journeys_deleted_total	counter
wayscribe_retention_last_success_timestamp_seconds	gauge
wayscribe_unreadable_values	gauge	table
process_resident_memory_bytes	gauge
nodejs_eventloop_lag_seconds	gauge

route is the route pattern the router matched, such as /v1/journeys/:journeyId, never the path. A request that matched no route is unmatched, so a scanner walking random paths adds one series, not one per path. method is one of the seven methods the API uses or other. No label carries a project id, key prefix, entity, or any value from a request.

A URL the router refuses before any route runs is counted too, as unmatched: 400 for malformed percent-encoding and 414 for a path parameter longer than any id. A request too malformed for Node’s HTTP parser never becomes a request, so it is not counted; at LOG_LEVEL=trace it is logged as client error.

rejected counts every event the API told a client it did not store: a validation failure, an environment the key does not cover, a conflicting event id, and also a storage failure or a timed-out statement, which the client may send again. duplicate is an event already stored, accepted and not stored twice.

The request duration buckets are 5, 10, 25, 50, 100, 250 and 500 milliseconds, then 1, 2.5, 5, 10 and 30 seconds. Ingestion into a nearby database lands in the first few. So does a search for a value that matches a few journeys, which is under a millisecond in the database at a million journeys; one matching tens of thousands of journeys takes tens of milliseconds. Anything in seconds is worth a look. A request that ran into the default 15-second statement timeout lands in the 30-second bucket, apart from ordinary slow ones.

Every counter is per process and starts at zero when the API starts, so alert on increase() or rate(), never on the raw value. The retention gauges are per replica too: the replica that holds the lock sweeps, and the others count locked.

wayscribe_unreadable_values is set by the boot check a moment after the API starts, and not afterwards. The check is skipped while migrations are pending, so an API started before migrate ran (the infrastructure/compose.yaml stack has no migrate service) has no such series at all until it is restarted after migrating. Alert on its absence only if every API is started after migrations, as compose.published.yaml and the Helm chart do.

Alerts worth starting with

groups:
  - name: wayscribe
    rules:
      # Events refused. A new service with a mistyped environment shows up here
      # first, whatever the sending service does or does not report itself.
      - alert: WayscribeRejectingEvents
        expr: sum(increase(wayscribe_events_total{result="rejected"}[15m])) > 0
        for: 15m

      # Retention has not completed anywhere for a day. Disk grows and data
      # outlives its retention window.
      - alert: WayscribeRetentionStalled
        expr: time() - max(wayscribe_retention_last_success_timestamp_seconds) > 86400
        for: 2h

      # Queries are being cancelled, or requests are waiting for a connection.
      - alert: WayscribeDatabaseStrained
        expr: >
          sum(increase(wayscribe_query_timeouts_total[10m])) > 0
          or max(wayscribe_db_pool_connections{state="pending"}) > 0
        for: 10m

The gauge is 0 until a replica completes its first sweep, which runs an hour after it starts and hourly after that. The max across replicas carries the answer while some are young, and for: 2h keeps a restart of every replica at once from firing it.

Statement timeout

Every statement the API runs is cancelled after DATABASE_STATEMENT_TIMEOUT_MS milliseconds, 15000 by default. That covers ingestion, search and the other reads, so one slow query cannot hold a connection ingestion needs.

Deletions are the exception. Each retention batch, and each transaction of an admin’s journey deletion, erasure, or destination deletion, lifts the timeout for itself with SET LOCAL statement_timeout = 0. They are bounded by batch size and started by the system or an operator, a journey with many events legitimately takes longer to cascade than a search should, and a retention batch cancelled every hour would never let the sweep reach the next environment. On a database with 1,000 expired journeys of 200 events each, one retention batch took 2.4 seconds with migration 016’s index and 9.5 seconds without it, beside only 1,000 replay runs (measured on 2026-09-15 on PostgreSQL 17, on the development machine; commit 4c7ab65 records it and does not name the machine).

The request gets 503 query_timeout with its request id; the API logs one warning naming the route and the request id, never the SQL or its parameters, and counts it in wayscribe_query_timeouts_total.

It is set on each connection when the pool opens it, with SET statement_timeout. Behind PgBouncer in transaction pooling mode a session setting does not stay with one client, so there set the timeout on the database role instead (ALTER ROLE wayscribe SET statement_timeout = '15s') and DATABASE_STATEMENT_TIMEOUT_MS=0.

0 disables it, and doctor warns when it is. The database CLI never applies it: migrate may build an index on a large table for a long time, and rotate:reencrypt and the deletion commands bound each statement by batch size.

Logs

The API writes JSON lines to standard output, at LOG_LEVEL (info by default). At info, every request writes a line when it arrives and one when it completes.

A request line carries the method, the path, and the names of its query parameters with every value replaced by [REDACTED]:

{ "req": { "method": "GET", "url": "/v1/search?q=[REDACTED]&limit=[REDACTED]" } }

A searched value is usually a customer identifier, and the Journeys page’s filters name services and environments and carry text a reader half remembers, so no query value is ever logged. A parameter name that does not look like one (an email address pasted without =, or anything longer than 64 characters) is replaced too, and so is anything after a ; in the path, where some clients put session ids. The path is logged whole, so a journey id in /v1/journeys/:journeyId does appear. A request that matches no route writes no line of its own beyond these two.

Headers are not logged. As a second guard, the logger censors authorization and cookie in any headers object a log call includes, and also x-api-key, x-wayscribe-api-key, and a response’s set-cookie under req and res.

A request too malformed for Node to parse never becomes a request line. At trace it is logged as client error, with the parser’s error code and message. Node attaches the raw bytes it received to that error as rawPacket, headers and query string included; no error the API logs ever carries that property.

Before this, the request line carried the full URL, so logs kept from an earlier version hold searched identifiers and the filters of the Recent page (now the Journeys page) in the clear. Treat them as personal data, and let them age out or delete them.

An error’s own properties are logged, except the ones a database error fills with row contents. PostgreSQL’s detail prints the row a constraint refused (“Failing row contains (…)”) or the key a unique violation found, and where and internalQuery can quote a statement with its values, so all three are logged as [REDACTED]. The SQLSTATE code, constraint, table, column and routine stay, which is enough to tell which rule failed. The message is kept: for these failures it holds the statement with $1 placeholders, not the values bound to them. Logs from an earlier version can hold detail in the clear; treat their failure lines as personal data too.

14. When something is wrong

Symptom	Look at
Not sure what is wrong	run doctor (§12)
/ready 503 migrations_pending	run pnpm db:migrate
Requests return 503 query_timeout	a statement ran past DATABASE_STATEMENT_TIMEOUT_MS: the route is in the API’s warning log and wayscribe_query_timeouts_total; check the database’s load before raising the timeout (§13)
Every search returns nothing	which project the session selected; see /projects
A search that worked stops working	rotate:status: an unknown key id means ENCRYPTION_KEY_PREVIOUS was removed before re-encryption finished (§6)
Boot log warns of stored data the configured keys cannot read	restore the old key as ENCRYPTION_KEY_PREVIOUS and recreate the API (§6)
API exits at boot: ENCRYPTION_KEY_PREVIOUS is the same key as ENCRYPTION_KEY	set it to the key being replaced, not the new one
New keys changed nothing after docker compose restart	restart does not re-read env_file; recreate with docker compose … up -d
Replay blocked with headers_key_not_configured	the destination’s headers are under a key that is not configured; restore it as ENCRYPTION_KEY_PREVIOUS and run rotate:reencrypt
rotate:reencrypt exits 1 saying the lock is held	another run is still going; let it finish, then rotate:status
delete:range exits 1 saying the retention lock is held	the retention sweep is running; run it again when that finishes
An audit row for a deletion says complete: false	the run stopped part way; run the same command again (§8)
SDK sends nothing	Troubleshooting walks through it: key validity, environment match, and the SDK’s onDiagnostic reports and counters()
An event is refused unauthorized_environment	the key’s environment does not match the event’s. POST /v1/events/batch, which the SDK uses, answers 202 with httpStatus: 403 in that event’s result; only POST /v1/events answers the request 403 (Troubleshooting)
Ingestion returns 401 after working	the key was revoked, which pnpm key:list shows, or it had not authenticated before ENCRYPTION_KEY_PREVIOUS was removed (§6)

The SDK’s shutdown() returns counters, and counters() returns them at any time: recorded, sent, rejected, dropped, transportErrors, captureErrors, breakerOpened and others. sent, rejected, and dropped add up to recorded; payloadsOmitted counts payloads replaced by [PAYLOAD_TOO_LARGE] on events that were still sent, and payloadsTruncated payloads sent with a string cut to the 65,536 character limit. If you raised MAX_EVENT_PAYLOAD_BYTES, raise the SDK’s maxEventBytes to match, or the SDK keeps fitting events to the default. A non-zero dropped means events were not delivered: the bounded queue shed them under backpressure, the server kept refusing them for now past the retry budget, its reply gave no verdict for them, or shutdown() finished with them undelivered. Each diagnostic’s code says which (packages/sdk-node/README.md, “Is it sending?”), and droppedByCause counts them by that code, every cause present at zero, so a health endpoint can tell a collector slower than the shutdown timeout (shutdown) from a proxy answering with the wrong body (no_verdict). Five sends in a row whose replies gave no verdict at all open the breaker, so breakerOpened rises for that proxy too (ADR-063).

Optional annotated OTLP logs

OTLP_LOGS_ENABLED=false leaves /v1/logs absent. Set it to true to accept annotated JSON/protobuf logs on the existing API listener. No Collector or extra service is required, and native SDKs work with the default. Use an environment API key and configure exporters for at most 100 records per batch.

OTLP_MAX_REQUEST_BYTES=4194304 independently bounds compressed and expanded request bytes; valid range is 1..67108864. Native event limits and fixed codec structure caps still apply. Compose exposes these two environment variables; For source Compose, set them in repository-root .env; published Compose reads them from the interpolation environment (shell or --env-file). Helm uses api.otlpLogsEnabled and api.otlpMaxRequestBytes. The separate failed-admin-authentication throttle does not apply to ingestion.

For 503, retry the unchanged export, including stable IDs/timestamps: earlier records may already have committed and will deduplicate. A 500 is not retryable: it means a non-transient server failure, so check the API log rather than letting the Collector retry. A 200 permanent partial success reports the rejected count and, in error_message, the first refusal codes with counts; the API also logs the codes. See OTLP logs and the official exporter example. The published API images include the receiver from v0.2.1, disabled unless OTLP_LOGS_ENABLED=true.