Updating

Stops and removes the backend, frontend and cron apps from PM2, then proves the frontend port (3000) and the backend port (NEXT_PUBLIC_BACKEND_PORT, default 4000) are actually free and scans for a backend process PM2 does not own. If either check fails it exits non-zero and the update stops before touching anything — a second backend on the same database during a schema migration is far worse than a failed update.

It then puts the maintenance server on those two ports: 503 JSON for anything under /api/, a 503 HTML page for everything else, both with Retry-After: 300. Port 4001 is deliberately left unbound so the migration step can use it.

pnpm stop:all removes the maintenance server too, which closes the ports entirely instead of serving a page — use it only when you want the site to be visibly down.

Runs pnpm install -r --no-frozen-lockfile, then verifies that every declared dependency and every executable the rest of the chain shells out to actually resolves, and repairs the tree if they do not.

This is not belt-and-braces. When a release changes dependency resolution, pnpm rebuilds the root store but leaves backend/node_modules in place, and its links now point at store paths that were just replaced. A dangling link is not an empty directory — it looks present and fails at require() time, so the symptom is the backend dying on Cannot find module 'bullmq', one package per restart, with nothing saying "your install is incomplete".

If it cannot produce a working tree it stops the update rather than let a schema migration run against a backend that cannot boot. The message tells you which of the three usual causes to check: a full disk, something still running and holding files open, or a damaged pnpm store.

Sequelize applies the schema during boot, so the migration step is a boot. It runs backend/dist/index.js directly — not under PM2, so nothing can restart it behind your back — with the scheduler switched off and boot restricted to the schema phase, on the first free port from NEXT_PUBLIC_BACKEND_PORT + 1 upward (4001–4020 on a default install).

It then waits for GET /api/settings to answer, which is the observable that says initialisation finished, and kills the process. The cap is 180 seconds. A very large schema legitimately needs longer:

node scripts/updator-migrate.js --timeout=600000

If it exits early or never becomes ready, nothing is seeded and the site stays on the maintenance page — which is the correct place to stop, because the alternative is seeders writing through models the database does not match yet.

Schema behaviour is controlled by DB_SYNC in .env: lazy (the default) syncs only what changed, using the fingerprint in backend/.sync-hash; none authenticates without altering anything; always forces a full ALTER sync, which is the escape hatch when the schema has drifted outside Sequelize. force drops and recreates every table and destroys all data — never set it on a live install.

Writes the rows the code expects to exist: permissions, roles, deposit gateways, notification templates. It runs through the new models, which is exactly why it comes after the schema step and never before it.

Rebuilds frontend/.next. This is not optional and it is not skippable to save time.

Every NEXT_PUBLIC_* value is inlined into the browser bundle when the frontend is built. NEXT_PUBLIC_SITE_URL is the origin every client-side API call uses and the hostname in the next/image allowlist. An un-rebuilt frontend serves the previous release's pages against the new backend, and if you also changed the domain, the browser calls the old one and images 400.

The backend needs no equivalent step: backend/dist ships pre-built inside the release package.

Clears maintenance mode and starts the three PM2 apps from production.config.js: backend on NEXT_PUBLIC_BACKEND_PORT, frontend on 3000 (hardcoded — it is not moved by .env), and cron on 4001 with CRON_MODE=only. Nothing should ever connect to 4001; the port exists only so the scheduler process does not collide with the backend.

Setting CRON_MODE=inline in .env drops the cron app entirely and schedules jobs inside the backend process instead, so you will see two apps rather than three.

The chain stopped at one of its links. Read the last error it printed — every failure mode above prints what to check. When the update was started from the admin panel, that output is in updates/finalize.log, and updates/finalize-status.json names the step. Nothing is left half-applied: the schema step will not have seeded, and the extraction step either refuses before it writes anything or rolls itself back.

To bring the site straight back up on whatever code is currently on disk:

pnpm start

Before it writes anything, the extractor checks that it can write every directory and every existing file the release will replace. If it cannot, it refuses and names the paths with the errno that said so:

Update refused — nothing was changed. Refusing to extract the update: 37 of
1841 paths it must write cannot be written. NOTHING HAS BEEN CHANGED - the
application is exactly as it was. Blocked: /home/app/backend/dist/src/utils
(EACCES), ... The update runs as app (uid 1001). EACCES, EPERM and EROFS mean
ownership or permissions ...

A refusal is the good outcome: the tree is untouched and there is nothing to roll back. Read the codes:

• EACCES / EPERM — ownership. Almost always a subtree created by an account an earlier deploy ran as. ls -ln on a named path shows a bare uid with no name when that account no longer exists. chown -R it to the account the platform runs as, then download again.
• EROFS — the filesystem is mounted read-only.
• ENOTDIR / EISDIR — something of the wrong kind sits at that path: a file where the release needs a directory, or the reverse.

The extractor snapshots every file it overwrites into .update-backup-<timestamp> at the project root before writing, and restores the lot if any single file fails. It then re-reads every file it believes it wrote and compares it against what the archive declares — because the zip library reports success for writes it silently declined to make. Either check failing rolls the tree back and names the files.

There is nothing to undo by hand — the tree is the pre-update one. Fix the paths named in the message and download again.

Delete a leftover .update-backup-* directory only after you have confirmed the platform is healthy on the new version.

An update that reports success has been verified on disk. "Update downloaded and applied successfully. 1841 files updated and verified on disk" means every file the archive carries was read back off the filesystem at the size the archive declares. Older versions counted what the zip library claimed to have written, which is not the same thing: a subtree the platform could not write was reported as updated, and the first sign of it was Cannot find module at the next restart.

78 is EX_CONFIG, used by the platform for "this box is not configured to run this". It means one of two things: Redis is unreachable, or Node is on an unsupported major version. Supported majors are 22, 24 and 26 — the range is fixed by the prebuilt WebSocket binaries the backend ships, not by preference. Every PM2 config treats 78 as a stop rather than a crash, so the app will sit there stopped instead of looping.

Usually an incomplete dependency install, not a bad release. Re-run pnpm updator; the install step detects and repairs dangling links. If it recurs on every run, you almost certainly have two different pnpm versions on the machine fighting over the same node_modules — which -a pnpm will show them.

If the module it names is one of ours (utils/… rather than a package), the download did not fully land. Check that the path exists:

ls -ln backend/dist/src/utils/

An empty or stale directory whose owner shows as a bare uid is the case the refusal above now catches before writing. Fix the ownership and download the update again — the extraction will not report success over it.

A leftover from a previous version that the release deleted. Run pnpm clean:stale --check to see it, then pnpm clean:stale to remove it, and rebuild with pnpm build:frontend.