Updating
How to apply a release safely — what to back up first, what the update chain actually does, and how to confirm the platform came back up on the new code.
An update is two separate actions, and treating them as one is the usual cause of "I updated and nothing changed".
The download fetches a ZIP from the licence service and extracts it over the project root. It replaces files on disk. It does not touch the database, does not reinstall dependencies, does not rebuild the frontend and does not restart anything — the running processes carry on executing the code they loaded at boot.
The finalise chain is pnpm updator, run in a shell on the server. That is
what makes the new files take effect: dependencies, schema, seed data, frontend
build, restart.
Pressing Update Now in the admin panel only performs the download. Until you
run pnpm updator on the server, the platform is serving the old code from a
tree that has already been replaced underneath it. Do not leave it in that state
overnight.
Back up before you start
The platform ships a MySQL backup and nothing else. Everything in the second list below has to be your own copy.
The screen is at /admin/system/database/backup. It has no menu entry anywhere —
type the URL. Dumps land in backup/ under the project root, named for the
timestamp (2026_08_03_14_21_09.sql), and the restore screen reads from that same
directory. It covers MySQL only.
Not covered by anything in the product:
.env— the single most expensive file on the box. See the warning below.frontend/public/uploads/— KYC documents, dispute evidence, legal files, avatars. It is gitignored and it is in no release package, so nothing will ever restore it for you.lic/— the encrypted, machine-bound licence files.backend/ecosystem/wallets/— master wallet exports.- Redis — sessions, CSRF tokens, rate limits, locks and job queues. Cheap to lose, but the backend will not boot at all without a reachable server.
- ScyllaDB — the keyspaces named by
SCYLLA_KEYSPACEandSCYLLA_FUTURES_KEYSPACE. Order books, candles, the trade tape and the open-order index for ecosystem and futures trading live there, not in MySQL. The installer never installs Scylla and the backup screen does not know it exists.
ENCRYPTED_ENCRYPTION_KEY and ENCRYPTION_KEY_PASSPHRASE are what decrypt every
custodial wallet private key the platform holds. They exist only in your .env
and they are not in .env.example, so a fresh copy of that file will not contain
them. Restoring a database without them leaves the wallets present and permanently
unspendable. Copy .env somewhere off the box before every update.
What survives an update
| Path | What happens to it |
|---|---|
.env |
Never shipped in a release package — yours survives untouched. Releases do add new keys, so compare it against .env.example afterwards. |
frontend/public/uploads/ |
Untouched. Not in the package, and the extractor never deletes. |
lic/ |
Untouched. Reactivation is only needed if the machine fingerprint changes. |
| The database | Not in the package. Changed in place by the migration and seed steps of the chain. |
backend/dist/ |
Replaced. This directory is the production backend — it ships pre-built, which is why the update chain never runs build:backend. |
node_modules/ |
Rebuilt as far as necessary by the install step. |
frontend/.next/ |
Rebuilt by pnpm build:frontend. Stale until that step runs. |
Extraction only ever writes. A file a release removed is still sitting on your
disk afterwards, and that is not always harmless: a deleted page.tsx under
frontend/app is still a route and can break the next build, and a deleted module
that shares a name with a directory beside it wins module resolution over the
directory that replaced it. Clear those out after the update:
pnpm clean:stale --check # preview what would be deleted
pnpm clean:stale # delete itThe update screen
Admin → System → System Updates, at /admin/system/update.
The page itself is gated on access.system.update, but every button on it calls a
route that requires create.license. A role granted only the first sees the
screen fully rendered and gets a permission error from the check and from the
download. Grant both, or neither.
Without a verified licence the page replaces itself with a purchase-code activation form — there is no update UI at all until activation succeeds. Once licensed, three tabs:
Licence status, current version and whether an update is waiting, plus links to support, documentation and the extension manager.
The check, the changelog for the pending version, and the button that downloads it.
Every published version for this product, selectable from a list. Hidden entirely if the notes could not be fetched.
Updates are applied one version at a time. The licence service returns the full list of pending versions and the panel offers only the next one; after a successful download it re-checks and offers the one after that. Jumping straight to the newest version is not available, and a queue of more than one shows a "Sequential Updates Required" banner with the whole path spelled out.
Extensions, blockchains and exchange providers use the same download route with a
type field, driven from /admin/system/extension and the blockchain detail
pages. The admin dashboard's update widget calls a batch check across every
installed product at once.
If the licence service is unreachable, the check is caught and answered with
You have the latest version of the product. — the same message as a genuine
up-to-date result. A missing or unreadable licence file answers No purchase code found under the same green heading. Read the message line, not the heading.
Where release notes come from
The changelog shown on this screen is not stored on your server. The browser calls a backend proxy, which fetches a JSON bundle from the documentation site:
That request fails gracefully and silently. If the docs host is unreachable from your server the proxy returns an empty bundle, the Changelog tab disappears and the Updates tab reads "No changelog available for this version" — a network symptom, not a release that shipped without notes. The panel then falls back to whatever changelog string the update-check response happened to carry.
The same notes are published at /docs/releases, so you can read what a version changes before you take the site down for it.
Applying an update
-
Take the backups. Database dump, plus your own copy of
.env,frontend/public/uploads/andlic/. -
Download the release from Admin → System → System Updates. Repeat for each pending version if more than one is queued, or do the whole estate from the server instead:
pnpm update-all --dry-run # list what would be applied pnpm update-all # download everything, then finalise once -
Run the chain from the project root. This is the step that takes the site down and brings it back:
pnpm updator -
Verify, using the checklist further down. The site is live again as soon as the last step of the chain finishes.
pnpm update-all finishes by running the same chain itself, so do not run
pnpm updator after it unless you passed --no-finalize.
What the chain does, link by link
pnpm updator is pnpm stop, then dependencies, then schema, then seed, then
frontend build, then pnpm start. Each link stops the chain if it fails, which is
deliberate: every one of them exists to prevent the next one running against a
half-updated install.
Stops and removes the backend, frontend and cron apps from PM2, then
proves the frontend port (3000) and the backend port
(NEXT_PUBLIC_BACKEND_PORT, default 4000) are actually free and scans for a
backend process PM2 does not own. If either check fails it exits non-zero and the
update stops before touching anything — a second backend on the same database
during a schema migration is far worse than a failed update.
It then puts the maintenance server on those two ports: 503 JSON for anything
under /api/, a 503 HTML page for everything else, both with Retry-After: 300.
Port 4001 is deliberately left unbound so the migration step can use it.
pnpm stop:all removes the maintenance server too, which closes the ports
entirely instead of serving a page — use it only when you want the site to be
visibly down.
Runs pnpm install -r --no-frozen-lockfile, then verifies that every declared
dependency and every executable the rest of the chain shells out to actually
resolves, and repairs the tree if they do not.
This is not belt-and-braces. When a release changes dependency resolution, pnpm
rebuilds the root store but leaves backend/node_modules in place, and its links
now point at store paths that were just replaced. A dangling link is not an empty
directory — it looks present and fails at require() time, so the symptom is the
backend dying on Cannot find module 'bullmq', one package per restart, with
nothing saying "your install is incomplete".
If it cannot produce a working tree it stops the update rather than let a schema migration run against a backend that cannot boot. The message tells you which of the three usual causes to check: a full disk, something still running and holding files open, or a damaged pnpm store.
Sequelize applies the schema during boot, so the migration step is a boot. It runs
backend/dist/index.js directly — not under PM2, so nothing can restart it
behind your back — with the scheduler switched off and boot restricted to the
schema phase, on the first free port from NEXT_PUBLIC_BACKEND_PORT + 1 upward
(4001–4020 on a default install).
It then waits for GET /api/settings to answer, which is the observable that says
initialisation finished, and kills the process. The cap is 180 seconds. A very
large schema legitimately needs longer:
node scripts/updator-migrate.js --timeout=600000If it exits early or never becomes ready, nothing is seeded and the site stays on the maintenance page — which is the correct place to stop, because the alternative is seeders writing through models the database does not match yet.
Schema behaviour is controlled by DB_SYNC in .env: lazy (the default) syncs
only what changed, using the fingerprint in backend/.sync-hash; none
authenticates without altering anything; always forces a full ALTER sync, which
is the escape hatch when the schema has drifted outside Sequelize. force drops
and recreates every table and destroys all data — never set it on a live install.
Writes the rows the code expects to exist: permissions, roles, deposit gateways, notification templates. It runs through the new models, which is exactly why it comes after the schema step and never before it.
Rebuilds frontend/.next. This is not optional and it is not skippable to save
time.
Every NEXT_PUBLIC_* value is inlined into the browser bundle when the frontend is
built. NEXT_PUBLIC_SITE_URL is the origin every client-side API call uses and the
hostname in the next/image allowlist. An un-rebuilt frontend serves the previous
release's pages against the new backend, and if you also changed the domain, the
browser calls the old one and images 400.
The backend needs no equivalent step: backend/dist ships pre-built inside the
release package.
Clears maintenance mode and starts the three PM2 apps from production.config.js:
backend on NEXT_PUBLIC_BACKEND_PORT, frontend on 3000 (hardcoded — it is not
moved by .env), and cron on 4001 with CRON_MODE=only. Nothing should ever
connect to 4001; the port exists only so the scheduler process does not collide
with the backend.
Setting CRON_MODE=inline in .env drops the cron app entirely and schedules
jobs inside the backend process instead, so you will see two apps rather than
three.
Updating every product at once
pnpm update-all enumerates the core product plus every installed extension,
blockchain and exchange provider, downloads each pending version in order, and
then finalises once — graceful-stop, dependencies, schema, seed, frontend
build, restart — instead of taking the site down per addon.
pnpm update-all --dry-run # report only, downloads nothing
pnpm update-all --no-finalize # download everything, leave the site running old code
pnpm update-all # download and finaliseIts graceful stop waits for in-flight withdrawals to quiesce before entering
maintenance mode, up to GRACEFUL_STOP_TIMEOUT_MS (default 120 seconds), rather
than killing the backend mid-broadcast.
update-all reads backend/dist without booting it, and falls back to the source
tree if dist will not even load. That matters when a build crashes on startup:
the admin panel's Update button needs a running site, so a backend that dies at
boot locks you out of the only UI that could replace it. This script can still
fetch its own replacement.
Verifying the update
pm2 list—backend,frontendandcronall online (two apps ifCRON_MODE=inline). A backend restarting in a loop is the one thing you must not walk away from:pm2 logs backend.- Load the site over HTTPS and sign in. Cookies are issued
SecureandSameSite=Nonein production, so a login that fails on every browser usually means TLS terminated somewhere it should not have. - Admin → System → System Updates — the version badge should read the version you just applied.
pnpm clean:stale --check, thenpnpm clean:stale, to remove files the release deleted.- Diff
.envagainst.env.examplefor keys the release added. pnpm check:permissionreports permission drift and lists screens waiting on a grant. New permission gates ship strict: a newly gated screen is reachable by Super Admin only until you grant its key per role under Admin → Users → Roles & Permissions.pm2 save, so a server reboot resurrects the apps you just started. The installer runspm2 startupbut neverpm2 save, so on many boxes a reboot brings PM2 back with an empty list.
When it goes wrong
The chain stopped at one of its links. Read the last error it printed — every failure mode above prints what to check. Nothing is left half-applied: the schema step will not have seeded, and the extraction step rolls itself back.
To bring the site straight back up on whatever code is currently on disk:
pnpm startThe extractor snapshots every file it overwrites into .update-backup-<timestamp>
at the project root before writing, and restores the lot if any single file fails.
There is nothing to undo by hand — the tree is the pre-update one. Fix the cause
(almost always a full disk or a permissions problem on the project tree) and
download again.
Delete a leftover .update-backup-* directory only after you have confirmed the
platform is healthy on the new version.
78 is EX_CONFIG, used by the platform for "this box is not configured to run
this". It means one of two things: Redis is unreachable, or Node is on an
unsupported major version. Supported majors are 22, 24 and 26 — the range is fixed
by the prebuilt WebSocket binaries the backend ships, not by preference. Every PM2
config treats 78 as a stop rather than a crash, so the app will sit there stopped
instead of looping.
An incomplete dependency install, not a bad release. Re-run pnpm updator; the
install step detects and repairs dangling links. If it recurs on every run, you
almost certainly have two different pnpm versions on the machine fighting over the
same node_modules — which -a pnpm will show them.
A leftover from a previous version that the release deleted. Run
pnpm clean:stale --check to see it, then pnpm clean:stale to remove it, and
rebuild with pnpm build:frontend.