Updating

How to apply a release safely — what to back up first, what the update chain actually does, and how to confirm the platform came back up on the new code.

11 min readUpdated 3 August 2026

An update is two separate actions, and treating them as one is the usual cause of "I updated and nothing changed".

The download fetches a ZIP from the licence service and extracts it over the project root. It replaces files on disk. It does not touch the database, does not reinstall dependencies, does not rebuild the frontend and does not restart anything — the running processes carry on executing the code they loaded at boot.

The finalise chain is pnpm updator, run in a shell on the server. That is what makes the new files take effect: dependencies, schema, seed data, frontend build, restart.

Pressing Update Now in the admin panel only performs the download. Until you run pnpm updator on the server, the platform is serving the old code from a tree that has already been replaced underneath it. Do not leave it in that state overnight.

Back up before you start

The platform ships a MySQL backup and nothing else. Everything in the second list below has to be your own copy.

Dumps the MySQL database with mysqldump

The screen is at /admin/system/database/backup. It has no menu entry anywhere — type the URL. Dumps land in backup/ under the project root, named for the timestamp (2026_08_03_14_21_09.sql), and the restore screen reads from that same directory. It covers MySQL only.

Not covered by anything in the product:

  • .env — the single most expensive file on the box. See the warning below.
  • frontend/public/uploads/ — KYC documents, dispute evidence, legal files, avatars. It is gitignored and it is in no release package, so nothing will ever restore it for you.
  • lic/ — the encrypted, machine-bound licence files.
  • backend/ecosystem/wallets/ — master wallet exports.
  • Redis — sessions, CSRF tokens, rate limits, locks and job queues. Cheap to lose, but the backend will not boot at all without a reachable server.
  • ScyllaDB — the keyspaces named by SCYLLA_KEYSPACE and SCYLLA_FUTURES_KEYSPACE. Order books, candles, the trade tape and the open-order index for ecosystem and futures trading live there, not in MySQL. The installer never installs Scylla and the backup screen does not know it exists.

ENCRYPTED_ENCRYPTION_KEY and ENCRYPTION_KEY_PASSPHRASE are what decrypt every custodial wallet private key the platform holds. They exist only in your .env and they are not in .env.example, so a fresh copy of that file will not contain them. Restoring a database without them leaves the wallets present and permanently unspendable. Copy .env somewhere off the box before every update.

What survives an update

Path What happens to it
.env Never shipped in a release package — yours survives untouched. Releases do add new keys, so compare it against .env.example afterwards.
frontend/public/uploads/ Untouched. Not in the package, and the extractor never deletes.
lic/ Untouched. Reactivation is only needed if the machine fingerprint changes.
The database Not in the package. Changed in place by the migration and seed steps of the chain.
backend/dist/ Replaced. This directory is the production backend — it ships pre-built, which is why the update chain never runs build:backend.
node_modules/ Rebuilt as far as necessary by the install step.
frontend/.next/ Rebuilt by pnpm build:frontend. Stale until that step runs.

Extraction only ever writes. A file a release removed is still sitting on your disk afterwards, and that is not always harmless: a deleted page.tsx under frontend/app is still a route and can break the next build, and a deleted module that shares a name with a directory beside it wins module resolution over the directory that replaced it. Clear those out after the update:

pnpm clean:stale --check   # preview what would be deleted
pnpm clean:stale           # delete it

The update screen

Admin → System → System Updates, at /admin/system/update.

The page itself is gated on access.system.update, but every button on it calls a route that requires create.license. A role granted only the first sees the screen fully rendered and gets a permission error from the check and from the download. Grant both, or neither.

Without a verified licence the page replaces itself with a purchase-code activation form — there is no update UI at all until activation succeeds. Once licensed, three tabs:

Licence status, current version and whether an update is waiting, plus links to support, documentation and the extension manager.

The check, the changelog for the pending version, and the button that downloads it.

Every published version for this product, selectable from a list. Hidden entirely if the notes could not be fetched.

Updates are applied one version at a time. The licence service returns the full list of pending versions and the panel offers only the next one; after a successful download it re-checks and offers the one after that. Jumping straight to the newest version is not available, and a queue of more than one shows a "Sequential Updates Required" banner with the whole path spelled out.

Asks the licence service which versions are pending
Downloads one version and extracts it over the project root

Extensions, blockchains and exchange providers use the same download route with a type field, driven from /admin/system/extension and the blockchain detail pages. The admin dashboard's update widget calls a batch check across every installed product at once.

If the licence service is unreachable, the check is caught and answered with You have the latest version of the product. — the same message as a genuine up-to-date result. A missing or unreadable licence file answers No purchase code found under the same green heading. Read the message line, not the heading.

Where release notes come from

The changelog shown on this screen is not stored on your server. The browser calls a backend proxy, which fetches a JSON bundle from the documentation site:

Proxies the published patch notes for every product

That request fails gracefully and silently. If the docs host is unreachable from your server the proxy returns an empty bundle, the Changelog tab disappears and the Updates tab reads "No changelog available for this version" — a network symptom, not a release that shipped without notes. The panel then falls back to whatever changelog string the update-check response happened to carry.

The same notes are published at /docs/releases, so you can read what a version changes before you take the site down for it.

Applying an update

  1. Take the backups. Database dump, plus your own copy of .env, frontend/public/uploads/ and lic/.

  2. Download the release from Admin → System → System Updates. Repeat for each pending version if more than one is queued, or do the whole estate from the server instead:

    pnpm update-all --dry-run   # list what would be applied
    pnpm update-all             # download everything, then finalise once
  3. Run the chain from the project root. This is the step that takes the site down and brings it back:

    pnpm updator
  4. Verify, using the checklist further down. The site is live again as soon as the last step of the chain finishes.

pnpm update-all finishes by running the same chain itself, so do not run pnpm updator after it unless you passed --no-finalize.

pnpm updator is pnpm stop, then dependencies, then schema, then seed, then frontend build, then pnpm start. Each link stops the chain if it fails, which is deliberate: every one of them exists to prevent the next one running against a half-updated install.

Stops and removes the backend, frontend and cron apps from PM2, then proves the frontend port (3000) and the backend port (NEXT_PUBLIC_BACKEND_PORT, default 4000) are actually free and scans for a backend process PM2 does not own. If either check fails it exits non-zero and the update stops before touching anything — a second backend on the same database during a schema migration is far worse than a failed update.

It then puts the maintenance server on those two ports: 503 JSON for anything under /api/, a 503 HTML page for everything else, both with Retry-After: 300. Port 4001 is deliberately left unbound so the migration step can use it.

pnpm stop:all removes the maintenance server too, which closes the ports entirely instead of serving a page — use it only when you want the site to be visibly down.

Runs pnpm install -r --no-frozen-lockfile, then verifies that every declared dependency and every executable the rest of the chain shells out to actually resolves, and repairs the tree if they do not.

This is not belt-and-braces. When a release changes dependency resolution, pnpm rebuilds the root store but leaves backend/node_modules in place, and its links now point at store paths that were just replaced. A dangling link is not an empty directory — it looks present and fails at require() time, so the symptom is the backend dying on Cannot find module 'bullmq', one package per restart, with nothing saying "your install is incomplete".

If it cannot produce a working tree it stops the update rather than let a schema migration run against a backend that cannot boot. The message tells you which of the three usual causes to check: a full disk, something still running and holding files open, or a damaged pnpm store.

Sequelize applies the schema during boot, so the migration step is a boot. It runs backend/dist/index.js directly — not under PM2, so nothing can restart it behind your back — with the scheduler switched off and boot restricted to the schema phase, on the first free port from NEXT_PUBLIC_BACKEND_PORT + 1 upward (4001–4020 on a default install).

It then waits for GET /api/settings to answer, which is the observable that says initialisation finished, and kills the process. The cap is 180 seconds. A very large schema legitimately needs longer:

node scripts/updator-migrate.js --timeout=600000

If it exits early or never becomes ready, nothing is seeded and the site stays on the maintenance page — which is the correct place to stop, because the alternative is seeders writing through models the database does not match yet.

Schema behaviour is controlled by DB_SYNC in .env: lazy (the default) syncs only what changed, using the fingerprint in backend/.sync-hash; none authenticates without altering anything; always forces a full ALTER sync, which is the escape hatch when the schema has drifted outside Sequelize. force drops and recreates every table and destroys all data — never set it on a live install.

Writes the rows the code expects to exist: permissions, roles, deposit gateways, notification templates. It runs through the new models, which is exactly why it comes after the schema step and never before it.

Rebuilds frontend/.next. This is not optional and it is not skippable to save time.

Every NEXT_PUBLIC_* value is inlined into the browser bundle when the frontend is built. NEXT_PUBLIC_SITE_URL is the origin every client-side API call uses and the hostname in the next/image allowlist. An un-rebuilt frontend serves the previous release's pages against the new backend, and if you also changed the domain, the browser calls the old one and images 400.

The backend needs no equivalent step: backend/dist ships pre-built inside the release package.

Clears maintenance mode and starts the three PM2 apps from production.config.js: backend on NEXT_PUBLIC_BACKEND_PORT, frontend on 3000 (hardcoded — it is not moved by .env), and cron on 4001 with CRON_MODE=only. Nothing should ever connect to 4001; the port exists only so the scheduler process does not collide with the backend.

Setting CRON_MODE=inline in .env drops the cron app entirely and schedules jobs inside the backend process instead, so you will see two apps rather than three.

Updating every product at once

pnpm update-all enumerates the core product plus every installed extension, blockchain and exchange provider, downloads each pending version in order, and then finalises oncegraceful-stop, dependencies, schema, seed, frontend build, restart — instead of taking the site down per addon.

pnpm update-all --dry-run       # report only, downloads nothing
pnpm update-all --no-finalize   # download everything, leave the site running old code
pnpm update-all                 # download and finalise

Its graceful stop waits for in-flight withdrawals to quiesce before entering maintenance mode, up to GRACEFUL_STOP_TIMEOUT_MS (default 120 seconds), rather than killing the backend mid-broadcast.

update-all reads backend/dist without booting it, and falls back to the source tree if dist will not even load. That matters when a build crashes on startup: the admin panel's Update button needs a running site, so a backend that dies at boot locks you out of the only UI that could replace it. This script can still fetch its own replacement.

Verifying the update

  • pm2 listbackend, frontend and cron all online (two apps if CRON_MODE=inline). A backend restarting in a loop is the one thing you must not walk away from: pm2 logs backend.
  • Load the site over HTTPS and sign in. Cookies are issued Secure and SameSite=None in production, so a login that fails on every browser usually means TLS terminated somewhere it should not have.
  • Admin → System → System Updates — the version badge should read the version you just applied.
  • pnpm clean:stale --check, then pnpm clean:stale, to remove files the release deleted.
  • Diff .env against .env.example for keys the release added.
  • pnpm check:permission reports permission drift and lists screens waiting on a grant. New permission gates ship strict: a newly gated screen is reachable by Super Admin only until you grant its key per role under Admin → Users → Roles & Permissions.
  • pm2 save, so a server reboot resurrects the apps you just started. The installer runs pm2 startup but never pm2 save, so on many boxes a reboot brings PM2 back with an empty list.

When it goes wrong

The chain stopped at one of its links. Read the last error it printed — every failure mode above prints what to check. Nothing is left half-applied: the schema step will not have seeded, and the extraction step rolls itself back.

To bring the site straight back up on whatever code is currently on disk:

pnpm start

The extractor snapshots every file it overwrites into .update-backup-<timestamp> at the project root before writing, and restores the lot if any single file fails. There is nothing to undo by hand — the tree is the pre-update one. Fix the cause (almost always a full disk or a permissions problem on the project tree) and download again.

Delete a leftover .update-backup-* directory only after you have confirmed the platform is healthy on the new version.

78 is EX_CONFIG, used by the platform for "this box is not configured to run this". It means one of two things: Redis is unreachable, or Node is on an unsupported major version. Supported majors are 22, 24 and 26 — the range is fixed by the prebuilt WebSocket binaries the backend ships, not by preference. Every PM2 config treats 78 as a stop rather than a crash, so the app will sit there stopped instead of looping.

An incomplete dependency install, not a bad release. Re-run pnpm updator; the install step detects and repairs dangling links. If it recurs on every run, you almost certainly have two different pnpm versions on the machine fighting over the same node_moduleswhich -a pnpm will show them.

A leftover from a previous version that the release deleted. Run pnpm clean:stale --check to see it, then pnpm clean:stale to remove it, and rebuild with pnpm build:frontend.