Troubleshooting

What you see. pm2 list shows backend as stopped seconds after pnpm start. The site loads but every API call fails, and the browser console is full of failed requests. pm2 logs backend ends in a boxed message rather than a stack trace.

What causes it. Exit 78 is raised by backend/preflight.ts before anything else loads, for one of three things:

• Wrong Node major. The supported range is 22 || 24 || 26, and it is not a preference. uWebSockets.js has no build step — it loads a prebuilt .node file chosen by your Node ABI, and the pinned version ships binaries only for those three. Node 20 dies with Cannot find module './uws_linux_x64_115.node', four frames deep, which reads like a corrupt install rather than a wrong runtime.
• An incomplete node_modules. The preflight resolves dotenv, module-alias, ioredis, sequelize, mysql2, bullmq and uWebSockets.js in one pass and lists everything missing at once, because an interrupted pnpm install otherwise surfaces them one restart at a time.
• Redis unreachable. Redis is a hard dependency, not a cache: sessions, rate limits, distributed locks, the BullMQ scheduler and cross-process settings invalidation all live in it. The in-memory fallback was removed, so an unreachable Redis stops the boot.

How to confirm. The message names the fault. It prints the running Node version and ABI against the supported list, or the missing packages, or the Redis host and port it tried and which variables chose them.

How to fix.

# Wrong Node major
curl -fsSL https://deb.nodesource.com/setup_26.x | sudo -E bash -
sudo apt-get install -y nodejs
pm2 kill && npm install -g pm2     # the daemon keeps whatever Node started it
pnpm rebuild -r
pnpm start

# Incomplete node_modules
pnpm reinstall

# Redis
sudo systemctl enable --now redis-server
redis-cli -h 127.0.0.1 -p 6379 ping    # expects: PONG

Step two of the Node fix is not optional. The PM2 daemon runs every app under the Node it was itself started with, so node -v can report 26 at your shell while PM2 is still handing the backend a Node 20.

If REDIS_PASSWORD is set, an authentication failure looks identical to a refused connection in that message — check the password before you go hunting for a network fault.

What you see. pm2 list shows backend as errored with a climbing restart count — a real crash loop, not the boxed exit-78 stop above. pm2 logs backend repeats:

Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found
       (required by .../node_modules/uWebSockets.js/uws_linux_x64_147.node)

What causes it. The operating system is too old. The pinned uWebSockets.js prebuilt binaries are linked against glibc 2.38, and Ubuntu 22.04 ships 2.35 — so the HTTP server the backend is built on cannot be loaded at all. Ubuntu 24.04 is the oldest release that works; Debian 12, RHEL 9 and Amazon Linux 2023 are below the line too.

Nothing catches this earlier. backend/preflight.ts checks the Node major and that uWebSockets.js resolves — it never loads the binary — so the install completes, the frontend builds, and the fault only appears when the API actually starts. Because it is not exit 78, PM2 restarts it until it gives up on unstable restarts.

How to confirm.

ldd --version | head -1     # 2.38 or higher is required
cat /etc/os-release         # Ubuntu 24.04 or newer

How to fix. Move to a supported OS — there is no in-place workaround. uWebSockets.js has no build-from-source path, and glibc is not a package you upgrade under a running distro. Either rebuild the box on Ubuntu 24.04 and restore from backup, or upgrade in place and rebuild the native modules afterwards:

pnpm stop
sudo do-release-upgrade          # snapshot the server first
pnpm rebuild -r                  # every native module, against the new libc
pnpm start

Downgrading uWebSockets.js is not an escape either: an older release moves the ABI range and breaks Node 26 instead.

What you see. In pm2 logs backend:

FATAL: failed to bind port 4000 (already in use?). Exiting so a duplicate process
cannot run cron jobs against the shared database.

The app then restarts, fails the same way, and PM2 eventually gives up with "too many unstable restarts". Exit 1 is not in stop_exit_codes, so unlike a boot misconfiguration this one does loop.

What causes it. Something outside this PM2 daemon is already holding the port: a backend started by hand, a second PM2 daemon under another user, or an orphan left over from an earlier crash loop that PM2 has already stopped managing. The refusal to keep running is deliberate — cron registration is not gated on winning the port, so a second backend that failed to bind would still schedule jobs against the same database.

How to confirm.

pm2 list
lsof -i :4000        # or: ss -lptn 'sport = :4000'
lsof -i :3000
ps -ef | grep -i "dist/index.js"

pnpm stop performs the same check for you and refuses to continue when it fails, printing the offending PIDs. It probes 3000 and the backend port for a listener and scans for backend processes PM2 does not own, then exits 1 rather than let an update run against a live database.

How to fix. Identify each process before killing it — this cannot tell your own tooling from a leftover, and killing a backend mid-withdrawal is its own damage.

ps -p <pid> -o pid,etime,args
kill <pid>
pnpm start

Two related traps. Port 4001 belongs to the cron app; if you run the threaded backend (pnpm start:thread) its worker threads also start at 4001, so the two shapes collide. And the backend and cron apps both pin their port in the config now, because PM2 passes the environment it was first started with to every app that does not override a variable — a stray NEXT_PUBLIC_BACKEND_PORT=4001 anywhere in that daemon's history used to move the API silently onto the scheduler's port.

What you see. The server comes back, pm2 list is empty, the site is down.

What causes it. The installer runs pm2 startup, which registers the boot hook, but it never runs pm2 save, which is what writes the process list that hook resurrects. The hook faithfully restores nothing.

How to confirm. pm2 resurrect brings back an empty list, or ~/.pm2/dump.pm2 does not exist.

How to fix. Start the platform, then save the list — once, as the user PM2 runs as.

pnpm start
pm2 save

Repeat pm2 save after any change to which apps run — switching to CRON_MODE=inline, or to pnpm start:backend on an API-only host.

What you see. nginx answers 502 for the whole site, or only for /api.

What causes it. 502 means nginx reached no upstream. Which path fails tells you which app is down: / is the frontend app on 3000, /api is the backend app on 4000.

The installer writes no nginx configuration at all — configure_nginx() only restarts the service. Every server block is yours, and there are two ways to get it wrong that do not look like configuration errors:

• No location /api block. Next.js does not proxy /api in production; those rewrites are development-only. Requests then fall through to Next on 3000 and come back as its HTML 404 page, so instead of a 502 you get JSON parse failures everywhere and a site that looks half-alive.
• localhost instead of 127.0.0.1. On a dual-stack box localhost can resolve to ::1 while the upstream listens on IPv4, giving a connection refused that reads as 502.

How to confirm. Bypass the proxy and ask the upstreams directly.

curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/
curl -sS http://127.0.0.1:4000/api/health | head -c 200
tail -n 50 /var/log/nginx/error.log

/api/health is unauthenticated and answers 200 while the backend can serve, 503 when it cannot — so it is also the right health check to point a load balancer at. Its body names the dependency that failed, which is usually the next thing you want to know. See Monitoring.

How to fix. Start whichever app is missing (pm2 list, then pnpm start), and make sure the server block has both locations, the ACME challenge above them, and a body limit that matches the platform:

client_max_body_size 10m;

location ^~ /.well-known/acme-challenge/ { root /var/www/html; }

location /api {
    proxy_pass http://127.0.0.1:4000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout 300s;
}

location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
}

nginx's default client_max_body_size is 1 MB, which fails KYC and avatar uploads with 413. The backend's own ceiling is 5 MB, so 10m at the proxy is the right side of it.

$remote_addr, not $proxy_add_x_forwarded_for. The second one appends to whatever header the client sent, so a request carrying a forged X-Forwarded-For arrives as <forged>, <real client>. $remote_addr discards the client's value and writes only the address nginx actually saw.

You do not need to set TRUST_PROXY for a proxy on this same machine. The backend honours forwarding headers automatically when the connection came from a loopback or private address, and ignores them for anything that reached port 4000 straight off the internet. Set TRUST_PROXY=true only when your load balancer is on a different host — and firewall port 4000 if you do, because that setting also makes a direct caller's header believable.

A 503 with a maintenance page is not this fault. pnpm stop leaves the maintenance server holding 3000 and the backend port, answering 503 with Retry-After: 300 — JSON for /api/*, HTML for everything else. pnpm start clears it.

What you see. Pages load, but prices, order books, tickers and order updates are frozen. The browser console shows the socket opening and closing, then a warning that it gave up.

What causes it. Every WebSocket in the platform lives under /api and connects to the page's own origin on port 443 — there is no separate socket port and no second hostname. If your location /api block does not carry proxy_http_version 1.1 plus the Upgrade and Connection headers, the upgrade is refused and the feed never starts.

Timeouts are the second cause. The server pings every 30 seconds and closes a socket after roughly one and a half intervals of silence; uWS itself idles a connection out at 120 seconds. An nginx proxy_read_timeout below that kills a healthy connection. It only has to happen once: the browser retries five times with exponential backoff up to 30 seconds, then stops permanently for that tab and only recovers when the tab regains focus or the network comes back.

How to confirm.

curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" \
     -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \
     https://yourdomain.com/api/exchange/ticker

A 101 Switching Protocols means the proxy is right. Anything else — 400, 426, 502 — is the proxy, not the app.

How to fix. Correct the /api block as shown under the 502 entry, then reload nginx. Three more things break sockets specifically:

• HTTPS is mandatory. In production the session cookies are set Secure with SameSite=None, so a browser on plain HTTP discards them — nobody can log in, and an unauthenticated upgrade is rejected by the auth gate. There is no ACME automation anywhere in the product; the certificate is yours to obtain and renew.
• Geo restrictions run on upgrades too, and answer 403. If sockets fail only for some countries, check Admin → System → Geo Restrictions.
• Binary options is the exception to same-origin. It builds wss://<host>:4000/... unless NEXT_PUBLIC_WS_URL is set, and nginx does not listen on 4000. Set NEXT_PUBLIC_WS_URL=wss://yourdomain.com in .env and run pnpm build:frontend — that variable is inlined into the browser bundle at build time, so editing .env alone changes nothing. Note that the market and ticker feeds read a different variable, NEXT_PUBLIC_WEBSOCKET_URL.

What you see. The trading page renders, the market list is populated, but the candle chart stays blank or shows a spinner that never resolves.

What causes it. Chart data has three separate sources, and which one is broken depends on the market type.

• Spot markets pull from your exchange provider through a gzipped cache. The backend reads the credentials dynamically, building the variable names from the provider chosen in the admin: APP_BINANCE_API_KEY / APP_BINANCE_API_SECRET for Binance, APP_KUCOIN_* plus APP_KUCOIN_API_PASSPHRASE for KuCoin, and so on. Because the names are built at runtime they never appear literally in the code, so a typo in .env produces no error beyond one log line: API credentials for <provider> are missing. After three failed attempts the loader backs off for 30 minutes, so fixing the keys and waiting looks like it did not work — restart the backend.
• Ecosystem and futures markets store their candles in ScyllaDB, not MySQL. If Scylla is unreachable, or SCYLLA_ENABLED=false, those endpoints answer 503 and the charts stay blank while the rest of the site is fine. The installer never installs Scylla and the SCYLLA_* variables are not in .env.example, so this is easy to miss on a fresh box.
• The cache is simply empty. Historical candles live in a data/chart/<BASE>/<QUOTE>/<interval>.json.gz tree relative to the backend's working directory, and it starts out empty.

How to confirm. In the admin, go to Finance → Trading Infrastructure → Exchange Providers, then open Chart Data from that page (it has no menu entry of its own). It reports per-market candle counts, file sizes, oldest and newest candle, and gap counts. Zero candles everywhere points at credentials; gaps in one market point at the cache. Check pm2 logs backend | grep -i "CHART\|EXCHANGE\|SCYLLA" alongside it.

How to fix. Correct the provider credentials in .env, restart the backend, verify them on the Exchange Providers screen, then build the cache from the Chart Data screen. Confirm the market itself is active — a disabled market renders its page and no data. Chart tooling needs the manage.exchange.chart permission, so a non-super-admin will find the buttons missing rather than failing.

What you see. Registrations complete but no verification mail arrives; withdrawals are approved with no notification. No error is shown anywhere in the UI.

What causes it. In order of how often it is the answer:

• MAIL_DISABLED is set. true, 1 or yes drops every outbound message before it reaches the queue, logging MAIL_DISABLED: dropping ... and reporting success upstream. It exists because each failed send is a real SMTP login, and a burst gets a Gmail account throttled with 454 4.7.0 Too many login attempts — which then blocks mail to genuine users. Somebody may have set it during testing.
• APP_EMAILER disagrees with itself. .env.example ships nodemailer-smtp, but the value the code falls back to when the variable is missing is nodemailer-service. Deleting or commenting the line therefore does not "use the default" — it switches provider. Valid values are local, nodemailer-service, nodemailer-smtp and nodemailer-sendgrid; anything else raises "Unsupported email provider".
• Port and encryption contradict each other. The connection is treated as implicitly secure when the port is 465 or APP_NODEMAILER_SMTP_ENCRYPTION is ssl — and ssl is the built-in default. Moving to port 587 without also setting the encryption to tls leaves the client negotiating SSL against a STARTTLS port, which hangs or resets. The SMTP host defaults to smtp.gmail.com when unset, too.
• The address cannot receive mail. Anything ending .invalid, .test, .example or .localhost is dropped before the queue. Seeded and fixture accounts use these.

How to confirm. Admin → System → Communication Tools → Notification Service. The Health and Queue tabs show what the queue is doing, and the Test tab sends a real message through the configured provider. The GET test route deliberately ignores any address you pass and sends to the calling admin's own account, so you cannot use it as an open relay. Then:

pm2 logs backend --lines 200 | grep -i "EMAIL\|MAIL_DISABLED"

How to fix. Correct .env and restart the backend — the mail settings are read from the environment, not from the settings table, so a restart is required. If the provider account has been throttled, no configuration change helps until the throttle lifts; leave MAIL_DISABLED=true on while you test anything that emits notifications.

What you see. The activation form returns "invalid" or a network message, and gated screens stay locked. A previously working install can also start refusing after being moved.

What causes it.

• Blocked outbound HTTPS. Validation talks to https://updates.mashdiv.com. Firewall it and activation cannot complete. An already-activated install keeps working for a 72-hour grace period, which is why this often surfaces three days after the firewall change.
• The licence file is bound to the machine. lic/<productId>.lic is AES-256-GCM encrypted with a key derived from the host's hardware fingerprint. Copying lic/ to a new server, restoring a backup onto different hardware, or cloning a VM produces a file that cannot decrypt — it does not fail over, it fails shut.
• lic/ is not writable. The directory is created next to the project root and must be writable by the user PM2 runs the backend as. The installer's blanket permission pass sets every file to 644 and chowns the tree to the app owner, so a licence written earlier as root can end up unwritable.

How to confirm. From the server itself:

curl -sS -o /dev/null -w '%{http_code}\n' https://updates.mashdiv.com
ls -l lic/
pm2 logs backend --lines 200 | grep -i license

The activation endpoint answers HTTP 200 with success: false and a message rather than an error status, so the reason is on screen — read it rather than the status code.

How to fix. Allow egress to updates.mashdiv.com on 443, make lic/ writable by the app user, then reactivate with the purchase code from the licence screen. Activation needs the create.license permission. After a server move, always reactivate on the new box; do not carry the old .lic across.

What you see. pnpm updator stops partway. The chain is stop → ensure-deps → updator:migrate → seed → build:frontend → start, joined by &&, so a failure at the migration step leaves the platform stopped in maintenance mode with nothing seeded and nothing rebuilt.

What causes it. There are no migration files. The schema is Sequelize auto-sync, driven by DB_SYNC (none, lazy — the default, always, force) against a fingerprint manifest at backend/.sync-hash. The migration step boots backend/dist/index.js directly, with CRON_MODE=off, on a port it has proven free (backend port + 1 through + 20), and waits for GET /api/settings to answer within 180 seconds. It fails when:

• The wait times out. A large install genuinely takes longer than the default deadline.
• No free port. Something is holding the whole candidate range, or a foreign backend answers on one of them — the script refuses any port that accepts a connection, because a port that answers instantly would report "schema is up to date" having migrated nothing.
• A foreign backend is running. pnpm stop exits 1 before the migration is ever reached when it finds a backend outside PM2 still listening. Do not work around it: a live backend writing to a database being migrated and seeded is how data gets damaged.
• Foreign-key ordering. An alter sync on MySQL rewrites foreign keys in an unstable order and can try to drop a constraint an earlier statement in the same pass removed: Can't DROP FOREIGN KEY 'user_ibfk_1'; check that it exists. This is retried three times and usually converges. If it still cannot finish, the server starts anyway and logs that the schema may be behind the models — a running server you can fix beats a boot loop.

How to confirm. Read the migration step's own output first, then:

pm2 list                                     # nothing but maintenance should be up
node scripts/updator-migrate.js --timeout=600000

For a stubborn constraint problem, find the duplicates the sync is tripping over:

SELECT TABLE_NAME, COLUMN_NAME, COUNT(*)
FROM information_schema.KEY_COLUMN_USAGE
WHERE TABLE_SCHEMA = DATABASE() AND REFERENCED_TABLE_NAME IS NOT NULL
GROUP BY TABLE_NAME, COLUMN_NAME, REFERENCED_TABLE_NAME
HAVING COUNT(*) > 1;

How to fix. Drop the duplicate constraints the query lists and run the step again. If the schema has drifted outside Sequelize — a hand-written ALTER, a restored dump — the fingerprint manifest no longer describes what is in the database; set DB_SYNC=always for one run to force a full alter sync, then put it back. Take a database backup first: this writes to production tables. Never reach for DB_SYNC=force, which drops and recreates every table and loses all data.

Once the migration succeeds, finish the update rather than restarting it from the top:

pnpm seed && pnpm build:frontend && pnpm start

What you see. RSS grows steadily in pm2 monit; the site stalls for seconds at a time; or a build dies with no message at all.

What causes it. Three different things wear the same symptom.

• Cron work growing the heap. Every scheduled job is a BullMQ worker that runs in the process that created it, on the same event loop that serves HTTP. A job that grows the heap stalls request serving through garbage collection — a mark-compact pause near the heap limit was measured at 1.9 seconds with the site answering nothing. That is exactly why the scheduler is a separate cron process with its own heap and a max_memory_restart of 2 GB; when cron misbehaves, PM2 recycles it alone while the web process keeps serving. The backend and frontend apps have no ceiling in the default config, so an unbounded leak there is not caught by PM2.
• The scheduler running twice. If the backend app predates the cron split it has no CRON_MODE and schedules inline, so starting cron beside it runs every job twice against the same rows — withdrawals included — at double the memory. pnpm start reconciles this for you by deleting an app whose scheduling role disagrees with the config it is about to start; starting PM2 by hand skips that.
• The build, not the server. Every build and migration script sets NODE_OPTIONS=--max-old-space-size=7780, so pnpm build:frontend will happily ask for 7.6 GB of heap. On a box near the installer's 4 GB minimum the kernel kills it and the output just stops.

How to confirm.

pm2 monit
pm2 list                                  # a rising restart count on cron = the 2G ceiling firing
free -m
dmesg | grep -i "killed process"          # the OOM killer, if the build vanished
grep -i CRON_MODE .env

How to fix. Restart the platform with pnpm restart so the scheduler layout is reconciled, and confirm exactly one process registers jobs. Give the box more RAM or add swap before a build if dmesg shows an OOM kill. Two things that look like fixes and are not:

• CRON_MODE=inline does not save memory. It removes the cron app entirely and puts every job back on the web process's event loop — the arrangement the split exists to undo.
• pnpm start:thread does not isolate cron either. Worker threads share one V8 heap limit with the process, and cron registration is main-thread-only, so a leaking job still exhausts the memory every thread depends on and the resulting pause stops all of them at once. Threading helps only when request handling itself is CPU-bound.

What you see. The Confirm dialog sits on "Confirming…" and the trade panel on "Processing"; or the order is accepted quickly but takes seven to ten seconds to show as filled. It gets worse with bots quoting the market, and worse the longer the account has been trading — but it happens with no bots running at all.

What causes it. Placement, the matching engine and every order list share one Node event loop, so anything that occupies it for a second delays everything else by a second. Several costs on those paths used to grow with the account or the market rather than staying fixed:

• The order-history list, which is the one to check first. The trade panel refreshes its history on every order update, and that list used to read the account's ENTIRE history on the market and convert seven high-precision columns of every row. Measured on a live install: 787 ms for 3,666 rows, against 20 ms for the open list over the same data — and under load, 2.3 s. The fill's own wallet steps could only run in the gaps between those reads, which is what turned a 264 ms placement into a seven-second fill. The history is now capped at the 250 most recent orders (?limit=, max 1000); the OPEN list is deliberately never capped, because a resting order holds money and must never be hidden.

• Reads sized by the market's depth. Writing one price level read the whole aggregated book to compute one number, and every placement, cancellation and fill then read the whole book again to draw the ladder. Both are proportional to how deep the market is, and a bot ladder is what makes it deep — so two makers did not merely add their own writes, they enlarged the read that every other order on that market performed.

• Reads sized by the account's history. A bot re-reads each of its working orders on every tick. That lookup scanned the account's entire order history, across every market it has ever traded, to return one row — so it got slower with every order the bot had ever placed, on the same Scylla session the manual placements were queued behind.

• Wallet row locks. Trading-bot orders are placed as the account that owns the bot, on that account's ECO wallets. Every placement, cancel and fill takes an exclusive lock on the same two wallet rows for the length of its transaction, so your own manual order queues behind the bots' — up to MySQL's innodb_lock_wait_timeout, 50 seconds by default. Running bots on a dedicated account is what separates them.

How to confirm. Ask the backend which step is slow rather than guessing:

# .env — then pnpm restart
SLOW_REQUEST_MS=1500

Place an order and read pm2 logs backend. Any request over the threshold prints one extra line naming its four slowest steps, for example:

[ECO_ORDER] SLOW (9120ms): POST /api/ecosystem/order — "Updating wallet balance" 8730ms, ...

That name is the answer: Updating wallet balance is wallet-row contention (bots on the same account, or a wallet job holding the row); Checking for self-matching orders or Adding order to matching engine is the Scylla side; before the first step (gates, auth, body) is the rate limiter or the auth chain, not trading at all.

How to fix. Update the platform — the read costs above are fixed there, and book frames are now collapsed per market rather than read once per order event (ECO_BOOK_FRAME_INTERVAL_MS). The update is not complete until both halves are rebuilt: the backend runs from backend/dist, and the trade panel is compiled into the browser bundle, so a git pull alone leaves the old behaviour running.

pnpm build          # backend/dist + frontend bundle
pnpm restart

A quick way to tell whether the new code is live: watch pm2 logs backend while the trade page is open. Repeated List user orders lines finishing in hundreds of milliseconds, or several of them per second, mean the old panel is still deployed.

If the slow step is the wallet, move the bots onto their own account so they stop competing with your manual orders for the same rows. Leave SLOW_REQUEST_MS set afterwards if you like; below the threshold it prints nothing.