Logs: locations, format and rotation

Almost every diagnostic instruction in this documentation ends with "read the log". This page says where they are, how to read a line, and — the part that eventually bites every install — that nothing rotates them for you.

The three logs that matter

A production install is three PM2 apps, and PM2 captures each one's output into two files:

~/.pm2/logs/backend-out.log      ~/.pm2/logs/backend-error.log
~/.pm2/logs/frontend-out.log     ~/.pm2/logs/frontend-error.log
~/.pm2/logs/cron-out.log         ~/.pm2/logs/cron-error.log

production.config.js sets no out_file or error_file, so these are PM2's defaults: $PM2_HOME/logs/<app>-out.log and <app>-error.log, where PM2_HOME defaults to .pm2 in the home directory of whoever the PM2 daemon runs as.

The frontend app writes whatever Next.js prints. The interesting one is backend, and cron is the same code with a different job.

Reading them

pm2 list                            # which apps exist, and under which daemon
pm2 logs                            # all apps, live, interleaved
pm2 logs backend --lines 200        # the last 200 lines of one app, then follow
pm2 logs backend --lines 200 --nostream   # print and exit — what you want in a script
pm2 logs --err                      # stderr only, across every app
pm2 flush                           # truncate every log file to zero

--lines is the flag to reach for when something has already happened. --nostream prints and returns rather than tailing, which is what you want when piping into grep.

Two things pm2 logs shows that are easy to miss.

Warnings go to stderr, not just errors. The logger writes info and success through console.log and both warn and error through console.warn / console.error — so pm2 logs --err and <app>-error.log contain warnings as well as failures. An -error.log with content in it is not by itself evidence of a crash.

A boxed message is a configuration refusal, not a stack trace. The platform exits 78 (EX_CONFIG) for the misconfigurations a restart cannot fix — an unsupported Node major, an incomplete node_modules, an unreachable Redis, or a maintenance server that could not bind a port. Every PM2 config lists 78 in stop_exit_codes, so PM2 stops the app with that message still on screen rather than restart-looping sixteen times and scrolling it away.

pm2 list                              # an app at `stopped` right after a start
pm2 logs backend --lines 50 --nostream

The box names the fault, prints the running Node version and ABI against the supported list, or the missing packages, or the Redis host and port it tried and which variables chose them. Read it before changing anything. An app at errored with a climbing restart count is a different situation — a real crash loop.

LOG_LEVEL in .env sets the floor: debug, info (the default), warn, error, silent. It is not in .env.example; add the line yourself. It is read once at boot, so changing it needs a restart.

What a line looks like

CRON 14:22:07 [WITHDRAW_2FA] ✓  Reconciled 12 spot withdrawals

Three parts before the message.

The role prefix. Four characters plus a space, stamped by backend/src/utils/process-role.ts (roleLogPrefix, measured by ROLE_PREFIX_WIDTH). It is derived from CRON_MODE:

This is how you tell a line written by the web backend from one written by the scheduler. It matters because pm2 logs with no app name interleaves every app into one stream, and both processes run the same code and print the same module names. It is a word rather than a colour on purpose: a line pasted out of a scrollback into a chat window still names its process after the escape codes are gone.

It is empty in the single-process arrangement, where there is nothing to disambiguate — so a log with no WEB/CRON column is telling you this install runs CRON_MODE=inline.

The timestamp is HH:MM:SS, UTC, with no date. For anything older than a day you need the file's own context, or ls -l on the file.

The module is the bracketed upper-case tag — [WITHDRAW_2FA], [EVM_DEPOSIT], [HEALTH], [AUDIT]. It is the first argument to every logger call and is what you grep for:

pm2 logs backend --lines 500 --nostream | grep -E "\[(WITHDRAW|DEPOSIT|EXCHANGE)\]"

The log files are not plain text — they carry colour. Only the role prefix and the role badge in the startup banner are TTY-aware (supportsAnsi in process-role.ts, the one thing NO_COLOR and FORCE_COLOR control; the prefix falls back to the plain word WEB/CRON). Everything else is coloured unconditionally: backend/src/utils/console/colors.ts is a static map of escape codes with no TTY check, and the logger wraps the timestamp in grey, the [MODULE] tag in cyan, and each status icon — plus the entire message on a warn or an error — in its own colour. So backend-out.log is full of \x1b[ sequences under PM2, and neither environment variable removes them.

pm2 logs renders them, so reading on the terminal is unaffected, and grepping still works because [WITHDRAW_2FA] appears literally between the codes. It matters when a file is opened somewhere that shows escapes as text — a ticket, an editor, a log shipper. Strip them on the way out:

sed -E 's/\x1b\[[0-9;]*m//g' ~/.pm2/logs/backend-out.log > /tmp/plain.log

Why the output can look reordered

Backend output does not always arrive in the order things happened, and this is by design rather than a fault.

Everything — ordinary lines, multi-line startup groups and animated tasks — passes through one queue (backend/src/utils/console/log-queue.ts) so that they cannot interleave and corrupt each other. Two consequences:

• Grouped output is buffered and printed atomically. A subsystem that opens a group (logger.group) collects its items and prints the whole block when the group ends. So a startup step's lines all appear at the moment it finished, after lines from work that began later.
• Live tasks hold everything else. A spinner-based task (backend/src/utils/console/live-console.ts) claims the terminal while it runs; ordinary logs queued during that window are held and flushed when it completes.

The practical rule: do not infer causality from adjacency in this log. Two lines next to each other are not necessarily two things that happened in that order. When ordering genuinely matters — reconstructing an incident — use the admin audit trail, whose rows carry their own createdAt, durationMs and a step-by-step narrative.

Nothing rotates these

pm2-logrotate is not installed, and no logrotate stanza ships with the product. On a busy install backend-out.log grows without limit until the partition fills.

Fix it once, before you need to:

pm2 install pm2-logrotate

pm2 set pm2-logrotate:max_size 50M          # rotate at 50 MB
pm2 set pm2-logrotate:retain 10             # keep 10 rotated files per stream
pm2 set pm2-logrotate:compress true         # gzip the rotated ones
pm2 set pm2-logrotate:rotateInterval '0 0 * * *'   # also rotate nightly

pm2 save

/home/YOUR_APP_USER/.pm2/logs/*.log {
    daily
    rotate 14
    size 50M
    missingok
    notifempty
    compress
    delaycompress
    copytruncate
}

copytruncate is not optional in the second form: PM2 holds the file handle open, so a plain rename leaves it writing to an unlinked inode and the new file stays empty.

Either way, keep an eye on the total:

du -sh ~/.pm2/logs
df -h

pm2 flush truncates everything immediately when you need space back now. It is destructive — whatever explains the current incident goes with it, so copy the last few thousand lines out first.

logModule / logTitle and audit are two different destinations

Operators conflate these constantly, so it is worth being exact.

Every API route can declare logModule and logTitle in its metadata. Those name the console line — the [MODULE] tag above and the human title of the operation. They are about watching the platform work.

The admin audit trail (/admin/system/audit, table admin_audit_log) is a different destination with a different question: which operator changed what, and why. A row is written when all of these hold:

• the method is POST, PUT, PATCH or DELETE — reads are never recorded;
• the path starts with /api/admin/ — customer trades, deposits and transfers are not on this trail, they have the wallet audit log and the transaction ledger;
• the route does not declare audit: false.

audit: false is the opt-out, and it exists because a large minority of admin POSTs change nothing: version checks, connectivity probes, credential tests, dry-runs, and queries that are POSTs only because their filter does not fit in a query string. Those are the calls a screen fires automatically on mount. Before the flag existed they were 89 of the table's 94 rows.

So a route with logModule set but audit: false still writes a console line and deliberately writes no audit row. And the trail records the operator's reason, requestId, ip, durationMs and the handler's step narrative — which the console line does not.

Every other log on the box

The Hummingbot files are the only application log the platform writes to disk itself, and the only one that rotates: at 5 MB the current file is renamed <instanceId>.log.1 and a fresh one started, keeping exactly one previous generation. Lines are stamped with a full ISO timestamp, and lines the supervisor wrote rather than the bot are marked [supervisor]. You can read the tail from the admin instead of the disk, with view.hb.instance:

It takes lines (up to 2000, default 300) and a since byte offset for incremental polling. It reads the file rather than the supervisor's memory, so the last output of a bot that crashed survives a backend restart — which is exactly when you want it.

Before you attach a log to a ticket

Backend logs are operational, not sanitised. They routinely carry things you should not paste into a public tracker.

Redact, or do not send:

• Customer identifiers — email addresses, user ids, phone numbers. Deposit, withdrawal and notification lines all name them.
• Wallet addresses and transaction hashes. These are public on-chain, but together with a timestamp they deanonymise a specific customer's holdings.
• RPC endpoint URLs. Provider keys are embedded in the path — https://polygon-mainnet.infura.io/ws/v3/<project-id> — and the RPC probe prints the endpoint in its failure messages. Anyone with that URL can spend your quota.
• Session and request identifiers. A requestId is safe; anything that looks like a session or bearer token is not.
• Your licence purchase code, and the contents of .env in any form. The backend does not print these, but the installer log and any command you ran by hand may.
• Internal hostnames and private IPs, if your deployment topology is not public.

Safe to send, and usually the whole answer: the boxed exit-78 message, the [MODULE] tag and the message text of the failing line, the exception class and stack frames, pm2 list output, node -v, ldd --version, and the Platform Health card's service list.

A practical filter for a first pass — check the result by eye before sending, it is a starting point and not a guarantee:

pm2 logs backend --lines 500 --nostream \
  | sed -E 's/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/<email>/g' \
  | sed -E 's#(https?|wss?)://[^ ]+#<url>#g' \
  | sed -E 's/0x[a-fA-F0-9]{40,}/<address>/g' \
  > /tmp/redacted.log

Related

• Troubleshooting — the symptoms these logs explain.
• The health screen and what each probe means — which probe's message you are chasing.
• Processes and ports — which app writes which file and what stops when it is down.
• The admin panel — the audit trail screen, and what it records that the console does not.