When money does not move

This page is the money half of triage. The infrastructure half — a backend that will not boot, a dead port, GLIBC, mail, licences — is Troubleshooting.

Everything here assumes the platform is otherwise up: pages load, the API answers, customers can sign in. When money still does not move, the cause is almost always one of five things, and there is a first check for each.

Before anything else: is the scheduler alive

The scheduler is its own PM2 process. Under the default split the web process registers no cron jobs at all, so a stopped cron app leaves a deployment where every page loads, every health tile is green and no withdrawal is ever processed. Nothing else in the product makes that obvious.

Open System → System Monitoring → Scheduled Tasks (/admin/system/cron). The page keeps three signals separate on purpose:

• The scheduler heartbeat. Beaten every 15 seconds with a 90-second TTL, and read by GET /api/admin/system/cron/scheduler. It reports running, missing, stale, duplicate or unknown. unknown is not the same as "no scheduler" — it means the heartbeat could not be read. duplicate is worse than missing: two processes scheduling means money jobs run twice over the same rows.
• The job registry, with each job's run state and any sticky refusal. The registry exists identically in all three processes and its run bookkeeping is hydrated from a Redis snapshot with a 24-hour TTL — so a scheduler that died an hour ago still produces a full page of jobs with plausible "last run" times. Only the heartbeat tells you whether any of them will run again.
• A live log stream, which is where a job's own account of a run appears.

Viewing needs view.cron; triggering a job by hand needs manage.cron.

The three jobs that matter on this page:

A withdrawal sits at PENDING forever

What it means. The customer's wallet was debited when they submitted the request. The money has not left the platform. Nothing is lost yet.

A withdrawal sits at PROCESSING

What it means. Someone or something called exchange.withdraw(). The money may already have gone. This is the state that needs a human, and the one nothing else in the product will tell you about.

PROCESSING is written by an atomic claim: the approve endpoint updates the row PENDING → PROCESSING in a single compare-and-set, and only the request that wins that update is allowed to call the exchange. A concurrent request, a retry or a crashed-and-restarted process loses the claim and gets a 409 "Withdrawal is already being processed". That is the guard against a double on-chain send, and it is why re-pressing Approve is never the fix.

reconcileSpotWithdrawals runs every 5 minutes over PROCESSING + WITHDRAW rows on SPOT wallets, skipping anything younger than 5 minutes because the endpoint may still be mid-flight. What it does depends on one field:

referenceId is set — the exchange accepted the payout. The job converges to whatever the exchange reports: completed/ok → COMPLETED, cancelled/canceled → CANCELLED, failed → FAILED. A vendor status of pending deliberately leaves the row PROCESSING rather than demoting it, because demoting it would push the row out of the reconciler's own coverage.

The payout is not in the page the exchange returned — that is not evidence it never happened. fetchWithdrawals is paginated and lookback-limited on most venues. The job refuses to auto-refund; after 24 hours it flags the row for review with the reason recorded, and backs off so it does not re-scan it every five minutes.

referenceId is empty — the process crashed somewhere around the exchange call, and the payout may or may not have been sent. The job only refunds on positive evidence of absence, and only after 24 hours.

What auto-refunds. When the exchange returns a terminal failure — FAILED or CANCELLED — the refund and the status flip happen in the same database transaction, refund first. The customer is credited back the full amount including the chain fee, a "Withdrawal Failed" notification is sent, and the row lands in its terminal state. The ordering is deliberate: flipping the status first meant a refund failure left a terminal row that no cron would ever retry, stranding the money permanently.

So: a PROCESSING row older than 24 hours that is flagged for review is waiting on you to go and look at the exchange's own dashboard. It is not waiting on the platform.

Filter the withdrawal queue to PROCESSING once a day. Both PENDING and PROCESSING are decidable from that screen; deposits and transfers accept PENDING only.

A bulk approve reported failures

The bulk decision endpoint is PUT /api/admin/finance/withdraw/log/status (edit.withdraw), taking ids, status (COMPLETED or REJECTED) and a reason that is mandatory for a rejection and must be at least three characters — it is emailed to the customer.

It does not re-implement approval. Each id is handed to the same single-row handler the queue uses, so the wallet-type routing, the fee accounting, the idempotency key and the customer email cannot drift from the single-row behaviour.

Partial success is reported, not rolled back. Approving forty withdrawals is forty independent money movements. If three fail, the thirty-seven that already left are not un-done.

The toast shows the first three failure messages. The full list is in the response and, because every row's steps are threaded into the same audit context, in the audit trail entry for the batch at /admin/system/audit under module ADMIN_FIN.

Common per-row messages and what they mean:

If nothing succeeded, the endpoint raises a 400 with the first failure rather than returning a green toast over a batch that did nothing.

A deposit was paid but never credited

Split this by how the deposit was made, because the two paths fail differently.

Fiat gateway deposits

A gateway deposit is credited by its webhook. If the webhook never arrives or never lands, the payment is real and the balance is not.

Spot (on-chain) deposits

These are credited by processSpotPendingDeposits, every 15 minutes. The same window applies as for withdrawals: PENDING, non-empty referenceId, a SPOT wallet, created in the last 7 days, oldest first, 100 rows per run.

Rows younger than about 25 minutes are handed to the WebSocket verification schedule; older ones are verified directly, once per run, using the same idempotency key as the WebSocket path so the two can never double-credit.

A deposit sitting at PENDING with no referenceId was never given a transaction hash to verify, and no amount of waiting will change that.

A customer's balance is wrong

There is a complete, append-only record of every balance movement, and it is the first thing to read.

wallet_audit_log carries one row per balance operation with operation, amount, previousBalance, newBalance, previousInOrder, newInOrder, transactionId and the idempotencyKey. Filter it with ?filter={"walletId":"..."} for one wallet, or {"userId":"..."} for the customer's whole trail.

In the panel it is the Balance ledger, on the Audit Trail tab of the customer's page (/admin/crm/user/{id}) and of each withdrawal, deposit and transfer detail page. The value is the arithmetic: previousBalance → newBalance beside the amount that caused it.

The operations you will see are WALLET_CREATED, CREDIT, DEBIT, HOLD, RELEASE, TRANSFER_IN, TRANSFER_OUT and EXECUTE_FROM_HOLD. HOLD and RELEASE move money between balance and inOrder on the same wallet — they do not change what the customer owns, and reading one as a debit is the commonest way to mis-diagnose a "missing" balance.

Transfers stuck in the queue

Internal transfers are PENDING until decided, and they appear in the Operations inbox as the transfers queue with a 3-day target, linking to /admin/finance/transfer.

Each row opens a settlement page at /admin/finance/transfer/{id} with five tabs — Details, User, Wallet, Manage and Audit Trail. Manage is where the decision is made; Audit Trail carries the admin trail and the balance ledger for the wallet involved.

• Single decision: PUT /api/admin/finance/transfer/{id} (edit.transfer). Only a PENDING transaction can be decided; the row is locked and its status re-checked inside the transaction, so a concurrent decision loses.
• Bulk: PUT /api/admin/finance/transfer/status with COMPLETED or REJECTED.
• REJECTED and CANCELLED refund the sending wallet.

A transfer that is still PENDING is waiting on an operator, not on a job. Transfers have no cron.

The trap that makes a hand-written check lie

Every DECIMAL column comes back from mysql2 as a string, not a number — deliberately, to avoid float loss on a DECIMAL(30,18) balance. Nothing in the product touches a raw balance without coercing it first.

It matters the moment you leave the product. In a quick script, a Node REPL or anything that reads the rows directly:

// balances arrive as strings
rows.reduce((a, r) => a + r.balance, 0)   // "0" + "1.5" + "2.25" -> "01.52.25"
rows.reduce((a, r) => a + Number(r.balance), 0)   // 3.75

amount.toFixed() throws on a string, and a + b concatenates. A "total balance" that is impossibly large, or a comparison that is always false, is this and not a corrupted database.

The same caution applies to any total you compute yourself across currencies: per-currency figures are kept separate throughout the platform precisely because a number that sums BTC to USD means nothing.

Before you escalate

Capture these, in this order. They are what anyone diagnosing after you will ask for, and most of them stop existing once a queue is worked.

And what not to do:

• Do not re-run an approve to retry it. See the top of this page.
• Do not credit a customer by hand before reading the balance ledger. Half the "missing" balances are a HOLD against an open order.
• Do not clear a PROCESSING row by editing its status. The reconciler owns that transition and refunds inside the same database transaction as the flip; a hand-edited status skips the refund and strands the money.
• Do not fix a webhook problem by crediting and moving on. The next payment fails the same way.

Related

• Troubleshooting — infrastructure faults.
• The admin panel — the Operations inbox and the daily routine.
• Settings reference — the Wallet tab, auto-approval and the withdrawal 2FA policy.
• Roles and permissions — edit.withdraw, edit.deposit, edit.transfer and edit.wallet.