Troubleshooting

This is the designed behaviour when ScyllaDB is unavailable. Wallets, deposits and withdrawals do not depend on Scylla; orders, the book and candles do.

Check three things in order. Is SCYLLA_ENABLED set to "false"? Is the node running (systemctl status scylla-server)? Does the backend log show ScyllaDB unreachable at … at boot?

The backend probes the first contact point with a short TCP check before it retries, so an absent node costs a warning rather than a minute of exponential backoff. If the port is open but CQL is not yet serving, you will instead see connection retries with doubling delays and then "Max retries reached".

Also check SCYLLA_DATACENTER. It must match the datacenter name the node reports, not a name you chose, and a mismatch fails the connection rather than degrading it.

The vault is locked. Either ENCRYPTION_KEY_PASSPHRASE is not set and nobody has unlocked it since the last restart, or ENCRYPTED_ENCRYPTION_KEY is missing or malformed.

Open /admin/ecosystem. The masthead shows Initiate Vault when locked — enter the passphrase there. If the badge is green and you still see this error, the process serving your request is not the one that was unlocked: on a multi-process deployment each process holds its own decrypted key in memory.

Check the format too. ENCRYPTED_ENCRYPTION_KEY must be exactly four colon-separated hex parts. Three parts, or a truncated copy-paste, will not decrypt.

Crediting is done by one job — verifyPendingEcoDeposits, every 60 seconds. Check Admin → System → Cron that it is registered and running. If the whole addon's job group is absent, the extension is disabled or the cron process did not pick up the change.

If the job is running, the transaction has not reached confirmation depth. Required depth is 3 for Bitcoin, 6 for Litecoin, Dogecoin and Dash, and 12 by default for anything that does not specify its own.

If neither, look for the pending record. Between detection and crediting the transaction lives in Redis, not MySQL. A Redis flush in that window loses the pending record — the coins are still at the user's address and will be found again on the next scan of it, but nothing will happen until that scan runs.

Finally, check that the token is active and its network matches the chain's configured network. A token whose network no longer matches is skipped when addresses are issued.

Deposit addresses are generated for the network configured at the time. If BTC_NETWORK was flipped after users had addresses, those addresses belong to a different chain and are invalid — they must be regenerated, and anything sent to them is on the old network.

The same applies less dramatically to Solana: SOL_NETWORK set to anything other than mainnet or testnet silently falls through to devnet, and SOLANA_NETWORK is a separate key that only stamps metadata. If the two disagree, the stored record lies about which cluster the address is on.

Older Tron wallets are a special case that self-heals: a Tron entry holding a 0x… EVM address predates proper Tron derivation and is regenerated automatically on the next fetch. Seeing that in the logs is the fix working, not a fault.

Work through these in order.

Is the vault unlocked? Signing needs key material.

Does the chain have a master wallet, and is it enabled? The diagnostics downgrade withdrawal readiness to failed for either condition, and a disabled wallet behaves exactly like a missing one.

Did the queue lose the row? The queue is in-memory and the debit is durable, so a restart between the two strands the row. Three recovery passes exist — one at boot with no age filter, a watchdog every 5 minutes for rows older than 3 minutes, and a legacy 30-minute pass. If a row is older than that and still PENDING, it is in the set the recovery deliberately refuses to touch, which is exactly the count the admin overview breaks out.

Is it a token withdrawal on a chain with no gas? Network fees are paid by the master or custodial wallet in the chain's native asset. Native withdrawals still succeed in this state, which is why the failure looks selective.

A PROCESSING row that already carries a transaction hash is never re-broadcast — recovery promotes it to COMPLETED, because the funds have irreversibly left. That is correct, not a bug.

Deposit sessions take an exclusive lock on a custodial address, and the lock lasts an hour unless released. With no unlocked contract available, the request fails.

Deploy more custodial contracts from Admin → Ecosystem → Wallets → Custodial Wallets. The number of contracts is your concurrency limit for non-permit token deposits.

The lock is in-memory and per process, so it does not survive a restart and each process keeps its own view. That is an ordering convenience rather than a guarantee — a duplicate assignment shows two users the same address, and deposits are still attributed correctly by transaction.

Three known shapes of this.

The Arbitrum typo. ARBIRUM_MAINNET_RPC, missing the second T, is read by the admin balance endpoint and the legacy health check but never by the real provider path. Set only that and health says Up while deposits and withdrawals are broken. Set ARBITRUM_MAINNET_RPC.

Display-only keys. SOL_<NETWORK>_RPC and XMR_<NETWORK>_RPC are checked by the overview badge and read by no service. A green badge from those proves nothing.

Explorer coverage gaps. Etherscan's free tier is paid-only for BSC, OP Mainnet, Base and Avalanche, and Fantom, Cronos and HECO are not on V2 at all. Only three of those have a keyless instance the provider chain falls back to on its own — OP Mainnet and Base on hosted Blockscout, Avalanche on Routescan. BSC, Fantom, Cronos, HECO and Polygon Amoy have neither a hosted Blockscout instance nor Routescan coverage, so on those chains no keyless provider is appended to the order. BSC is the one where a key is the normal answer for a production install: set NODEREAL_API_KEY for mainnet, which is free for BSC, or MORALIS_API_KEY / COVALENT_API_KEY for testnet, the only two that index chain 97. Without one, transaction history and native-deposit detection fall back to an RPC block scan, which is slower and only walks forward from now.

<CHAIN>_NETWORK holds a value that is not one of that chain's valid network names, so the <CHAIN>_<NETWORK>_RPC key it derives does not exist.

The classic is Polygon: its mainnet is called matic, not mainnet. Also check that the RPC you configured actually serves the chain ID the network expects — the diagnostics report a mismatch explicitly.

Remember that a fresh value needs a restart. Provider instances are constructed at module load and nothing re-reads .env.

The matching engine boots inside the backend process, and exactly one process may own it. Order placement puts the order in the serving process's in-memory queue, so if the wrong process holds the lease, orders rest forever.

A dedicated cron process is refused the lease structurally, precisely to prevent this. If you are running a custom process layout, check the engine_lease table and the boot logs for a message about another process holding the lease.

The same rule is why the AI Market Maker must run in the engine's process: its bots enqueue into that same in-memory queue, and an order enqueued anywhere else is refused.

The market has no precision.amount and precision.price configured, and order placement refuses rather than guessing. Edit the market and set both, along with the amount, price and cost limits.

This is the usual cause of a newly created market that accepts nothing — the market row is created before its metadata is filled in.

The symptom is specific: the book displays depth at a price, the customer's Open Orders tab does not list it, and both cancelling it individually and Cancel All leave it exactly where it was. It usually appears after a restart.

Check the order index first — not the book. Every list in the UI reads the orders ledger, so an order the ledger has already cancelled or deleted is correctly absent from all of them. The matching engine reads a different table: open_orders_by_market, the book-ordered index. A row left there with no OPEN order behind it is loaded at every boot as a live resting order, and the five-minute reconciler then rebuilds the aggregated level to match it. The depth is real to the engine and invisible to every cancel door the customer has.

pnpm eco:index:check                       # read-only; names the markets that disagree

A market reported with STALE=n is this case. The repair:

pnpm eco:index:repair                      # removes the stale rows

Then restart the backend. The engine is still holding those orders in memory, and until it reloads it will keep the level standing.

From 6.6.4 the server does this for itself: it compares both counts at boot, and one market an hour while it runs, and rebuilds any market whose index holds more rows than the ledger has open orders — before a single order is loaded, so one restart clears both the index and the depth it was producing. eco:index:check stays the way to confirm it, and the only way to see the state on a version that does not yet self-heal.

Two things this is NOT, and both look identical from the trade page: a genuine ghost level (below), and funds still held for an order that no longer exists (further down).

Three ways the aggregated book can diverge from the orders table: a level with no backing open order (ghost), an open order with no level (missing), and a level whose amount does not equal the sum of remaining at that price (mismatch).

pnpm rebuild:eco-orderbook                                 # report
node backend/scripts/rebuild-eco-orderbook.mjs ETH/USDT --execute

Run pnpm eco:index:check before this one. This script rebuilds the book from the orders ledger, which is the right authority — but the running engine rebuilds it from what it loaded out of open_orders_by_market, so if that index is holding rows the ledger disowns, the level you delete here comes straight back. Repair the index, restart, then rebuild the book if anything is still wrong.

It is non-destructive: orders, trades, candles and user funds are untouched. It also reports ecosystem levels stranded in the futures keyspace by an older defect where unqualified writes landed in the wrong keyspace.

Restart the backend afterwards. The engine holds the book in process memory and a repaired table plus a stale process disagree immediately.

Symbols with an active AI market maker are skipped by default — that book is maintained without real order rows, so rebuilding from orders would wipe its quotes. Pass --include-ai to override, having stopped the market maker first.

The backend's own five-minute sweep is more discriminating than this script, and from 6.6.4 more so again. It identifies the market maker's own levels individually and reconciles only the rest, so an AI market is normally swept without touching its quotes. Where it cannot tell them apart — a market whose maker is configured ACTIVE while none of the levels are recognisably its own — it now waits for two consecutive quiet sweeps and then treats the book as customer depth, on the grounds that a maker which is quoting rewrites its quotes continuously. Before 6.6.4 it gave up on such a market permanently, which is how an unbacked level there became impossible to remove.

If you keep a maker configured ACTIVE while deliberately not quoting, its old ladder is now cleaned up like any other unbacked depth. Stop the market maker for that market if you want the ladder left alone. The transition is logged once per market rather than on every sweep, so Reconciling <symbol>: in the backend log is where to look.

Hold residue. Report it, then release it:

pnpm reconcile:eco-inorder                 # report
node backend/scripts/reconcile-eco-inorder.mjs --apply
node backend/scripts/reconcile-eco-inorder.mjs --user <userId> --apply

It recomputes the correct hold from the user's open orders and releases only the positive difference. It never debits a balance and never raises a hold, so it cannot over-credit. An under-locked wallet is reported rather than fixed — that is pnpm fix:eco-orders.

Restart the backend after applying, so cached wallet rows refresh.

The available unspent outputs would cost more in network fees to spend than the amount being withdrawn — usually many small inputs from repeated small deposits.

Check Admin → Ecosystem → Wallets → UTXO for that wallet. If the inputs are genuinely fragmented, the withdrawal needs to be larger or the wallet needs consolidating. If inputs show as LOCKED against a transaction that never completed, that is a stuck payout rather than a fee problem — check the withdrawal queue.

The driver caps in-flight requests per connection and refuses beyond it. The write path that used to cause this is bounded now, so seeing it today usually means genuine concurrency on an under-provisioned pool.

SCYLLA_LOCAL_CONNECTIONS tunes it, accepting 1 to 32 and defaulting to 4. A shared two-core VPS wants fewer; a dedicated cluster more. A bigger pool is headroom, not a fix — if it recurs immediately after raising it, something is issuing unbounded writes and the pool size is not the problem.

Two causes, usually both.

Permission gates ship strict and fail closed, so a newly enabled addon is reachable by Super Admin only until an operator grants its keys per role in Admin → Roles. The addon defines twenty-six keys; access.ecosystem is the one that opens the console.

And the route gate is held in memory, so a grant does not take effect until the API process restarts.

If the menu is missing for a Super Admin too, the extension is not actually enabled — check Admin → System → Extensions and confirm the licence validated.