Troubleshooting

Three gates run when the Monero service starts, in order: the licence file, the ecosystem_blockchain row's status, and a get_version call against the wallet RPC. Only the third fails silently from the admin UI's point of view — the row shows enabled while wallet creation, balance reads and transaction history refuse with "Chain 'Monero' is not active."

If probe 1 above answers, but the chain is still inactive, check the fourth prerequisite the diagnostics test: Chain service installed. It loads the addon's code module, and reports "The XMR blockchain addon code is not installed" in two different situations that no amount of configuration will fix:

• The files are not there. The product was never extracted onto the server.
• The files are there and do not load. The backend log then has Failed to load module @b/blockchains/xmr: … under the IMPORT tag, each time something asks for the Monero service, and the text after the colon is the cause. "Cannot find module" naming a file in the backend's utils folder means the Core is older than this addon needs: Monero 6.2.1 needs Core v6.7.9 or later. Until Core is updated no XMR address is created, no deposit is checked, and XMR withdrawals fail before anything is sent.

The licence gate reads the file each time it runs: it must decode on this server, not merely exist. A request to one of the chain's own routes is checked by a separate licence gate that caches its answer for five minutes.

While the chain is inactive, every use of the service checks the gates again, so operations start working once the fault is fixed. The cron process's background loop may not: that process never registers a live deposit monitor, so the loop that runs its dormant-wallet refresh starts only if the chain was active when the service was first created there. After fixing a wallet RPC that was down when the service started, restart both processes — pm2 restart backend cron.

They do: the status toggle on Admin → Ecosystem → Blockchains writes the chain's row and nothing else. A process whose Monero service is already active never reads the row again until it restarts, and no part of a withdrawal — the request or the handler — reads it at all, so even after a restart XMR withdrawals are still accepted, debited and sent.

To refuse new requests, disable the XMR token under Admin → Ecosystem → Trading → Tokens. The withdraw form lists only enabled tokens, so XMR drops out of it, and a request that still arrives is refused with "Failed to withdraw: Token not found for chain: XMR and currency: XMR" before anything is debited. Rows already queued are still sent.

Disable it while the chain is still on, because the token's toggle and edit form refuse every change while the chain's row is off — with "XMR Blockchain is disabled", or with "Please install the latest version of the blockchain XMR to enable this token" when the chain row's version is still 0.0.1. If the chain is already off, switch it on, disable the token, and switch it off again. The bulk token-status endpoint is the one door that does not check the chain's row; no admin screen calls it, and Withdrawals has the request.

That is every failed and refunded XMR withdrawal, not a second fault. The handler writes its reason to the row, the withdrawal queue rewrites it as "Transaction failed: …", and the refund then rewrites it once more as "Refund of" and the row's amount, eighteen decimals long. The reason is gone from the row by the time you look.

Find it in the log of the process that ran the attempt — usually the web process for a first attempt and the cron process for a retry, but the cron process also runs a first attempt that waited more than three minutes in the web queue, so check both:

pm2 logs backend --lines 5000 --nostream | grep "Failed to process transaction <withdrawal id>"
pm2 logs cron --lines 5000 --nostream | grep "Failed to process transaction <withdrawal id>"

The customer got the same text as the reason in their failure email. "Refund of …" is written before the refund itself, so confirm the money went back by the REFUND transaction whose referenceId is the withdrawal's id — see What a failed withdrawal leaves behind on Withdrawals.

The wallet RPC could not reach monerod. Note the asymmetry: the backend may be able to reach the daemon perfectly well and still see this, because monero-wallet-rpc maintains its own separate connection.

The addon tries to repair it. On this error — code -38 ("no connection to daemon"), or an answer whose text says the daemon is busy or could not be connected to — it issues set_daemon on the wallet the session has open, with the host of the daemon the backend itself last got an answer from and XMR_RPC_USER / XMR_RPC_PASSWORD as the daemon login, and retries the refresh once. If it still fails, check in this order:

1. monerod is running and answering probe 2.
2. monero-wallet-rpc was started with --daemon-address and, when monerod requires an RPC login, --daemon-login. set_daemon applies only to the wallet that is open, so without --daemon-login every session's first refresh hits -38 and has to be repaired this way — and if XMR_RPC_USER / XMR_RPC_PASSWORD are unset, or are not the daemon's login, every refresh fails.
3. XMR_RPC_USER / XMR_RPC_PASSWORD match the daemon's --rpc-login. The same pair is used for the wallet RPC and forwarded as the daemon login, so one mismatched password breaks this specific path while the wallet RPC itself appears healthy.
4. The wallet RPC can actually route to the daemon's host and port — a containerised wallet RPC pointed at 127.0.0.1 is pointed at itself.

While this condition persists the monitoring loop backs off exponentially, from 30 seconds up to five minutes, and daemon errors are not counted against a wallet's retry budget. Monitors survive an outage and resume; you do not need to restart anything once the daemon is back. A withdrawal that runs into this error is failed and refunded — nothing was sent.

A customer submitted an XMR withdrawal and the backend could not price it. Every new XMR withdrawal asks monerod for a fee estimate before it debits anything, so this refusal means nothing happened: no debit, no transaction row, nothing to refund. The customer can submit again once the daemon answers.

The text after the second colon is the daemon call's own error:

• A connection error, or "RPC call timed out after 30 seconds". The backend cannot reach the daemon — it made three attempts first. Run probe 2.
• "requires authentication but XMR_RPC_USER and XMR_RPC_PASSWORD are not configured", or "authentication failed". The credentials — see the next item.
• "No fees array received from daemon." The daemon answered without the fee tiers the estimate reads. A monerod still early in its initial sync answers that way; wait for the sync to finish.
• "every configured RPC endpoint failed for get_fee_estimate". You have a daemon list (XMR_<NETWORK>_RPC) and none of its entries answered.

A withdrawal that was already queued when the daemon went away is a different case: it fails inside the handler and is refunded.

A daemon that answers but is still synchronising can pass this check and then refuse the broadcast. A withdrawal that gets as far as its relay then ends as TIMEOUT with a hash the network never saw — see A withdrawal is TIMEOUT below. Confirm "synchronized": true in probe 2 before you let customers withdraw.

A service returned 401 and there are no credentials to answer with. The message names which service — daemon or wallet — so read it rather than guessing.

Either set both variables to match the --rpc-login you configured, or start both services with --disable-rpc-login and leave the variables unset. What does not work is authenticating one service and not the other.

Bad credentials stop deposit monitoring permanently. A 401 that survives the Digest retry raises "Monero RPC authentication failed. Invalid credentials", and the deposit monitor treats that as fatal: it removes the wallet from monitoring rather than retrying. Unlike a daemon outage, this does not heal when you fix it — restart both backend processes after correcting the credentials.

Read the rest of the line — it names one of two different faults.

"challenge marked stale — … did not see the challenge and the answer on the same TCP connection." The credentials were accepted: monerod only marks a challenge stale after the username matched and the digest verified. What it rejected was the nonce, which it keeps on the connection that issued it — something between the backend and the service (a reverse proxy, a load balancer, a tunnel that closes idle connections) delivered the answer on a different one. Point the key the failing row is about — the daemon row is titled with its source key, XMR_<NETWORK>_RPC when that is set and XMR_DAEMON_RPC_URL otherwise, and the wallet row is always XMR_WALLET_RPC_URL — at the service directly, or make the proxy keep one upstream connection per client connection. The addon's own calls fail the same way through that path.

"rejected user …: XMR_RPC_USER / XMR_RPC_PASSWORD do not match its --rpc-login." The service judged the pair and refused it. Both services must share the same --rpc-login; check the one the row names.

Earlier builds of the diagnostics produced the first fault themselves — the probe answered the challenge on a fresh connection — and reported it as the second, against credentials the deposit monitor was using without complaint. The chain was healthy; the test was wrong. On such a build, the curl --digest calls at the top of this page are the authoritative check.

The withdrawal could not start a wallet-rpc session, or lost it before the relay, and went back to PENDING. The watchdog retries it every five minutes; nothing was failed, refunded or emailed. The code after "sent:" says why:

• XMR_SESSION_BUSY — another session held the wallet RPC for the whole 90-second budget. The message names the holder (<session number>:<hostname>:<pid>:<id>), and so does probe 4. One long session — a dormant wallet's first refresh, a pool-backing send — is enough. A line ending "(it only tried: …)" belongs to a withdrawal that did not wait at all because another one had just waited in vain; that is by design.
• XMR_SESSION_UNAVAILABLE — Redis could not be reached, so no session started. See the next item.
• XMR_SESSION_LOST — the session stopped being able to renew its lock, or passed the 30-minute session cap, and refused to go on. The log has wallet-rpc session N (…) LOST its lock: with the reason — a renewal that could not reach Redis, a key that had expired or been taken, or the cap.
• XMR_WALLET_IDENTITY — the wallet that was open could not be proved to be the customer's. See the identity item below.

A session whose wallet call timed out keeps the wallet RPC until it answers get_version again, so a stuck or overloaded monero-wallet-rpc looks like a long run of XMR_SESSION_BUSY. If it never answers, after fifteen minutes the log says ALERT wallet-rpc session N: still no answer to get_version after 15 min of draining and the lock is released anyway — check monero-wallet-rpc then.

Every wallet-rpc session needs the Redis lock, and none starts without it: "the wallet-rpc session lock (Redis) is unavailable (…); … did not start". While Redis is unreachable, no deposit is checked, no withdrawal is sent (each is deferred and retried), and wallet creation and balance reads fail. Nothing is lost; everything resumes when Redis is back.

Every session proves which wallet it is talking to, and these lines mean a proof failed. Nothing was credited or sent because of it.

• WALLET IDENTITY MISMATCH wallet=<id> expected=<recorded address> got=<address> fence=<n> (right after open_wallet) — the file named after that wallet opened as another wallet. If it happens on every attempt for the same wallet, the file itself is wrong: a copy or restore over it, or an address recorded for the wrong file. got=(error …) instead of an address means no wallet answered — typically it was closed between the open and the check.
• The same line without "(right after open_wallet)" — the wallet changed in the middle of a session.
• OWNERSHIP MISMATCH wallet=… expected=… got=… txid=… — a transfer list or a transfer record named an address that is not the proved wallet's. The whole session is discarded and nothing in it is credited.

A one-off mismatch means something that ignores the lock touched the wallet RPC: an older backend build still running beside a new one, a script, a person with curl, a monero-wallet-cli. Find it before anything else — restart every backend process together on the same build, and stop driving the wallet RPC by hand. A mismatch that repeats for one wallet is a fact about that wallet: compare the address in a copy of its file (below) with the XMR address recorded on its wallet row.

A withdrawal whose file keeps opening as another wallet — or whose wallet row has no recorded XMR address at all — is failed and refunded after XMR_WITHDRAWAL_REFUSAL_FAIL_AFTER_HOURS (24 hours by default), with an ALERT withdrawal … Marked FAILED line, unless it carries any sign of a relay, in which case it is never failed automatically. A mismatch in the middle of a session, or a got=(error …), only defers the withdrawal; it never counts towards that bound. Afterwards the row's description reads "Refund of …" like any other failed withdrawal: what ties it to this cause is the ALERT line and the row's metadata.xmrPersistentRefusal, whose reason is file_is_another_wallet or no_recorded_address.

[CRITICAL] Wrong wallet context! is a different, internal check: it compares the wallet a piece of code expects with the wallet its own session was opened for. It should never appear; if it does, report it.

When open_wallet returns an error, the addon closes whatever may be open and tries once more. "Failed to open wallet on retry: {…}" carries the wallet RPC's own error — read it. In order of likelihood:

• The wallet RPC was not started with --wallet-dir. Wallets are opened by filename. With --wallet-file instead, only that one wallet exists.
• The wallet file has a password. The addon always passes an empty password; XMR_WALLET_PASSWORD is read by nothing. A wallet created with a password by external tooling cannot be opened.
• Wrong directory or wrong permissions. The service user must own the wallet directory.
• Something outside the platform is using the same wallet RPC — a monero-wallet-cli session, a script. The web and cron processes share the wallet RPC by design, taking turns under the lock; anything else does not take turns.

"Wallet RPC is not responding. Please ensure monero-wallet-rpc is running on …" is what the addon reports when opening a wallet fails because the wallet RPC refused the connection or a call timed out — including the wallet refresh that follows the open, when it runs past its budget (60 seconds for a deposit check, two or fifteen minutes for a withdrawal). A withdrawal that fails this way is refunded; nothing was sent. While monero-wallet-rpc is down, expect one of these for every new XMR withdrawal: the request needs only the daemon, so it is accepted and debited, and the handler then fails it and refunds it.

This is about master_wallet on the admin's master-wallet screens — the one balance those screens read out of a wallet file. Customer balances come from the ledger. A freshly opened wallet answers from its on-disk cache, so the addon refreshes it before every balance read, with a 120-second budget, and what a zero (or an old figure) means depends on how that read ended.

The read failed, and the screen shows the balance last stored. A refresh that cannot reach the daemon ("Monero daemon not reachable. Wallet cannot sync."), a refresh that outlasts its 120 seconds ("Wallet RPC is not responding…"), a wallet RPC that stays busy through the two-minute wait (XMR_SESSION_BUSY) — each fails the whole read. None of them comes back as 0. The screen keeps the figure stored by the last read that succeeded, and for a master wallet that has never been read successfully that figure is 0. The log has "Error fetching wallet balance: …" under the XMR tag when the refresh failed, and a line beginning [XMR] from the screen's own read in every case. Do not take the list's [XMR] line at its word: it prints "Monero daemon not synchronized" for every refusal that says the chain is "not active" — a licence that does not decode, a chain row that is off, a wallet RPC that did not answer get_version — whatever the daemon is doing. Run probe 1 when you see it.

The read succeeded against a wallet that is behind. A monerod that is still synchronising lets the refresh finish, but the wallet can only learn what the daemon already has, so a transfer in a block the daemon has not reached is missing from the balance. A refresh error the addon treats as non-fatal has the same effect: the log says "Wallet refresh warning (non-fatal): …" or "Wallet refresh failed: …", and the balance is then read from the wallet as it stood.

Check probe 2 first — "synchronized": true — then read those log lines. If the reads are timing out, the file has more to catch up on than one refresh covers, and nothing refreshes master_wallet in the background: the cron process's dormant-wallet refresh covers ECO wallets with a positive balance only, and even those it only keeps from falling far behind (see Deposits and confirmations). Reloading the screen repeatedly does not help — every load starts another session, and they queue. To see what the file really holds in the meantime, open a copy of it: Looking inside a wallet file safely, below.

Work through these:

1. Six confirmations have not passed. Roughly twelve to twenty minutes. This is the answer more often than anything else.
2. It is not in a block yet. The checks do not ask the wallet RPC for transfers that are still in the mempool, so a deposit shows nothing at all — no counter on the deposit screen, nothing in the log — until its first confirmation.
3. The daemon is not synced. get_transfers against an unsynced wallet returns an incomplete picture and no error.
4. The checks are being refused. Look for WALLET IDENTITY MISMATCH, OWNERSHIP MISMATCH or XMR_SESSION_UNAVAILABLE against that wallet — a refused check credits nothing, by design.
5. The monitor stopped. With nothing in flight, a monitor ends after 120 checks — somewhere past ten minutes — and unconditionally at ninety minutes. (A wallet that has never received anything stops sooner: three empty checks once it has run for ten minutes.) If the deposit arrived after that, only the background scanner will find it.
6. The background scanner is disabled. ECOSYSTEM_BACKGROUND_SCAN="false" removes the only path that catches a deposit made after the deposit page was closed. On a Monero install, leave it on.
7. The address is not in the scanner's working set. An XMR address is scanned for 72 hours after the customer last opened the deposit page or their XMR wallet; signing in does not add it. A deposit sent to an address nobody has looked at for longer is found only when the customer opens one of those pages again.

Re-opening the deposit page is the supported way to force a re-check: it re-registers the address and starts a live monitor.

The Monero background sweep rate is one wallet every twenty seconds, and every sweep queues behind live sessions for the one wallet RPC. On a busy install with many watched wallets, a deposit found by the scanner rather than by a live monitor can be minutes behind the chain. That is the cost of a single serialised wallet RPC, and it is deliberate — raising ECOSYSTEM_SCAN_RATE_XMR starves the live sessions that customers are actually watching.

Two different refusals come back from the withdrawal form, and they mean different things.

"Invalid Monero address: … It is not a well-formed Monero address" — the string failed the length, network-byte or checksum test. It really is broken: usually a truncated paste or a single wrong character.

"This is a Monero stagenet address, but this platform is configured for mainnet…" — the address is fine, but it belongs to another network than XMR_NETWORK names (mainnet when it is unset). If XMR_NETWORK disagrees with the network monerod actually runs, every genuine address is refused this way. The diagnostics compare the two and fail with "XMR_NETWORK=… but the daemon runs … — valid withdrawal addresses will be rejected". Fix .env, restart both backend processes, re-run the test.

Read the transaction's description. If it says "Funds are locked (x/y XMR unlocked). Waiting to process", the withdrawal is working as designed: Monero locks received outputs for about ten blocks, and the queue watchdog will retry. The hold is bounded at two hours from the row's creation, after which it fails and refunds.

If the description says nothing new, look for "deferred, nothing was sent" in the log: the wallet RPC was busy or Redis was unreachable, and the watchdog retries it every five minutes.

Anything else: see Withdrawals for the full state machine.

Neither of these is refunded automatically, and that is correct — the transaction may already be on the Monero network, and paying the customer twice is worse than a delay.

Nobody is notified on your side. A TIMEOUT sends the customer a "Withdrawal Under Review" notification and tells no administrator. You see these rows on the Ecosystem admin dashboard, under Payouts needing review — "Outcome unknown" for TIMEOUT, "Abandoned by recovery" for stale PROCESSING — and in the logs: every TIMEOUT path writes a line containing NOT refunding.

"stale PROCESSING with no trxId" is not proof of a stuck row. The recovery pass that writes it runs in the cron process and judges a row by its age since creation — five minutes — without seeing what the web process is doing. A first attempt that is still inside a long wallet refresh is reported in the same words, and so is listed on the dashboard. A live attempt ends by itself within about three quarters of an hour at the very most; act on a row that is still PROCESSING after that.

Then resolve it from the hash, not from the wallet. Every relay is preceded by a claim that writes the built transaction's hash to the row's txHashPending column (and into the TIMEOUT description):

• txHashPending is set. Look that hash up on a block explorer or with your daemon's /get_transactions — Withdrawals has the command. Found: complete the row with that hash as its trxId, and book the platform fee by hand — a row completed outside the handler gets no adminProfit entry and no PLATFORM_FEE credit, although the platform's cut, if there was one, went to master_wallet in that transaction. Not found: check again later, and confirm on a copy of the customer's wallet file (below) that the funds are still there, before you fail it and give the customer their money back.
• txHashPending is empty on a stale PROCESSING row. The withdrawal never reached the relay, so nothing was sent; it is safe to return it to PENDING or to fail and refund it.

Every one of those steps is yours to do by hand: no admin screen will approve, reject or edit a TIMEOUT or PROCESSING Ecosystem withdrawal. Each is a guarded update of the row in the database, and failing a row refunds nothing — the customer's debit has to be credited back separately. Resolving a row by hand on Withdrawals has the statements and the refund.

A TIMEOUT whose description reads "relay_tx answered -4 Failed to commit tx." and whose hash no daemon has ever seen is what a withdrawal looks like when the wallet RPC's daemon refused the broadcast — a monerod that is still synchronising does. Nothing was sent, but only the hash check proves it.

Do not re-run a withdrawal that carries a hash — the handler refuses it anyway — and do not open the customer's wallet in the live wallet RPC to look.

Disk and I/O, almost always. Check free space on the data-dir partition first. Then consider --prune-blockchain, which substantially reduces the on-disk size and still supports wallet syncing.

If memory is the constraint, lower max-concurrency in bitmonero.conf and add swap. A daemon that is being OOM-killed and restarted by systemd every few hours never completes an initial sync.