Scheduled jobs reference
All 79 jobs the scheduler registers — exact name, cadence, which extension gates it, what stops when it stops, and a symptom-to-job lookup for stuck withdrawals, uncredited deposits and stale prices.
The cron process runs 79 job definitions. Processes and ports
describes the process; this page describes the work.
The registry is built in backend/src/cron/index.ts. Every entry carries a name
(the key the admin panel and the trigger endpoint use), a period and a
function. period is a fixed interval in milliseconds, not a crontab
expression. There is no 0 3 * * * anywhere in this platform: a job with a
24-hour period fires 24 hours after the scheduler started it, not at midnight, so
a restart moves every daily job to a new hour of the day. Nothing in the admin
panel or .env changes a period — it is a constant in the source.
17 jobs run on every install. The remaining 62 are gated on an enabled extension and are absent from the registry entirely when that extension is off. A short list on the cron console is not a fault; it is an install with few addons.
The 17 core jobs
These are registered regardless of which addons you own. category is normal
for all of them.
| Job | Every | What stops if it does not run |
|---|---|---|
processPendingOrders |
15s | Binary orders whose in-process expiry timer was lost (deploy, crash) never settle. This is the backstop, and the tight cadence is deliberate — a slow settle is indistinguishable from a deliberate refusal in the audit trail. |
processPendingSpotOrders |
60s | Open spot orders are never reconciled against the exchange: fills are not credited, cancellations are not refunded, partial fills hang. |
processCurrenciesPrices |
2m | Every currency price in the database freezes at its last value. Anything that converts to USD — wallet totals, fee figures, dashboards — drifts silently. |
reconcileSpotWithdrawals |
5m | A spot withdrawal whose exchange payout committed but whose local status write was lost to a restart stays PROCESSING forever. Idempotent; short cadence so crash-orphaned rows surface fast. |
reconcileTransfiDeposits |
5m | TransFi deposits stop crediting when a webhook is lost. fund_processing is emitted by no webhook and TransFi stops retrying after roughly two hours, so this is required, not optional. |
reconcileTransfiPayouts |
5m | A dispatched TransFi payout is never settled or failed when its webhook is lost, and a row orphaned between debit and dispatch is never resolved — the customer waits that long for a refund. |
processSpotPendingDeposits |
15m | Pending spot wallet deposits are not processed. |
processExpiredUserBlocks |
15m | A temporarily blocked user is never unblocked. The block has an expiry date; nothing else acts on it. |
syncMarketNews |
15m | The trading terminal's news feed stops refreshing. Quiet no-op with no provider key configured — operator-authored stories still serve. |
fetchFiatCurrencyPrices |
30m | Fiat rates freeze. Every fiat-denominated conversion keeps using the last figure fetched. |
processPendingWithdrawals |
30m | Pending spot wallet withdrawals are never picked up. |
processGeneralInvestments |
1h | General investment plans never mature: no settlement, no profit, no principal returned. |
cacheExchangeCurrencies |
1h | Warms the Redis exchangeCurrencies blob the currency endpoints read. See the note below — this one is less load-bearing than it looks. |
licenseHeartbeat |
6h | The licence server is not told this install is alive. |
processWalletPnl |
24h | No wallet PnL snapshot is written, so 24-hour change figures stop moving. |
cleanupOldPnlRecords |
24h | Old PnL rows and zero-balance rows accumulate. |
purgeGeoAccessLog |
24h | The geographic access log grows without bound. Does nothing when retention is set to keep entries indefinitely. |
The job writes the exchangeCurrencies Redis key with a 120-second TTL, on an
hourly period, so the key is expired for most of the hour. That is not a bug: the
blob carries price, which processCurrenciesPrices rewrites every two minutes,
and a longer TTL would let the cached price disagree with the row by up to fifteen
ticks. GET /api/exchange/currency falls back to the database on a miss and never
re-populates the key itself.
The practical consequence: this job stopping does not break the currency list. A currency you enable appears within two minutes either way.
The 62 addon jobs
Each block registers only when the named extension is enabled on
Admin → System → Extensions. Toggling an extension does not need a restart —
the scheduler re-evaluates gating every 60 seconds and logs
Extension <name> enabled — scheduled cron <job> or
Extension <name> disabled — stopped cron <job>.
| Job | Every | Owns |
|---|---|---|
verifyPendingEcoDeposits |
60s | Finalises pending on-chain deposits held in Redis: confirmation-depth checks and crediting. The only path that does this. |
backgroundDepositScanner |
60s | Detects deposits that arrive after the customer closes the deposit page. Scans recently-active addresses only, with a per-chain token bucket. Disable with ECOSYSTEM_BACKGROUND_SCAN=false. |
btcDepositScanner |
60s | Scans BTC wallets through the provider chain chosen by BTC_NODE (node, mempool or blockcypher; defaults to mempool with fallback). |
ecosystemWithdrawRecon |
5m | Re-enqueues orphaned PENDING ecosystem withdrawals whose rows outlived the in-memory queue — the wallet is already debited, so without this the money is gone and the payout never happens. |
processPendingEcoWithdrawals |
30m | The same recovery sweep on a longer cadence, kept as a second safety net. |
| Job | Every | Owns |
|---|---|---|
sweepFuturesPositions |
60s | Safety net that re-marks open positions against the live ticker, enforcing stop-loss, take-profit and liquidation. The matching engine already sweeps every 2 seconds where it holds the lease; this catches a realm whose engine never booted. |
reconcileFuturesPositions |
5m | Replays ScyllaDB status writes for positions whose wallet credit committed but whose Scylla write failed. |
reconcileFuturesOrders |
5m | The same, for orders. |
| Job | Every | Owns |
|---|---|---|
runFxExecutionReconciler |
60s | A-book correctness backstop: replays each enabled execution provider's broker ledger, detects orphans both ways, auto-suspends NEW routing on a cursor stall. Silent no-op with no provider rows. |
runFxHedgeMonitor |
60s | Syncs each enabled provider's balance / NAV / margin / closeout onto the provider row and alerts on a margin breach. |
reconcileFxAccounts |
5m | Expires due GTD/DAY orders, verifies the deals ledger (Σ pnl == balance) for accounts with open positions, refreshes denormalised equity and margin. |
processFxAffiliateRebates |
10m | Awards IB/partner rebates on committed commission deals. LIVE accounts only. |
syncFxCalendarAndNews |
15m | Refreshes the economic calendar and news from the active fx data provider. Operator-authored (MANUAL) rows are never touched. |
processFxSwaps |
1h | Settles overnight swap at the 17:00 New York cutoff. Hourly on purpose — every tick settles the most recent elapsed cutoff plus any earlier ones still inside the backfill window. A daily period would land at an arbitrary hour and a restart past a cutoff would lose that day's rollover, uncharged. |
runDexMarketRefresh (5m), dexConfirmationSweep (30s), dexFeeSweepSettle (5m)
and dexTokenRescreen (24h) are defined inside the forex_trading block of
the registry, even though each one carries category: "dex" and the console
groups them under their own heading. There is no dex bucket in the gating map.
With forex_trading disabled, none of the four is registered — so
swaps never advance past PENDING, no fee sweep is settled and the token
catalogue is never re-screened, while the DEX admin screens all look normal.
| Job | Every | Owns |
|---|---|---|
processPendingCopyTrades |
10s | Replicates leader trades to active followers. |
processClosedCopyTrades |
30s | Processes closed trades and distributes profit shares. |
monitorCopyTradingStopLevels |
30s | Triggers stop-loss and take-profit on open follower trades. |
checkCopyTradingDailyLossLimits |
60s | Pauses followers who exceed their daily loss limit. |
reconcileCopyTradingOrders |
5m | Releases COPY_TRADING wallet holds stranded by an order cancelled out of band. |
updateCopyTradingLeaderDailyStats |
5m | Daily statistics for active leaders. |
resetCopyTradingDailyLimits |
24h | Resets daily limits and reactivates paused followers. |
aggregateCopyTradingWeeklyAnalytics |
7d | Weekly leader performance analytics. |
| Job | Every | Owns |
|---|---|---|
processAiMarketMakerEngine |
5s | The engine loop. Runs in whichever process holds the ecosystem matching lease; on a dedicated cron process it is a structural no-op by design. |
processAiRiskMonitor |
10s | Volatility, loss limits and trading-pattern risk metrics. |
processAiPriceSync |
30s | External price feeds, and alerts on a major deviation. |
processAiAnalyticsAggregator |
15m | Trading statistics and performance metrics. |
processAiPoolRebalancer |
1h | Rebalances pools when asset ratios skew. |
processAiDailyReset |
24h | Resets daily volume and trade counters, writes the daily summary. |
processAiHistoryRetention |
24h | Prunes per-trade history past the retention window. Daily summaries and lifecycle audit rows are never pruned. |
| Job | Every | Owns |
|---|---|---|
aiSupportCacheHealthCheck |
1h | Alerts when prompt-cache reads fall below half of input tokens. Prompt-cache invalidation multiplies the bill with no error and no log line, so this alarm is the only detection that exists. |
aiSupportExpireStale |
1h | Closes off actions and step-by-step processes nobody used in time. Always on, whatever your auto-close setting. |
aiSupportProactiveSweep |
15m | Opens a ticket for deposits and withdrawals that failed in the last 45 minutes. Does nothing unless Proactive Support is switched on. |
aiSupportReindexKnowledge |
24h | Re-chunks the shipped documentation packs, your knowledge base and the FAQ. Skips packs whose checksum has not changed. |
aiSupportRetentionSweep |
24h | Deletes AI turn records past the retention window (default 90 days). |
aiSupportHarvestAnswers |
24h | Files a DRAFT article from the reply a human agent already typed. Makes no model call. |
aiSupportAutoCloseResolved |
24h | Closes tickets your team answered and the customer never replied to. Ships at zero days, which is off. |
aiSupportShareQuestions |
7d | Sends anonymised question fingerprints to MashDiv. Off unless you switch it on. |
| Job | Every | Owns |
|---|---|---|
processTradingBotEngine |
5s | Supervises the engine: starts it, stops it in maintenance mode, resyncs running bots. |
checkTradingBotStaleBots |
60s | Marks RUNNING bots whose engine tick went stale as errored and notifies their owners. |
updateTradingBotStrategyRatings |
1h | Aggregates review ratings for public approved strategies. |
aggregateTradingBotDailyStats |
24h | Daily per-bot statistics and resets the daily trade/profit counters — the daily loss limit depends on this reset. |
cleanupTradingBotOldData |
7d | Prunes audit logs and stale cancelled/failed orders past 90 days. |
| Job | Every | Owns |
|---|---|---|
processGatewayWebhookRetries |
60s | Redelivers failed merchant webhooks on a 1m / 5m / 30m / 2h / 24h backoff. |
processGatewayPaymentExpiry |
5m | Marks lapsed checkout sessions EXPIRED and emits payment.expired. |
processGatewayPayouts |
1h | Creates merchant payout records. INSTANT and DAILY balances are candidates every run, so an INSTANT merchant settles within this hourly cadence — not immediately — and still awaits admin approval. |
| Job | Every | Owns |
|---|---|---|
expireOffers |
5m | Expires NFT offers past their expiration date. |
settleAuctions |
10m | Settles ended auctions. |
processNFTBackups |
15m | Runs due blockchain-state backups. The finest schedule the admin route offers is hourly, so 15m keeps a backup close to its declared nextRun. |
| Job | Every | Owns |
|---|---|---|
p2pTradeTimeout |
1m | Expires trades past their expiration date and releases the escrowed funds. |
updateP2PReputationScores |
1h | Recalculates reputation for users with recent activity, records milestones. |
| Job | Every | Owns |
|---|---|---|
processIcoOfferings |
1h | Advances offering phases and status. |
processIcoVestingReleases |
1h | Announces due vesting tranches, closes finished schedules, cancels schedules behind refunded contributions. |
| Extension | Job | Every | Owns |
|---|---|---|---|
ai_investment |
processAiInvestments |
1h | Settles active AI investments. |
forex |
processForexInvestments |
1h | Settles active Forex investments. |
staking |
processStakingPositions |
1h | Processes staking positions and pays rewards. |
mlm |
processMlmReferralConditions |
1h | Evaluates referral conditions and creates rewards. |
mailwizard |
processMailwizardCampaigns |
1h | Sends campaign email. Each run sends at most speed messages per active campaign — speed is a per-run budget and a run happens once an hour. |
binary_ai_engine |
processBinaryAiEngine |
10s | Boots and reloads active engines after a restart, reconciles DB against memory, enforces the global kill-switch. |
"No error in the log" is not evidence a job ran
A job may refuse itself on the process it is registered in and still return
normally. backend/src/cron/refusal.ts is the only sanctioned way for that to
happen, and it is never quiet: every refusal lands in four places at once — sticky
state on the job row, logger.error, the live log on the cron console, and an
URGENT in-app plus email alert to every Admin and Super Admin.
The state is recorded on every refused tick. The three announcements are rate-limited to one per job per fifteen minutes, because the jobs that refuse tick every 5 to 60 seconds.
A refusal carries four fields, and the alert prints all four:
| Field | Means |
|---|---|
job |
The registry name, so you can find it on the console. |
reason |
Why this process must not run it. |
impact |
What is consequently not happening, in operator terms. |
fix |
The concrete change — an extension to enable, a setting to move. |
There is also a degraded variant. It means "running, but achieving nothing",
not "not running": the handler executed and no tick was dropped, but its work has
nowhere to land. The console shows those in warning colour rather than
destructive, and the job's status stays reportable rather than refused.
The three refusal gates
backend/src/cron/mode.ts holds the three runtime tests. All three answer false
immediately on a single-process (CRON_MODE=inline) install — none of them can
fire there.
When. On a dedicated cron process, whenever any fxExecutionProvider row has
status: true, or any fx order is EXTERNAL/ROUTING, or any fx position is
EXTERNAL/OPEN.
Why. Those two jobs drive the broker connection, not the quote feed. Two
reconcilers over one broker account replay the same fills and rewrite
syncCursor/externalMeta whole-column from two writers. No restart undoes that.
What you change. Nothing — and this one is deliberately not alerted, only
logged at debug level. armFxVenueSupervisor arms both on the process that holds
the forex-trading lease at the same 60-second cadence, so the work is demonstrably
being done elsewhere. It fails closed: if the check itself cannot answer, the jobs
stay refused.
When. ai_market_maker is enabled and ecosystem is not.
Why. The market maker trades ecosystem markets, and its bots enqueue into the
matcher's in-memory queue. With ecosystem off, nothing anywhere boots a matcher,
so there is nothing to quote. This is not about which process cron runs in — the
combination cannot make markets on any deployment shape.
What you change. Enable the ecosystem extension, or disable
ai_market_maker. The other six AI market maker jobs are unaffected and keep
running.
When. trading_bot is enabled and ecosystem is not.
Why. The same shape. Bots trade ecosystem markets and price off ecosystem candles, so even a paper bot has no price feed without it.
What you change. Enable the ecosystem extension, or disable trading_bot.
Strategy ratings, daily stats and weekly cleanup are unaffected and keep running.
Symptom to job
| Symptom | Look at | Every |
|---|---|---|
| A spot withdrawal is stuck at PENDING | processPendingWithdrawals |
30m |
| A spot withdrawal is stuck at PROCESSING with no transaction id | reconcileSpotWithdrawals |
5m |
| An ecosystem (on-chain) withdrawal never leaves | ecosystemWithdrawRecon, then processPendingEcoWithdrawals |
5m / 30m |
| A confirmed on-chain deposit is not credited | verifyPendingEcoDeposits |
60s |
| A deposit arrived after the customer closed the page | backgroundDepositScanner (or btcDepositScanner for BTC) |
60s |
| A card/bank deposit through TransFi is not credited | reconcileTransfiDeposits |
5m |
| A fiat rate is stale | fetchFiatCurrencyPrices |
30m |
| A crypto price is stale | processCurrenciesPrices |
2m |
| A blocked user is still blocked past their expiry | processExpiredUserBlocks |
15m |
| An investment matured but never paid out | processGeneralInvestments (or processAiInvestments / processForexInvestments for those addons) |
1h |
| A binary order expired but never settled | processPendingOrders |
15s |
| A P2P trade passed its deadline and escrow is still held | p2pTradeTimeout |
1m |
| A merchant webhook was never redelivered | processGatewayWebhookRetries |
60s |
| A DEX swap is stuck at PENDING | dexConfirmationSweep — check forex_trading is enabled |
30s |
Before you conclude a job is broken, confirm the scheduler itself is alive. A
stopped cron app produces every one of these symptoms at once, and the job list
keeps showing plausible "last run" times for 24 hours after it dies. That is what
the scheduler console is for.
The API
{ cronName }.The WebSocket at the same base path (/api/admin/system/cron) carries the live
log, and is gated on view.cron too. Opening the console is what puts a browser
in the relay audience, which is what makes the cron process start publishing
ordinary log lines at all — the scheduler holds no sockets of its own, so it asks
the web process whether anyone is looking before it builds a payload.
access.cron is the third key: it gates the page at /admin/system/cron. A
role needs access.cron plus view.cron to read the console, and manage.cron
on top to press Run now.