Market data, charts and the pool indexer

"The chart is blank" and "the pair list shows yesterday's price" are the same class of ticket with four possible causes, and they are not interchangeable. This page is the map.

Four layers, and which one you are looking at

The live surface is the /api/dex/market websocket, which starts a poller per subscribed symbol. The durable surfaces are GET /api/dex/chart and the dexPair statistics columns. Neither is a fallback for the other, and diagnosing from the wrong one is the usual reason a fault looks intermittent.

Providers

The registry is a literal switch over three names, and no vendor is named anywhere outside its own adapter file and that switch — so changing provider is a settings change, never a rewrite.

The API key is read from the environment and never from the settings table — a market-data key is a bearer credential for a paid account, and putting it in settings would place it behind an operator-editable, plain-text-audited PUT. The console is only ever told the boolean. See Environment variables.

A fallback equal to the active provider is treated as no fallback — it would be the same cool-off twice.

The cool-off policy

A provider that fails is paused for as long as its failure can plausibly last, and you are told once rather than every minute. This is what stops a rate-limited vendor being hammered once per cron tick for as long as the cron runs.

The floors matter: a vendor asking for a 1-second retry would keep the storm running at 1 Hz, and AUTH and PLAN_LIMIT are configuration states that no amount of retrying clears, so each retry is a guaranteed-failing round trip.

The two scopes are the important part. Cool-offs are held in Redis under dex:md_ban:<provider> (account-wide — tickers and the tape honour it) and dex:md_ban:history:<provider> (OHLCV only). Merged into one key, a single 403 on a chart timeframe would take the whole price feed down for the cool-off: a PLAN_LIMIT is an endpoint verdict and only ever originates from the chart path, while a rejected key is an account verdict.

While the active provider is cooling off, getActiveProvider() returns null and the refresh sweep logs once and does nothing. Redis being unavailable degrades the cool-off to "not banned" rather than taking the data path down with it.

Which timeframes the chart offers

The chart engine can draw thirteen timeframes, 1m to 1w, and GET /api/dex/chart accepts all thirteen: an interval the active provider does not serve natively is derived by rolling a finer native series up (3m from 1m, 30m from 15m, 2h and 6h from 1h, 3d and 1w from 1d), and gaps stay gaps rather than being carried forward as flat bars.

The swap terminal offers only the intervals the market's source serves natively, and opens on 1h. Derivation is honest for the bars it has, but a derived timeframe is several pages of a finer series the vendor may not retain — GeckoTerminal keeps roughly six months of minute bars — so it is exactly the timeframe whose chart goes blank when scrolled back, with nothing on screen to say why.

The list is the union of the active provider's set and the fallback's, so a codex fallback beside a GeckoTerminal primary adds 30m and 1w to every indexer-backed market. It is a capability, not a health reading: a provider serving out a cool-off still serves 4h, so the menu does not lose entries during a rate-limit pause. Each market carries its own list as metadata.chartIntervals on GET /api/dex/market, because marketDataSource is a per-pair choice.

Two things follow for a trader:

• The chart opens on 1h wherever the source serves it; on an ONCHAIN pair it opens on 1m, the nearest served timeframe. A timeframe the trader has already chosen for that market is remembered in the browser for 24 hours and wins over the default.
• Zooming steps between offered timeframes only. Zooming out of 1h on a GeckoTerminal market lands on 4h, not 2h.

Integrators calling the chart route directly are unaffected: it still serves every interval in its enum.

The polling and cache settings

The floors are not decoration: a mistyped 0 in the settings table would arm a setInterval at 0 ms against a third party's rate limiter.

Poll periods are memoised for 60 seconds, so a change takes effect on the next symbol subscribed rather than instantly — putting a database round trip in front of a metered indexer call on every tick would cost more than the staleness.

Under a threaded install every worker loads the websocket route, so the manager takes a short SET NX PX lease per (kind, symbol) and lets the losers read the Redis cache. It is deliberately weaker than the confirmation poller's engine lease: a market-data miss is a stale tick, a confirmation miss is a wrong balance.

The pair-statistics sweep

runDexMarketRefresh keeps dexPair.lastPrice, change24h, volume24hUsd and liquidityUsd current so the market rail can sort and render without one indexer call per row.

It is registered in the cron console as Refresh DEX Market Data, category dex — see Scheduled jobs. What it does, in order:

1. Returns immediately if dexEnabled is off — that is the normal state on most installs, so it does not throw the 503 an HTTP handler would.
2. Gets the active provider, or logs once and stops.
3. Reads every dexPair with status = 'ACTIVE'.
4. Resolves each row's chart source as indexerId ?? poolAddress, and skips a row that has neither.
5. Groups by chainId and fetches in chunks of 25.
6. On a chunk failure it calls the shared cool-off policy and moves to the next chain, not the next chunk — whatever refused that chunk will refuse the rest of the chain's, and each attempt is a metered call.

One dead chain therefore costs its own rows and nothing else.

Everything the sweep writes is display-only and lossy — DECIMAL columns carried so admin SUM() and ORDER BY happen in SQL. They are never authoritative and never used to reconstruct a transfer. See Pairs.

The pool indexer

A pool's own Swap logs, turned into 1m candles in the same chart cache the provider path writes. It exists for one case the vendors cannot serve: a pool created ten minutes ago is in no indexer, and that is exactly when an operator most wants a chart.

It runs over dexPair rows where marketDataSource = 'ONCHAIN' and status = 'ACTIVE', resolves each to its poolId, and skips any pool whose dexPool.state is not ACTIVE.

It is not a cron job and it is not on the cron console

It is a setInterval on the main thread, armed at boot by initializeDexPoolIndexer() after the chain registry seed, behind the dex-pool-index engine lease. It does not appear on Admin → System → Cron, and its cadence comes from dexPoolRefreshMs rather than from a cron period.

Both guards are load-bearing:

• Not a *.ws.ts module. Every route module is loaded once per worker thread, so a module-scope timer there gives N pollers hammering one RPC endpoint against a shared provider-ban key, N sets of duplicate frames, and N writers racing the same cache key through N independent write queues that do not serialise against each other.
• Not the cron-only process. /api/dex/market is not on the relay allow-list, so nothing published from there reaches a browser.
• isMainThread and the lease, not one or the other. The lease is held per process, so every worker thread of a threaded install would otherwise win one.

Three things follow for an operator:

• It stands down when the venue class is off. It checks dexDirectPoolsEnabled before taking the lease, so a disabled install does not hold one.
• Boot failure is non-fatal. A deployment where it fails to start still serves the exchange; on-chain charts degrade to whatever is cached, which is a visible absence rather than a broken platform.
• A process that loses the lease logs it and serves from the shared cache: "Pool indexer not armed: another process holds the dex-pool-index lease." It is a different lease from dex-confirmations, so a process that lost that race can still be the one indexing charts.

The indexer settings

Each tick also re-derives a 5-minute overlap plus dexPoolRefreshMs of history, so a log arriving at the edge of the previous pass still lands in its bucket. Re-deriving that bucket is correct where patching it would not be.

The reorg tripwire

Two columns on dexPool do the work:

• indexedToBlock is the cursor.
• indexedToBlockHash is the tripwire — the hash of the block the cursor sits on, stamped on the same update as the cursor. A cursor with no hash silently degrades reorg detection to never.

The sweep never reads past head − requiredConfirmations. That single rule is the whole defence: a reorg shallower than the chain's confirmation depth cannot change an emitted candle, because the blocks it rewrote were never read.

A deeper reorg is detected rather than assumed away. Before extending, the block at the cursor is re-read and its hash compared. On a mismatch:

1. A warning is logged naming the pool and the block.
2. Every dexPoolEvent above cursor − dexPoolIndexReorgRewindBlocks is deleted — the rows are the evidence behind the candles, and leaving them would make the re-derivation agree with a history that no longer exists.
3. The cursor rewinds and the affected buckets are re-derived, never patched. A partially-corrected bar is indistinguishable from a correct one afterwards.

An unreadable block is not treated as a reorg. Treating an RPC hiccup as one would rewind and re-derive on every transient failure — an RPC bill and a churned cache for nothing.

marketDataSource is per pair; dexMarketProvider is global

These are easy to conflate and they are not the same axis.

dexMarketProvider selects the vendor, and onchain is not one of its values — the three are geckoterminal, codex and mock.

dexPair.marketDataSource selects where one market reads its chart:

GET /api/dex/chart treats this as a provider override, not a second code path: the same cache, the same bucket grid, the same derivation rules, the same bare number[][] on the wire. The datafeed must never learn that two sources exist, because a client that could tell would eventually branch on it.

On-chain volume is in base-token units, not dollars — pricing it would need a USD rate for a token whose only price source may be that very pool. volume24hUsd and liquidityUsd come back null rather than 0, because zero is indistinguishable from unpriced.

Triage: a blank chart

Work down. Each step rules out a layer.

Triage: the pair list shows yesterday's price

Related: Scheduled jobs for the sweep's place among the four cron jobs, Requirements for what else to check when a chain is not serving, and Troubleshooting for the quote-side symptoms.