Core v6.7.7

Release Date: September 10, 2026 Tags: CONFIGURATION, SCHEDULER, CRON, WITHDRAWALS, ECOSYSTEM, MONITORING, ALERTS, RELIABILITY, DOCUMENTATION

Overview

A safety release for one setting, and the setting is RUST_OWNS. It is the list of work this backend hands to the Rust backend being brought up beside it, and every name you put in it makes this backend do less. It shipped in 6.7.6 with no entry in .env.example and no release note, described only in a code comment. This release documents it and fixes five ways it could look right and behave wrongly.

If you have never set RUST_OWNS, nothing here changes anything you can observe. The default is empty, empty means this backend does everything exactly as before, and every change below sits behind a non-empty value. There is no schema change and nothing to run by hand.

If you have set it, two of the five want reading before you update. One typo shape — a token ending in .* — could take six trading engines read-only while the boot log said it had done nothing. And two withdrawal jobs that settle money customers are already owed reported a completed run every five minutes while doing no work at all. Both are in Upgrade Notes, with what to check.

Update Instructions

pnpm updator

There is no schema change in this release — no new table, no seeder, and nothing to run by hand. The work is entirely server-side, so the backend restart is the half that matters; the frontend rebuild changes nothing here and is harmless.

.env.example gains a RUST_OWNS block. Your own .env is not touched by the update, and it does not need to be: an absent variable and an empty one mean the same thing. Copy the block across only if you want the documentation next to the value.

If you already have a value set, read the boot log after the restart. The grammar has been tightened, so a token that used to be accepted quietly may now be reported, by name, as naming nothing — which is what it always meant, said out loud for the first time.

Upgrade Notes

If you set a token ending in .*, check it now — six engines may be read-only

Only one wildcard has ever been documented: eco.deposits.*. The parser built a match for any token of that shape, so lease.*, eco.*, spot.* and binary.* were all honoured — while the part of the code that reports mistakes recognised only the documented one. The two disagreed about the same string, and they disagreed in the dangerous direction.

RUST_OWNS="lease.*" stood down every engine lease on the process — ecosystem matching, futures matching, forex trading, the AI market maker, DEX confirmations and the DEX pool index — and printed this at error level in the same boot log:

[error] RUST_OWNS names "lease.*", and NOTHING on this process is called that.
        This process is still doing that work.

One of those two behaviours was a lie and there was no way to tell which from the outside. eco.* was the same shape aimed at the withdrawal queue, every deposit monitor and every chain's deposits at once. Both are one keystroke from a token that is real (lease.candidate, eco.withdrawals).

What to do. Look at RUST_OWNS in your .env and in production.config.js. If any token contains a * and is not exactly eco.deposits.*, it was doing something you did not ask for and it now does nothing. Replace it with the tokens you meant, restart, and read the boot log: every token that names nothing is printed by name, and now that is the only thing such a token does.

Two withdrawal jobs reported success every five minutes while doing nothing

RUST_OWNS="eco.withdrawals" hands the ecosystem withdrawal queue to the other stack. It names the queue, not a scheduled job — so Process Pending Ecosystem Withdrawals (every 30 minutes) and the Ecosystem Withdrawal Watchdog (every 5 minutes) kept ticking. Each one reached the queue, the queue declined to do anything, and each handler returned normally with no error. The scheduler read that as a healthy run: completed, a fresh last run, and a success rate of 100%.

So the Ecosystem Withdrawal Watchdog ticked green every five minutes, forever, on the admin cron page — while PENDING ecosystem withdrawals piled up untouched. Those rows have already debited the customer's wallet. If the other stack was not draining the queue, the money was neither sent nor returned, and nothing on any screen would ever have said so.

What to do. If you have run with eco.withdrawals set, go to Finance → Transactions and look for WITHDRAW rows on ECO wallets still sitting at PENDING or at PROCESSING with no transaction hash. Those are the ones that were skipped. They are recoverable — remove eco.withdrawals from RUST_OWNS and restart, and the watchdog picks them up on its next pass — but check them against what actually went out on-chain first, because a row that was already broadcast by the other stack must not be sent a second time.

After this release both jobs refuse instead: the admin cron page shows them as refused with the reason, the success rate stops counting the tick, and every administrator gets an URGENT alert. A skipped job that reports success is the worst thing this switch can produce, and it can no longer produce it.

A handed-over job now shows as refused on the cron page, not as idle

This is a display change you will notice the moment you set any token, and it is deliberate.

Previously a job named in RUST_OWNS sat on the admin cron page at idle, never run — identical, on the only screen anyone looks at, to a job that is scheduled and simply waiting its turn. Setting a token on a deployment where no Rust scheduler was actually running produced one line in the boot log and then silence, indefinitely, with every signal reading healthy.

Such a job is now listed as refused, with a note saying it was handed over, which stack runs it, and how to take it back. The moment the other stack reports a run, that note is replaced by its real run state and the row goes green on its own. This is not an error state; it is the difference between "handed over" and "ran fine", which the page could not previously draw.

An overdue handover now alerts, so check your Rust scheduler is claiming what you gave it

A token this backend honours and the other stack does not claim is a job that runs nowhere. Nothing detected that before.

This release adds a watchdog: every five minutes, for each handed-over job, it compares the other stack's own reported run time against that job's schedule. A job that has not been reported inside twice its own interval — at least fifteen minutes, at most a day — raises the same URGENT alert a refused job raises, naming the job, saying it is running nowhere, and saying so more plainly when the job settles customer money.

Nothing happens on a healthy handover: an alert every fifteen minutes for a correctly configured deployment is an alert everyone filters. And a Redis that is unreachable is treated as no evidence rather than as an outage, so a blip does not alarm about every job at once.

If you set tokens optimistically and never confirmed the other side picked them up, expect to hear about it after this restart. That is the point.

A cron page showing every job as never run has one more explanation

If your Redis does not allow scripting — some managed tiers and connection proxies filter EVAL — then with RUST_OWNS set, the run-state snapshot the admin cron page reads was not being written at all, and the failure was swallowed. The page then showed all ninety-odd jobs as never run, on a deployment where they were running fine.

It now falls back to an ordinary write, publishes the snapshot, and says once, at error level, that it had to. If you see that line, the fix is a Redis that answers EVAL; until then the page is correct and one write of the other stack's is occasionally lost.

Added

• Added RUST_OWNS="" to .env.example, under a RUST_OWNS: leave this EMPTY heading which spells out the whole grammar — job names, the closed list of loop tokens, lease.<key> and lease.candidate, eco.deposits.<chain>, the one wildcard — and states plainly that tokens are case-sensitive, that all and * are refused, and that every token you add makes this backend do less.
• Added an overdue-handover watchdog on the scheduler process. Every five minutes it reads the shared run-state snapshot and, for each job named in RUST_OWNS, compares the other stack's last reported run against that job's own interval. Overdue raises an URGENT in-app and email alert to your administrators, writes the process log, and marks the job on the admin cron page — the same four places a refused job is recorded. It is armed only while RUST_OWNS is non-empty and only on the process that schedules.
• Added a standing handed to Rust state to every job named in RUST_OWNS, carrying the reason, the stack that owns it and the way to take it back, so the admin cron page can tell a handover from a job that has simply never run.
• Added a refusal to the two ecosystem withdrawal jobs that eco.withdrawals disarms downstream, so each one records a refused run instead of a completed one — with the reason, the impact in the operator's terms, and the fix.

Changed

• Changed the wildcard grammar to a closed list. eco.deposits.* is the only namespace wildcard; anything else containing a * is a token that names nothing, matches nothing, and is reported by name at boot. The one function that decides whether a wildcard is honoured is now the same one the report asks, so the predicate and the report cannot disagree about a token again.
• Changed the boot summary to name only the tokens actually honoured. It used to echo the operator's whole list, so a token that names nothing appeared on the "Rust owns these" line and on the "nothing is called that" line, in the same log, seconds apart.
• Changed the error line for a token containing a * to say that * is not a wildcard here except in eco.deposits.*, and that nothing was handed over for it. It is the one typo whose author believes it did something.
• Changed the run-state entry carried on behalf of the other stack to expire. It is kept only while it is newer than twice the job's own interval, capped at the snapshot's own 24-hour lifetime, and a job that has never been reported at all is not carried.

Fixed

lease.* and eco.* were honoured and reported as unknown at the same time

Covered in Upgrade Notes. The parser accepted any <namespace>.* while the reporter recognised only eco.deposits.*, so a single mistyped token could stand six trading engines down while the boot log said the work was still running here.

• Fixed the wildcard grammar: a wildcard is honoured only when it is on the documented list, which today holds eco.deposits.* alone.
• Fixed the case one segment deeper: eco.deposits.et* was reported as a token this build understands while matching no chain at all.

Two customer-money jobs reported completed runs while structurally unable to run

Covered in Upgrade Notes.

• Fixed Process Pending Ecosystem Withdrawals and the Ecosystem Withdrawal Watchdog so that, under eco.withdrawals, each refuses rather than returning quietly. The refusal is recorded as a standing state on the job, logged, shown on the admin cron dashboard's live log, and mailed to every administrator — and the success rate no longer counts the tick.
• Fixed the ordering so the refusal is raised before the watchdog's own "starting reconciliation" / "reconciliation completed" lines, which used to bracket a pass that read no rows.

Setting a token with no Rust scheduler running was silent

Covered in Upgrade Notes.

• Fixed the admin cron page reading a handed-over job as idle, never run. The row now says it was handed over and why.
• Fixed the absence of any overdue detection, with the five-minute watchdog described under Added.

A run state from before the handover was carried forward forever

The snapshot the admin cron page reads is written with a 24-hour lifetime — and this backend rewrote the whole snapshot after every completed run of every one of roughly ninety jobs, which refreshed that lifetime continuously. On top of that, the entry belonging to a handed-over job was carried through each rewrite exactly as found.

The entry being carried is usually this backend's own, from before the handover: the commonest way to hand a job over is to set the token on a process that has been running it for months. So the page showed a green, completed row, with a last-run time frozen at the restart and a next-run time permanently in the past, for a job that no stack had run since.

• Fixed the carry so an entry is kept only while it is fresh for that job's schedule, and dropped otherwise. A dropped entry puts the row back on the handover note above — which says no stack has reported a run — instead of on a green one that is no longer true.
• Fixed the case where the other stack registered a job but never ran it: an entry claiming no run at all is no longer carried as though it were one.

A Redis that refuses EVAL published no cron run state at all

With RUST_OWNS set, the snapshot write becomes a compare-and-set — so that this backend's ninety-odd per-run writes cannot erase the other stack's entries between them. A Redis that does not answer EVAL rejects that write, and the rejection was discarded without a word. Nothing was published, and the admin cron page, which reads that snapshot from the web process, showed every job as never run.

• Fixed the swallowed failure: an EVAL that is refused now falls back to an ordinary write of the same merged snapshot, so the page is served, and says so once at error level with the command and the consequence named.
• Fixed it without weakening the compare-and-set that works: a write that ran and lost is still left alone rather than escalated, because falling back there would turn the retry limit into a short delay in front of the exact overwrite it exists to prevent.
• Fixed the outer failure path, which discarded every publish error regardless of cause. A failure to publish is now logged, at most once a minute.