Hummingbot Connector v6.1.9

Release Date: September 11, 2026 Tags: RATE LIMITS, AUTHENTICATION, CLOCK SKEW, CANCELLATION, SHUTDOWN, BATCH CANCEL, CONNECTOR

Overview

The backend was billing its own queue delay to the bot's clock, calling it a broken clock, and charging it as a forged signature. Under load the platform told operators to go and sync the clock on a machine whose clock was correct — often the same host the platform was running on — and each of those refusals withdrew a little more of the bot's rate-limit waiver until it was dropped onto the browser cap. A load spike converted a healthy client into a hostile one, and the platform's own slowness pulled the trigger.

stop in Hummingbot's own terminal did not really stop. The connector already switched to a short cancel schedule in cancel_all, and that was believed to fix it. It does not: for every V2-controller strategy — which is every preset this platform generates — the ladder is cancelled by the strategy's own on_stop(), one order at a time, and cancel_all is never reached at all. Those cancels ran on the 51-second live schedule, sharing one rate-limit budget. This is the shape of the original report: "The bot more or less refuses the STOP command from terminal, i stop it by Emergency Cancel in Bot Console".

A batch cancel on a cross-process install reported total failure. Every cancel is deferred on such an install, and deferrals were reported in the same bucket as "the venue refused this" — so clearing a twenty-order ladder answered canceledCount: 0, failedCount: 20 while the orders were gone moments later.

Redeploy the connector. KIT_VERSION is now 2026.09.11.2. The stop hook and the batch-cancel sweep are in the connector's own Python and the backend half alone does not deliver them.

Requires Core v6.7.8. The arrival timestamp and the deferred body validation this release depends on are two core files (handler/Request.ts, handler/Routes.ts); they change in that release and are explained here.

Update Instructions

pnpm updator

Then redeploy the connector into your Hummingbot checkout and restart the bot.

Update Core to 6.7.8 first — without it the backend half of the clock and rate-limit fixes is not present, and this addon's auth path expects the arrival timestamp it stamps.

There is no schema change and no setting to configure.

• Both halves are required and they work together. The backend stops charging a real bot for its own latency; the connector stops spending a whole ladder's worth of budget on per-order cancels. Deploying one without the other leaves the other half of each failure in place.
• A perp-only bot needs this release too. The shutdown flag is a module global shared by both connectors, so a bot running spot and perp is covered by whichever installed the hook — but a perp-only bot has no spot connector to do it, and its stop would still cancel on the 51-second schedule. The perpetual connector now installs the hook itself.

Upgrade Notes

The server was billing its own latency to your clock

The HMAC gate rejects a signed request whose timestamp is more than the receive window (10s) away from "now" — and that call happens inside the handler, after the licence gate, the rate limiter and the body read have each yielded to the event loop. Under bot load those gates were measured taking 4.1 seconds. With the connector's own client-side throttling on top, the backend reported skews of 12-15 seconds and told the operator:

"this machine's clock is 13410ms behind the exchange ... Sync the clock on the machine running the bot."

The bot was on the same host. A process cannot be thirteen seconds out of step with itself. Every one of those was our own queue delay.

It did not stop at a misleading message. Each one charged the failed-signature floor — the budget that exists to catch someone guessing a secret — so thirty in a minute withdrew the bot's 1200/min waiver and dropped it onto the 100/min browser cap. A bot running comfortably at 272 requests a minute became 919 rate-limit refusals and a retry storm that made the stall worse.

Skew is now measured against the moment the request arrived, stamped in the request constructor, which is the earliest point the process can observe it at all. Replay protection is unchanged: the accepted age is still at most the receive window measured from arrival, and the nonce cache's TTL is twice that, so it always outlives what it protects.

The diagnostics now report how long we sat on the request separately from your skew. That is the difference between "sync your clock" and "the exchange was busy", and those send an operator to opposite ends of the problem.

A real bot's malformed body was charged as a forged signature

Signed bot routes authenticate inside the handler, so the framework's schema check ran before anything knew whether the caller held the secret — and a schema refusal then charged the failed-signature floor. A legitimate bot sending a body this platform's schema rejects therefore spent the budget meant for probes, and thirty of those in a minute produced exactly the 429 storm the waiver exists to prevent.

Those routes now hold the schema check until after the signature verifies. A caller who proved the secret is never charged for a bad body; a caller who did not never reaches the schema at all, because the signature check refuses them first — which is the outcome the floor is actually meant to count.

This costs an attacker nothing they did not already have. A forged request with a well-formed body has always paid for the indexed key lookup and charged the floor, so declining to do that work for a malformed one protected nothing: the cheapest effective probe was never the malformed one. What it buys is that a schema refusal now only ever reaches a caller who proved the secret, which also stops an unauthenticated caller reading a route's field list out of a 400.

Three refusals no longer count against you

The floor exists to catch someone guessing a secret. Three refusals happen after a correct signature has been produced, and they are all states of a legitimate key holder:

• an expired key,
• a key an administrator disabled,
• a request that arrived outside the receive window.

None of them charge the floor any more. The last is the one that mattered most: it charged precisely when the platform was too slow to answer in time.

Two refusals on the same side of the signature check deliberately still charge, because they are attack signals rather than accidents: a replayed nonce means someone resent a captured request without holding the secret, and an IP-blocked request means a key is being used from somewhere its owner forbade.

stop now counts as a shutdown

The connector keeps two cancel schedules. The live one fights for about 51 seconds across a rate-limit window, because for a running bot a cancel that does not land leaves an order holding the customer's funds and nothing else will clear it. The shutdown one is short and wide, because at stop time nothing is going to re-quote and persistence per order only delays every order queued behind it.

On a staging box a twenty-order ladder was rested from Hummingbot's own terminal and stop was typed. The connector log for that run contains zero occurrences of cancel_all's own log line:

stop_command.stop_loop
  -> strategy.on_stop()                 <- StrategyV2Base cancels every limit
                                           order ONE AT A TIME here
  -> trading_core.stop_strategy()
  -> trading_core.cancel_outstanding_orders()
       if len(connector.limit_orders) > 0:    <- already empty
           connector.cancel_all(KILL_TIMEOUT) <- never reached

So cancel_all selecting the short schedule was necessary and not sufficient: the orders are cancelled by the strategy's own shutdown, and cancel_all is only a backstop for whatever that missed.

The fix is a hook on the stop command itself, because "are we shutting down" is a property of the application, not of any one cancel — a connector cannot tell an executor refreshing a quote from a strategy tearing down. Panel-supervised checkouts already got this from the SIGTERM hook; a checkout an operator runs by hand has no such hook, and typing stop never sends a signal. That is the gap this closes. The two compose: the wrapper restores the previous value rather than clearing the flag, so a stop inside a real SIGTERM shutdown leaves it set on the way out.

A ladder is now cleared in one request before it is cleared order by order

A shorter schedule decides how long one cancel fights. It does nothing about a hundred-quote ladder costing a hundred DELETEs against a budget the bot has already been spending on placements — which is what produced 1,388 rate-limit refusals in one measured 37-minute run, and why the operator's own workaround was the panel's Emergency Cancel. That is precisely this endpoint: DELETE /api/hb/order?symbol=, one request that cancels the lot for one unit of budget.

The sweep issues that request first. The per-order pass then runs against orders the venue has already cancelled, gets "no longer open — it is CANCELED" for each, and the already-gone matcher turns that into a successful cancellation. That ordering is deliberate: it means the sweep needs no order-tracker surgery of its own and cannot report an order cancelled that is not, and if it fails outright nothing is lost — the per-order pass is exactly the behaviour that shipped before it.

The subset check, because the door is wider than this connector. cancel_all means "cancel the orders this connector is tracking". The batch door means "cancel everything this API key's user has open on that symbol", which is a strictly larger set — an operator's hand-placed order on the same account is in it, and is none of the bot's business. So the open orders are read first and the door is used only when every one of them is an order we are already cancelling. When it is not, the sweep returns quietly and the per-order path handles our subset exactly.

One race is worth naming: an order placed on the same account between the read and the cancel is swept too. The window is a few hundred milliseconds during a shutdown of the bot that owns the market, and the alternative — leaving a ladder resting because a cancel storm could not get through the rate limiter — is the failure that was actually reported.

Below three orders the batch path is not a saving (it costs a GET plus a DELETE), and the exact per-order path carries none of the wider-door risk, so that is what is used.

DELETE /api/hb/order gained two fields

deferred and deferredCount are a subset of failed, not a new bucket taken out of it. A caller that reads failedCount to decide whether to re-issue keeps behaving exactly as it did, and nothing ever claims an order is cancelled before it is.

They are counted out of failed and never out of canceled, because those orders are still open at the moment of the reply. What the number says is that a retry will find them gone, rather than that the venue refused them.

Added

• Added a shutdown hook on Hummingbot's stop and exit commands, installed lazily from the connector's constructor. Importing the client at connector-module import time would run inside Hummingbot's own connector discovery, which is the most import-cycle-prone moment in the process; by the time a connector is constructed the client is fully loaded, and that is still long before any strategy can place an order. Nothing in it may raise — a trading process must not fail to start because a convenience hook could not be installed.
• Added the same hook to the perpetual connector, for a perp-only bot that has no spot connector to install it.
• Added a batch-cancel sweep to cancel_all, with its own patient retry schedule (5 attempts over 15s) that is deliberately kept during a shutdown — unlike the per-order schedule, which is deliberately shortened. One request that clears a whole market has no queue behind it, so waiting through a rate-limit window costs nothing and buys everything; that is the opposite trade from the per-order case, and 15s still leaves room inside Hummingbot's 20s kill timeout for the per-order pass that follows.
• Added deferred and deferredCount to the batch cancel response.
• Added the exchange order ids the batch door reported cancelling to a per-sweep set, so the per-order pass costs nothing for them. It is scoped to one sweep: an id surviving into a later cancel_all would short-circuit a cancel that genuinely needs to happen.

Changed

• Changed cancel_all's shutdown flag handling from save-and-restore to depth counting. The flag is process-global and the method is not exclusive — a strategy can sweep while a stop is already running, and Hummingbot holds one connector across several markets. Two overlapping calls that each saved "the previous value" would have the inner one restore False underneath the outer one, putting the rest of a shutdown back on the 51-second schedule, which is the bug the method exists to remove. A counter has no such ordering hazard: the flag goes down only when the last sweep leaves, and a real SIGTERM (which sets it directly) is never cleared by us at all.
• Changed KIT_VERSION to 2026.09.11.2 in both connectors.

Fixed

• Fixed clock skew to be measured against the request's arrival rather than whenever the signature check got to run. Covered in Upgrade Notes.
• Fixed the failed-signature floor to skip an expired key, an administrator-disabled key and a late arrival, while still charging a replayed nonce and an IP-blocked request. Covered in Upgrade Notes.
• Fixed a real bot's schema-invalid body being charged as a forged signature, by deferring the body check until after the signature verifies. Covered in Upgrade Notes.
• Fixed a batch cancel on a cross-process install reporting canceledCount: 0, failedCount: 20 for a ladder that was cleared moments later. It was measured doing exactly that. Covered in Upgrade Notes.
• Fixed stop and exit in Hummingbot's terminal not selecting the shutdown cancel schedule. Covered in Upgrade Notes.