Crypto API retries should use capped exponential backoff with jitter only after deciding that repeating the operation is safe. Follow the exchange's wait instructions, limit both attempts and total elapsed time, and check the remaining deadline before every request. A timed-out order can already have executed, so its next step is reconciliation rather than another submission. Random delays reduce synchronized traffic; they do not prevent duplicate orders or restore stale trading permission.
The useful question is not simply how long to sleep. It is whether another attempt is allowed, useful and affordable. A balance read, an invalid order and an order with an unknown outcome need different answers. The schedule below is an original hypothetical example for a read-only request, with invented settings rather than exchange defaults.
Classify the operation before its error
A retry policy needs two inputs: what the request could change, and what the failure actually establishes. “Network error” describes what the client observed. It may say little about what the exchange did.
HTTP distinguishes idempotent operations from operations that cannot generally be repeated safely. RFC 9110 cautions against automatically repeating a non-idempotent request without knowledge that its semantics permit this or that the original was never applied. A trading adapter must establish that contract for its particular endpoint. HTTP retry semantics, section 9.2.2.
| Observation | Suggested next branch | What still needs checking |
|---|---|---|
| A documented read fails transiently | Consider another bounded read | Provider rules, freshness deadline and shared capacity |
| Provider returns a throttle or ban response | Apply its required wait and reduce demand | Which account, IP or endpoint scope is affected |
| Request has invalid quantity or missing permission | Fix or escalate the cause | Whether a separately authorized corrected request is appropriate |
| Order submission loses its response | Preserve an unknown outcome and reconcile | Original order identity, status and fills |
| Cancellation loses its response | Query the original order and executions | Whether it filled, remained open or was canceled |
This table is a design worksheet, not a universal mapping of HTTP codes. In particular, a server error on an order request is not proof of rejection. The trading idempotency guide explains how to keep repeated work attached to the original trading intent; backoff supplies timing after that safety decision.
Define backoff, jitter and three stopping limits
Exponential backoff increases the waiting window after consecutive eligible failures. Capping the window prevents it from growing indefinitely. Deterministic waits can still make many clients return together; jitter varies their timing. AWS's original comparison describes this effect and distinguishes several jitter methods. Its simulation is evidence about contention behavior, not a measured exchange performance result. AWS explanation of exponential backoff and jitter.
For this article's full-jitter policy, number retries from zero and calculate:
window(n) = min(maximum window, base delay × 2^n)
Draw the local delay uniformly between zero and that window. With a 0.5-second base and a 4-second cap, the first four windows are 0.5, 1, 2 and 4 seconds. A 4-second window is an upper bound on the random local delay, not a requirement that every client sleep exactly four seconds.
Write three separate limits beside that formula:
- Attempt limit: the total number of requests, explicitly including the initial call.
- Elapsed-time deadline: the latest completion time that still makes the result useful.
- Shared retry budget: the requests or weighted capacity available across workers, including other jobs recovering from the same failure.
A per-attempt timeout is a fourth necessary setting. It bounds one call; it does not bound a sequence of calls and sleeps. AWS discusses timeout coverage, including connection setup, and the load caused by excessive retries. Check what the actual HTTP client times out rather than assuming one setting covers every phase. AWS guidance on timeouts and retries.
For the policy below, admit an attempt only when the count and shared allowance permit it and now + planned wait + per-attempt timeout ≤ absolute deadline. Apply this before sleeping, then recheck the remaining timeout budget just before sending.
Measure local elapsed waits and timeout budgets with a monotonic clock; Python's monotonic-clock documentation describes a timer unaffected by system-clock updates. Store the job's durable expiry separately. After a restart or migration, reconstruct the remaining budget under a verified clock policy instead of treating a machine-local counter as a portable timestamp.
Provider wait instructions override a shorter random delay
The exchange may supply a minimum wait that is longer than the sampled backoff. For Binance Spot REST, the documentation describes Retry-After seconds on 429 and 418 responses; continued requests after throttling can lead to an IP ban. Those are Binance Spot rules, not a template for every provider. Binance Spot limits and response handling.
A client must parse the field using the relevant specification. HTTP's general Retry-After definition permits either a date or an integer delay in seconds. Do not interpret every reset value from every API as the same kind of timestamp. HTTP Retry-After definition.
The example uses planned wait = max(provider minimum remaining, sampled local delay). It never randomizes below a provider minimum. That maximum can still align clients at the same release boundary. Where the provider permits it, additional nonnegative spread after the boundary can reduce that clustering, provided it also fits the deadline. An explicit ban may require suspending the affected work entirely until expiry.
A 12-second read budget, checked attempt by attempt
Assume a hypothetical private read has a 12-second absolute deadline, measured from its initial start. Each admitted request receives a 1.5-second total timeout. The limit is five total attempts, including the initial one. Shared capacity is assumed available so this example isolates timing.
Use fixed illustrative jitter draws of 0.4, 0.6, 1.2 and 2.4 seconds. They are supplied for reproducibility, not reported as a random experiment. After attempt two returns a throttle response, the invented provider requires a two-second wait measured from receipt.
| Attempt | Jitter window / draw | Effective wait before call | Start → finish, seconds | Result |
|---|---|---|---|---|
| 1: initial | None | 0 | 0.0 → 1.5 | Timeout |
| 2: retry 1 | 0.5 / 0.4 | 0.4 seconds | 1.9 → 2.2 | Throttle response after 0.3 seconds |
| 3: retry 2 | 1.0 / 0.6 | 2.0 seconds: provider minimum | 4.2 → 5.7 | Timeout |
| 4: retry 3 | 2.0 / 1.2 | 1.2 seconds | 6.9 → 8.4 | Timeout |
| 5: proposed retry 4 | 4.0 / 2.4 | 2.4 seconds proposed | Would start 10.8; full timeout ends 12.3 | Skipped before sleeping |
After attempt four, the calculation is 8.4 + 2.4 + 1.5 = 12.3 seconds. That exceeds the deadline by 0.3 seconds. The attempt-count limit would permit one more call, but this policy requires room for the full per-attempt timeout, so it stops at 8.4 seconds.
Four requests were sent: the initial call and three retries. The fifth row consumed neither a sleep nor a request. A different explicitly designed policy could shorten an attempt's timeout to the remaining budget; it must still enforce the same absolute deadline. Do not accidentally create that behavior by allowing a request to run past expiry.
Recheck the gate when a sleeping job wakes. Queueing or scheduler delay can consume time that appeared available when the wait began. If no current read was obtained, mark the observation unavailable or stale; returning the last cached value as fresh would hide the failed recovery.
An unknown order follows a different timeline
Suppose an entry request is transmitted, its response is lost and the original signal expires. The execution state is now unknown. Expiry ends permission to initiate fresh trading work under that signal; it does not establish that the transmitted order failed.
Binance Spot explicitly documents an execution-unknown timeout and advises querying status when the result has not appeared in its User Data Stream. It also warns that 5xx responses can leave execution unknown. Binance Spot timeout and server-error semantics.
Keep the original identity and move from submission to unresolved execution, then to evidence gathering. A confirmed fill, documented rejection or other reconciled outcome can resolve the record. Exhausting status-query retries leaves it unresolved; it must not automatically release a new entry or turn an absent open-order row into proof that nothing traded.
The order-timeout reconciliation procedure covers that evidence path. Give reconciliation its own bounded requests and escalation policy. Stopping an entry's retry loop and abandoning an existing exposure are different decisions.
Coordinate workers and retry layers
One logical request can have several retry owners: a workflow worker, an exchange SDK and an HTTP transport. If the worker allows three total attempts and its SDK also allows three per call, one logical operation can generate up to 3 × 3 = 9 wire attempts. Both counts include the first call.
AWS describes retry multiplication across layers and discusses choosing one retry point for suitable operations. Inspect the real client configuration and designate an owner; don't assume a wrapper's attempt counter represents wire traffic. AWS guidance on retry layers.
A shared limiter should account for the provider's scope and request weights. Independent random delays cannot enforce a common quota. The exchange rate-limit budget shows how ordinary demand and recovery demand combine. Avoid letting optional refreshes exhaust the capacity needed to investigate existing orders.
Reducing redundant reads also changes the budget. The REST versus WebSocket market-data comparison helps decide which observations need polling and which can come from a stream with a recovery plan. A stream reconnect still needs its own bounded policy.
Turn the policy into an acceptance worksheet
For each endpoint family, record its side-effect classification, retryable conditions, provider wait fields, base and capped windows, attempt ceiling, per-call timeout, absolute expiry, shared budget scope and exhaustion outcome. Attach current primary documentation and the date checked. Separate public market reads, private account reads and order mutations rather than giving them one default loop.
Then run controlled failure cases:
- Replay the numerical schedule above and verify exactly four wire requests, including the early fifth-attempt rejection.
- Return a wait instruction longer than the remaining deadline. Verify that the client stops without shortening the provider wait.
- Lose an order response while recording a simulated fill. Verify that the workflow reconciles the fill without submitting another entry.
- Restart a worker during a wait and wake another worker late. Verify that attempt accounting and the original expiry survive recovery.
- Enable every actual transport layer. Verify the observed wire count and combined shared demand, not just application log messages.
The VPS versus managed-cloud operating checklist is useful for assigning responsibility for restart state, configuration and recovery evidence. Hosting alone does not preserve a retry budget.
For a failed job, retain a redacted attempt record: operation identity, observed error, timeout, sampled delay, provider minimum, remaining deadline and the reason the next attempt was admitted or refused. That record should let an operator explain both the traffic that occurred and the unresolved account state. Before the next request, require three answers: it is safe to repeat, the provider permits it now, and the remaining time and shared capacity can support it.