Trading bot single-writer control gives one controller authority to change a defined account scope. A lease selects the current owner, while a fencing token lets the submission boundary reject an older owner that resumes after expiry. Neither mechanism resolves an order already sent to an exchange. Safe handoff also requires durable intent records, exchange reconciliation and a rule for when the replacement may resume trading.
A bot can lose leadership without stopping. A suspended process, delayed network packet or stalled virtual machine can resume after another worker has taken over. Both processes may then believe they can place orders. This split-brain problem matters during deployments and recovery, even when the normal scheduler starts only one worker.
The architecture below is an educational design, with hypothetical timings and quantities. It does not establish that KLINEO implements a particular distributed lock, fencing service or exchange gateway.
Define the authority before choosing a lock
Write down the resource that must have one writer: the venue, account, environment, product and controlled order or strategy scope. A worker identifier alone says which process is running; it does not define which account changes that process may authorize.
Two writers on different symbols can still compete for the same quote balance or account risk limit. If those resources are shared, coordinate their reservations at the shared boundary. The competing order ownership ledger explains why available account balance does not allocate authority between strategies.
Four controls answer different questions:
| Control | Question it answers | Boundary it leaves open |
|---|---|---|
| Ownership lease | Which worker currently holds the renewable claim? | An old worker may resume without noticing expiry. |
| Fencing token | Is this command from the activated ownership generation? | A previously admitted command may already be traveling to the exchange. |
| Intent idempotency | Has this approved action already been attempted or completed? | Different intents can still conflict over inventory. |
| Exchange reconciliation | What orders, fills and exposure actually exist? | Observing state does not grant permission for a new trade. |
The workflow idempotency guide develops the intent-record layer. Keep that layer when adding leadership control; a fresh owner does not make an old purchase a fresh intent.
Why a lease can expire while its worker survives
A lease is a time-limited ownership claim that must be renewed. Failure to renew permits takeover. The old worker might have crashed, or it might merely be unable to communicate with the lease service.
AWS describes heartbeat-based leases and warns that a process can pause between checking a lock and performing work. Its implementation discussion distinguishes local elapsed durations from synchronized wall-clock time. These are general distributed-systems concerns, not a recommendation to implement a custom timer for trading. See AWS on leader election.
Checking ownership immediately before an order helps detect ordinary failures, but leaves a gap: the worker can pause after the check. Increasing the lease duration changes how long takeover waits; it does not prove that a paused worker has stopped.
Likewise, a heartbeat thread that remains healthy while execution is stuck can preserve ownership without useful progress. Monitor both ownership renewal and the work that ownership is meant to authorize. An apparently healthy process is not evidence that its last order outcome is known.
Put the fence at an enforced boundary
A fencing token is an ordered ownership generation, such as 17 followed by 18. A new owner receives a later generation. The protected receiver must reject commands from an older generation after the newer one is activated.
Hazelcast documents this pattern for its FencedLock: a paused client can lose its session, and an external resource uses the token to reject stale writes. Its documentation also warns that the CP Subsystem defaults to an unsafe mode with weaker guarantees unless enabled. Those details belong to that implementation; a lock name alone does not establish its operating guarantees. See Hazelcast fencing tokens.
For a proposed trading gateway, activating generation 18 and recording whether a command is admitted must share a serialized, durable authority boundary. Otherwise one process can check generation 17, another can activate 18, and the first can still record its command afterward. The acceptance record should include scope, generation, intent, frozen order parameters and submission state.
The worker must not be able to bypass that gate with exchange credentials. Apply the boundary to cancellations and amendments as well as entries. A stale cancellation can remove protection even when a stale purchase is blocked.
A token does not announce its own expiry. A receiver that remembers only the highest token it has seen may still accept 17 before it sees 18. Make installation of the new generation part of takeover, before the replacement receives trading permission. Keep generations durable and ordered across restarts; resetting a counter can make an old credential look current again.
If the authority store is unavailable, the gate cannot establish a new writer's permission. Hold new entries instead of accepting a worker's cached leadership claim. Preserve the separately declared response for existing exposure, and make the loss of authority evidence visible to the operator.
Follow a paused worker through takeover
Consider a hypothetical controller whose last successful lease renewal at second 0 permits ownership until second 10. Worker A holds generation 17. Renewal stops when A pauses at second 4. These are teaching timings, not suitable production defaults.
| Time | Ownership event | Required submission behavior |
|---|---|---|
| 0 seconds | A owns generation 17; lease ends at 10 seconds. | A may submit approved intents through the gate. |
| 4 seconds | A pauses after preparing an order. | Its prepared command remains attributable to generation 17. |
| 10 seconds | The lease expires without renewal. | Expiry permits acquisition; it does not prove A has stopped. |
| 11 seconds | B obtains generation 18. | B starts recovery rather than opening new exposure. |
| 12 seconds | The gate durably installs 18 in recovery mode. | Generation 17 commands are rejected; new entries remain held. |
| 14 seconds | A resumes and presents its delayed generation 17 command. | The gate rejects it before exchange submission. |
| 15–16 seconds | B resolves prior attempts and verifies current orders and exposure. | New approved intents may resume only after the recovery gate passes. |
At second 14, A may have stale local state saying it is the leader. The receiver's durable generation makes that belief irrelevant to admission. Logging the rejection is useful, but the acceptance criterion is that no exchange request escapes through that rejected path.
Recovery mode also needs an explicit controller for existing protective orders and urgent risk actions. Holding new entries must not silently abandon known exposure. Record who may take those actions and under which generation.
A fence cannot pull back an order already sent
Now change one fact in the example: A's request passed the gate and went onto the network at second 3, before its pause. Installing generation 18 at second 12 cannot withdraw those bytes from the exchange. A timeout or missing response leaves the result unresolved.
Suppose a hypothetical spot account starts with 8 ABC and the approved intent buys 2 ABC. Ignore fees, transfers and other activity to isolate the state change. If the old request filled, the account holds 10 ABC. If B blindly repeats the purchase and that also fills, it holds 12 ABC. Leadership transfer did not authorize the extra 2 ABC.
Binance explicitly states that a matching-engine timeout can leave execution status unknown and directs users to query status if the result is absent from the user data stream. See Binance Spot REST timeout behavior. A timeout is therefore not a clean boundary between the old and new owners.
Binance Spot also documents its newClientOrderId as unique among open orders, with reuse possible once the earlier order is filled. See the Spot new-order parameters. Adding a generation to that identifier can help attribution, but the cited interface does not promise to compare your ownership generations. Nor does a different identifier make retrying an unresolved intent safe.
Before enabling B, the handoff must establish that old workers cannot send further requests through any path and account for every request already admitted. If the gate itself has multiple dispatchers, its takeover needs the same scrutiny. An unresolved send, including one delayed before reaching the venue, stays unresolved until venue-specific evidence or a documented validity boundary settles it. An empty open-order list alone cannot show that nothing filled.
Use the order timeout reconciliation procedure to connect intent records with order history and executions. Canceling an identified open order is another asynchronous action to reconcile, not a way to erase fills already received.
Use a handoff worksheet with observable gates
For each ownership change, preserve the following evidence:
| Worksheet item | Evidence needed before the next step |
|---|---|
| Authority scope | The precise account resources and operations covered. |
| Old and new generations | A durable, ordered transition and the reason for takeover. |
| Admission boundary | Proof the new generation is installed and stale commands fail there. |
| Credential and dispatch paths | Confirmation that an old worker cannot bypass the boundary or later send a queued request. |
| Unresolved attempts | Original intent IDs, venue identifiers, request validity and known outcomes. |
| Current account state | Reconciled fills, balances, positions and retained protective orders. |
| Resume decision | Named authority, passed conditions and permission for specific future actions. |
A fluent status message saying “leadership recovered” does not supply this evidence. The explanation faithfulness guide shows how to tie a narrative to the actual decision trace. Recovery reporting should point to the generation change, stale-command rejection and order outcomes it summarizes.
Operator takeover also needs a fresh decision basis. An outage during a losing position does not authorize an extra trade to repair the experience. The disposition effect and sunk-cost comparison helps distinguish the current decision from attachment to entry price or past expenditure. Keep operational repair and economic authorization explicit.
Test the gaps between states
In a controlled environment, pause A after its local ownership check, activate B and resume A. Then repeat with A paused after gateway admission and after the exchange has accepted an order but before the response is saved. Test a gate restart, a delayed cancellation and an attempted direct submission outside the gate.
Inspect durable admission records alongside exchange orders and fills. Check that generations never move backward, stale commands create no new exchange attempts, and unresolved earlier attempts block conflicting replacements. These drills test a proposed implementation; the timeline above is a checked model, not a live failover result.
The concrete handoff criterion is one explainable chain from current authority to every admitted action, with prior wire attempts resolved before overlapping authority resumes. If that chain has a gap, leave the affected actions held and investigate the gap before granting a replacement permission to trade.