Trading bot health checks: heartbeat vs execution health

Crypto trading bot monitoring should check whether the bot can complete its intended work, as well as whether its process is alive. Track relevant inputs, completed decisions, unresolved orders and recovery progress separately. A recent heartbeat cannot make stale account state or an unknown order safe to ignore. Define which failures block new entries and what evidence permits a restart or resume.

A bot can answer a health endpoint every five seconds while its decision worker is stuck. It can receive current prices while private order queries fail. The operational question is whether the next action has the evidence and controls it needs.

Give each health check a job

Start with five separate observations. Do not combine them into an average score: four passing checks cannot cancel one unresolved order.

The distinction between liveness and readiness has an established infrastructure counterpart. Kubernetes uses liveness probes to decide when to restart a container and readiness probes to decide whether it should receive traffic. A container can remain running while it is not ready. These are Kubernetes semantics; a bot still needs application rules for trading eligibility. Kubernetes probe documentation.

For a bot, readiness for new entries might require fresh inputs, completed scheduled work, current risk checks and no unresolved submission. That is an example policy to design and test, not a universal standard. Keep the underlying checks visible so an operator can see why readiness changed.

Measure completed decisions, including decisions to do nothing

The absence of trades is ambiguous. A healthy strategy may find no qualifying entry for days. A broken scheduler may produce the same empty trade history.

Record expected work separately from completed work. For each scheduled evaluation, retain its intended boundary, start time, completion time, configuration version and outcome. Useful outcomes include no signal, risk rule blocked entry, order intent created, input unavailable and evaluation failed. An input failure is a completed diagnostic attempt, but it does not qualify as a successful strategy evaluation with valid inputs.

A heartbeat should not update the last successful decision timestamp. Nor should logging the start of a run count as completing it. If the worker keeps starting jobs that never finish, oldest pending age and the number of unfinished runs expose the problem more clearly than a busy activity feed.

For an event-driven bot, define progress relative to eligible triggers. If no trigger was expected, an unchanged evaluation count may be normal. If a completed candle or accepted webhook should have caused work, retain that trigger's identity and its final disposition. Otherwise, monitoring can mistake legitimate silence for a stall or miss a dropped trigger entirely.

Choose deadlines from the intended schedule and the consequences of late work. An hourly review and a short-lived execution condition need different policies. The market data freshness guide explains how to define the required input age; the progress check adds whether the bot actually finished using those inputs.

Reconstruct a bot that looks alive but is blocked

Consider a hypothetical bot with a 15-second evaluation schedule. Its invented policy permits a heartbeat age of at most 10 seconds, a relevant quote age of at most 3 seconds and a successful decision age of at most 30 seconds. Any unknown submission blocks new entries immediately. An unknown state lasting more than 20 seconds also alerts the operator. This escalation clock starts when the result becomes unknown, rather than when the request was sent.

These numbers are teaching inputs, not recommended limits or KLINEO defaults. All timestamps below are UTC, with aligned clocks assumed for this illustration.

At 12:00:40, heartbeat age is 40 − 38 = 2 seconds and quote age is 40 − 39 = 1 second. Both pass the example policy. Successful decision age is 40 − 0 = 40 seconds, which exceeds 30 seconds. The order result has been unknown for 40 − 14 = 26 seconds since the timeout, which exceeds its 20-second escalation deadline.

The bot has two passing observations and two operational failures. It is not ready for another entry. Notice that the 20-second deadline controls escalation, not permission to trade: the unknown result blocked entries at 12:00:14.

This uncertainty is a documented possibility on an actual venue. Binance Spot states that its processing timeout does not necessarily mean the matching engine failed to act. If request status has not appeared in its User Data Stream, its documentation directs clients to query the status. That is a Binance Spot instruction, not permission to assume every venue handles a timeout identically. Binance Spot REST API information.

Suppose the identified order's state is recovered at 12:00:50. The unknown count becomes zero, but any known open order or filled position still exists. The bot remains in recovery until it has current account evidence and completes the required review. At 12:01:00, a valid evaluation, all remaining checks and the chosen resume policy can permit readiness. A passing heartbeat alone cannot cause that transition.

Make alerts describe lost capability

An actionable alert says what the bot cannot establish, when that condition began, which account or market is affected and who owns the response. “Private order state unresolved for 26 seconds; new entries blocked” is more useful than “bot unhealthy.” Use a protected account label rather than secrets or a raw account dump.

Google's SRE guidance separates externally visible symptoms from internal diagnostic evidence and distinguishes what is broken from why it broke. Applied here, missed completed decisions or unresolved exposure are symptoms. CPU usage, queue depth and provider errors help investigate the cause. Google SRE monitoring guidance.

Keep one incident record as the condition evolves. If twenty bots share a failed dependency, an operator needs the affected scope and individual exposure records, not twenty indistinguishable notifications. Continue recording each bot's state even when notifications are grouped.

Also record the collector's own last successful observation. If the monitor has not fetched state recently, display that uncertainty instead of preserving its last green result indefinitely. Test alert delivery separately from the worker's ability to write a log line. A bounded test notification can verify delivery without submitting a live order.

Separate pause, restart and exposure management

Blocking new entries does not cancel existing orders or close positions. Define the permitted management path separately, including what happens if automated cancellation or exit cannot be confirmed. An operator may need the venue's official controls and an accurate handoff record.

Restarting a process can resolve some local faults, but it cannot establish what an exchange did with an earlier request. Before resuming, restore the saved request identities, inspect unresolved work and recover authoritative account state. Do not treat the loss of an in-memory queue as proof that nothing remains outstanding.

Kubernetes explicitly cautions that poorly designed liveness probes can cause cascading failures. For a trading deployment, that supports a careful separation between faults that justify a local restart and dependency failures that should leave the worker paused while recovery continues. The trading rule is an operating design choice, not a Kubernetes guarantee. Kubernetes liveness cautions.

Moving to another venue is a separate decision with its own account state and execution constraints. The exchange failover guide examines that handoff. A secondary connection does not inherit knowledge of the first venue's unknown orders.

For an on-chain workflow using ERC-20 tokens, readiness also depends on the actual execution prerequisites. ERC-20 defines allowance for a particular owner and spender; the remaining amount needs its own check. ERC-20 standard. A responsive RPC provider alone cannot establish that permission. Use the token approval and allowance guide for that separate check. This is general on-chain education, not a claim that KLINEO offers token approval management.

Use a worksheet before calling the bot ready

Complete this record for each deployed bot, including its venue, account, product and configuration version. The API connection health guide supplies the underlying account and permission checks; this worksheet assigns their operational consequences.

Validate the worksheet with controlled faults in a supported test or paper environment. Keep heartbeats running while blocking decision completion. Let public prices update while private reads fail. Simulate a timed-out submission and confirm that recovery clears only the identified uncertainty. Stop the collector and verify that its old status becomes visibly stale.

For each exercise, compare three things: the displayed state, whether another entry is permitted and the operator's received evidence. A warning badge with an entry path still open fails the intended control. A successful restart without recovered order state fails the recovery requirement.

Keep the final readiness statement concrete: the required work completed by its deadline, the necessary inputs are valid, and every outstanding request has an understood state. Attach observation times and a response owner. That record gives the next operator something they can verify before another action is allowed.