Crypto workflow checkpoint recovery restores saved execution state, then checks a durable journal of external actions before continuing. A checkpoint can identify the next step without proving whether an order sent just before a crash took effect. Reuse recorded observations for completed work, reconcile ambiguous actions under their original identities, and check current permissions before an unsent action. Recovery is complete when the next permitted action follows from both saved progress and verified external state.
A process can restart while its exchange orders keep working. Moving the workflow cursor backward does not move the account backward. The difficult case is the gap between an external action happening and its result becoming durable locally.
This article describes a general recovery design and an illustrative failure drill. Framework and exchange behavior is attributed where relevant; the design does not establish that KLINEO implements these recovery controls.
What a checkpoint can actually recover
A checkpoint records progress: the run identity, saved values, completed work and remaining work. It needs the workflow revision and enough preserved outputs to interpret that progress consistently. A status such as “entry node incomplete” describes the orchestrator's knowledge, rather than the exchange's account state.
LangGraph's Graph API saves state at super-step boundaries and retains successful task writes when another node in that step fails. Its checkpointer documentation distinguishes these pending writes from full state snapshots. An operator should inspect both forms of saved progress before deciding which work must run again.
Storage settings also matter. The same documentation describes synchronous persistence before the next step, asynchronous persistence while the next step runs, and persistence on exit. A checkpoint that exists only in an uncompleted write cannot establish a durable recovery point. Ask which write was acknowledged by the persistence system before the process failed.
The historical replay guide concerns reconstructing a recorded decision for investigation. Operational recovery has an additional responsibility: determine what can happen next against an account that may have changed during downtime.
Connect saved progress to an external-action journal
Maintain an action journal alongside the checkpoint. Here, “journal” means durable records of attempted external changes, rather than free-form diagnostic logs. The records must be discoverable even when they are newer than the last full checkpoint.
| Recovery record | Minimum useful contents | Question it answers |
|---|---|---|
| Run checkpoint | Run and checkpoint IDs, graph revision, saved outputs, next tasks | Where did durable orchestration progress stop? |
| Completed observation | Request identity, response, source time, receipt time, parser version | What evidence did the completed decision receive? |
| Action intent | Stable action ID, account and environment, exact authorized parameters | Which external change belongs to this run? |
| Dispatch history | Attempt identity, ownership, dispatch-start record, response or timeout | Could an external effect already exist? |
| Reconciliation receipt | Matched external ID, status, fills, observation time, unresolved conflicts | What is now established about that action? |
Keep an entry, its protective order and a notification as separate actions. A notification failure should lead back to its own record. Restarting the whole sequence to deliver a message can repeat an entry that already filled.
The idempotency guide explains stable trading-intent identity. The checkpoint-specific requirement is to consult that identity's journal before executing a step whose completion receipt is missing. A missing receipt means “investigate,” unless the system can positively establish that dispatch never began.
Put a durable boundary before dispatch
Persist the authorized intent before sending its external request. Record a dispatch-start transition before the request can leave the worker. Once that transition exists, recovery treats the attempt as potentially effective until evidence resolves it.
If local checkpoint references, action records and reservations share a database, use an appropriate transaction to commit their related changes together. PostgreSQL's transaction tutorial explains all-or-nothing database updates. That local atomicity does not make an exchange request part of the transaction. There remains a failure window after dispatch and before the response is saved.
The opposite window also exists: the process can die after recording dispatch-start but before sending anything. Recovery may conservatively investigate an action that never left. Prefer that explainable hold to treating every missing result as permission to submit again.
Separate calls with external effects into independently recoverable tasks. LangGraph's Functional API guidance says completed task results are restored during resumption, while a task that started without finishing may run again. It recommends idempotent operations or checking existing results. A task wrapper therefore helps preserve results; the action's external semantics still determine whether re-execution is safe.
Keep one recovery owner for the affected run. An expired worker lease does not prove that the earlier worker never dispatched. Block an obsolete worker from advancing local state, and retain uncertainty about any request it could already have sent.
A checkpoint that says “submit” after the order filled
Consider a hypothetical workflow with an approved entry action E1, a later protection action P1 and a reporting action N1. The identifiers and times below are invented. This fixture tests state transitions, with no price, fee or performance assumptions.
| Time | Durable local evidence | External state and required recovery |
|---|---|---|
| 09:00:00 | Checkpoint C17 says entry is next; E1 contains the approved request | No entry has been dispatched. |
| 09:00:01 | E1 moves to dispatch-started; that write commits | The worker sends the entry with its recorded client identity. |
| 09:00:02 | The response has not been saved | The fictional venue fills order O77; the response is lost and the worker crashes. Local execution knowledge remains unknown. |
| 09:00:20 | Recovery loads C17 and the newer dispatch record for E1 | It routes to lookup, rather than following the checkpoint directly into another submission. |
| 09:00:22 | Lookup links E1 to filled order O77; the receipt commits | Recovery can record entry completion with its executions. Protection remains a separate obligation. |
| 09:00:23 | Checkpoint C18 records the recovered entry and selects P1 | The protection path checks actual exposure and the applicable authorization before proceeding. |
The correct trace contains one entry dispatch. The stale checkpoint does not authorize a second one. If recovery crashes after committing the reconciliation receipt but before saving C18, another restart finds the receipt and advances orchestration without sending E1 again.
This failure window is realistic even though the fixture is fictional. Binance Spot's general REST API information says a processing timeout can leave execution unknown and instructs clients to query status if it has not appeared in the user data stream. Its documented behavior supports the uncertainty rule; it does not establish another venue's lookup or retry contract.
Use the order-timeout reconciliation procedure for the detailed account, order and fill investigation. Checkpoint recovery must select that procedure instead of silently treating an incomplete node as an unexecuted order.
Preserve past observations and obtain current evidence separately
A completed research task should retain the response and interpretation used for its decision. Re-running retrieval or an AI call can introduce new evidence and change the branch after an entry has already occurred. Save the selected source bundle, its availability times and the resulting output. The retrieval-ranking guide explains why saving a search query alone cannot preserve which evidence reached the model.
Current account observations serve a different purpose. Before an unsent external action, recovery may need a fresh order status, position, balance or trading-permission check. Record that as a new recovery observation with its own timestamp. Do not overwrite the original observation or describe the fresh result as what the earlier decision knew.
An approval can also expire during downtime. Restoring its saved “approved” value preserves history; it does not renew its validity. Reassess unsent actions against their actual authorization. Meanwhile, expiry does not eliminate the duty to reconcile an action already dispatched or manage exposure under the applicable policy.
For a DCA workflow, the restart budget-ledger guide covers rebuilding commitments. Here the boundary is broader: recover the lifecycle of each action before allowing the workflow to consume any restored budget.
Validate every crash boundary
Run a controlled drill with mocked external writes or a suitable test environment. Inject failure before the intent commits; after the intent but before dispatch-start; after dispatch-start without a saved result; after the receipt but before checkpoint advancement; and after advancement but before the next action. Within the unknown-result window, separately simulate no send and remote acceptance. Also fail the protection and reporting steps independently.
For each boundary, specify the expected action identity, durable records and next permitted transition. Verify the external-action trace as well as the final workflow label. In the fixture, two restarts after the missing entry receipt must still link to one E1 and one O77; no recovery branch may create another entry merely to finish the node.
Test conflicting evidence too. A restored backup may predate an external action. Absence from that old journal cannot prove that no request happened. A different graph revision may interpret saved fields differently. Two recovery workers may both try to claim the same action. Each case needs either a verified continuation or a visible hold with an owner and a reason.
Keep these crash fixtures alongside prompt regression tests, while scoring different outcomes: prompt checks assess responses; recovery checks assess durable transitions and attempted external actions. A plausible explanation cannot substitute for the expected trace.
Before enabling automatic resumption, select one action that changes an account and inspect all five boundaries in its drill. Require a receipt that names the restored checkpoint, journal records consulted, current external evidence and next permitted action. That gives the next restart something reliable to recover.