A crypto trading bot backup should preserve configuration, execution records and a tested way to recover credentials. Restoring those files does not restore the exchange account to its earlier state. Recover into an environment that cannot submit live orders, reconcile current orders and fills, verify who may trade, then explicitly decide whether new activity can resume. A restore drill passes when those boundaries work together.
Define what recovery has to prove
An archive that opens successfully answers a storage question. A recovered bot also needs to interpret its records, reach its dependencies and understand an account that may have changed during downtime. Treat those as separate acceptance checks.
Start with one named failure: loss of the bot's host and primary database, for example. Specify what survives outside that boundary. A backup on another directory of the lost disk cannot satisfy that scenario. Neither can a recovery procedure whose only copy lives in the unavailable application.
AWS's recovery methodology distinguishes a recovery time objective, or RTO, from a recovery point objective, or RPO. RTO limits the acceptable restoration delay; RPO limits the acceptable age of the recovered data. The workload owner sets both. AWS recovery objectives.
Define “restored” before timing the exercise. For a trading system, useful milestones include records readable, account reconciliation complete and fresh entries eligible. A process starting is an intermediate result. An unresolved account can legitimately remain paused after infrastructure is available.
These are general operating practices for a custom or hosted bot. Establish which exports and recovery controls your provider actually supplies. For the infrastructure responsibility split, use the VPS and managed-cloud operating checklist.
Build a recovery manifest instead of copying everything blindly
Keep a manifest alongside the backup set. It should identify the account and environment, application revision, database schema, backup time, integrity evidence, encryption-key reference and restoration procedure. Record identifiers and locations without placing secret values in the manifest.
| Recovery class | Preserve or reference | Check during restoration |
|---|---|---|
| Application and configuration | Release artifact, dependency versions, strategy parameters, risk rules and schema version | The selected release reads the recovered data and settings correctly |
| Execution and audit records | Intent IDs, dispatch attempts, order IDs, fills, approvals and pause decisions | Records are usable, time ordered and complete to a declared recovery point |
| Credentials and recovery access | Secret references, current lifecycle metadata and protected recovery procedure | Authorized operators can recover access without reviving revoked authority |
| Exchange observations | Historical observations with their timestamps | Current balances, positions, orders and fills are fetched separately |
Record a recovery point for each dataset. A nightly configuration export and a frequently replicated order journal can have different ages. “Backup completed at 08:00” is insufficient if the report does not say which components that timestamp covers.
A checksum helps detect changed bytes when compared with a trusted reference. It does not establish that the chosen backup contains every necessary record or that its contents were correct before backup. Open representative records, validate relationships and test the application against them.
AWS recommends repeatedly testing the ability to find particular recovery points and restore them; its guidance also highlights encryption and cross-account assumptions that testing can expose. Apply that principle to the complete dependency path. AWS data-recovery testing guidance.
Restore without inheriting permission to trade
Use an isolated drill environment with external order writes blocked. A disabled strategy checkbox alone may leave background workers, queued jobs or webhook consumers able to act. Verify the boundary by observing attempted outbound actions against a mock or test connector.
Restore data before enabling live credentials. Keep schedulers and event consumers paused. Mark recovered queues as quarantined until each item has an explicit disposition. A saved instruction to buy at a past candle close is historical evidence; its presence does not establish that the instruction remains timely or authorized.
Preserve audit history while appending recovery findings. Do not overwrite an old “unknown” submission with a fabricated successful receipt. Link the later reconciliation result to the original attempt so a reader can distinguish what the bot knew before failure from what recovery discovered.
If a workflow checkpoint points toward an execution step, consult the corresponding action history before advancing. The checkpoint recovery guide develops that narrower state-machine problem. The backup drill must additionally prove that the checkpoint, history and compatible application can all be recovered together.
Test credential recovery as a separate dependency
An encrypted archive is useful only if the recovery operator can obtain the authorized decryption capability. Draw that dependency explicitly. If restoring the secret store requires a password held only inside the same unavailable store, the runbook has a circular dependency.
OWASP recommends encrypted backups with reduced access rights, regular restore tests and tested emergency access procedures. It also identifies the risk of being unable to retrieve credentials needed to restore an unavailable secrets service. OWASP secrets-management guidance.
For a bot, separate restoring access to the secret service from choosing the exchange credential that should be active now. An old machine image may contain an API key that was later revoked. A restored configuration may reference a retired key version. Resolve that against the current credential inventory and exchange permissions before opening any submission path.
Verify the recovered worker's account identity, environment, approved outbound route and permissions. A successful authentication check alone does not prove it reached the intended account. Record the credential label and version, never the key value, in the drill receipt.
Audit exports can also retain sensitive webhook payloads. Use the webhook log-redaction procedure to decide which evidence can be kept without copying authentication material into every archive.
Worked drill: two recovery points and one missing receipt
Consider a hypothetical spot bot. All quantities, times and targets here are illustrative. Its failure occurs at 08:06:00 UTC. The scenario loses the host and primary database while preserving an offsite database backup, a separate journal replica and the secret service.
The database backup is from 08:00:00. The journal replica has a confirmed recovery watermark of 08:05:58. The chosen RPO targets are ten minutes for the database and five seconds for the journal. The chosen RTO is 25 minutes, measured until the resume gate can be evaluated with complete evidence.
The restored database reports 1,000 quote units and 2 asset units. At 08:05:55, however, the bot recorded and dispatched intent B-46 to buy 0.5 asset units. The simulated venue filled it at 120 quote units per asset; the response was lost. The journal replica contains the intent and dispatch attempt, but no result. Assume no fees or other account changes in this teaching fixture.
| Elapsed time after failure | Recovery milestone | Evidence required |
|---|---|---|
| 2 minutes | Containment complete | Original worker cannot submit; recovery starts with writes blocked |
| 7 minutes | Application and database restored | Correct release, schema and configuration identified |
| 10 minutes | Journal restored | Watermark checked; B-46 recognized as unresolved |
| 13 minutes | Credential path verified | Current credential recovered; account and route checked |
| 17 minutes | Account reconciled | Filled B-46 matched; balances and outstanding obligations checked |
| 20 minutes | Resume review complete | Quarantined work disposed of; current risk and ownership checks pass |
The database recovery point is six minutes before failure, within its ten-minute target. The journal watermark is two seconds before failure, within its five-second target. Recovery takes 20 minutes, leaving five minutes against the illustrative RTO. Passing those timing targets still requires resolving the missing result.
The fill spends 0.5 × 120 = 60 quote units. Current holdings are therefore 2.5 asset units and 940 quote units. Replaying B-46 from the restored queue would buy another 0.5 units, leaving 3 asset units and 880 quote units under the same simplifying assumptions. That would add an unintended 60 quote units of purchases.
The drill passes by finding the first fill and suppressing a second dispatch. Its RPO allowance does not authorize ignoring requests in the final two seconds. Investigate that uncovered interval using current venue evidence; if it cannot be resolved, retain the pause.
Reconcile account evidence before releasing restored work
Binance Spot documents that a processing timeout can leave execution status unknown and may occur despite the Matching Engine processing the request. If the result has not appeared in the User Data Stream, it directs clients to query the request's status. This is one documented venue behavior supporting the drill's uncertainty case. Binance Spot REST information.
Use the connector's supported account, order and trade-history reads. Open orders alone omit an order that has already finished. Match recovered identifiers to venue evidence and cover the interval between the oldest relevant recovery point and the present, including required pagination and account scope.
For Binance Spot specifically, the query-order endpoint accepts an order ID or original client order ID, with the required symbol. Other venues and products need their own lookup procedure. Binance Spot account endpoints.
Check protective orders and reserved funds as well as entries. A restore must not silently remove protection simply to obtain a clean local state. Handle each discrepancy under the existing operating policy, with an owner and evidence. An inaccessible history page or failed query is an unresolved check.
Orders can fill while recovery queries run. Timestamp each observation and repeat the relevant account and order checks immediately before resuming. If the readings cannot be reconciled into a current account view, keep the gate closed.
Before assigning submission authority to the recovered worker, prove the earlier worker cannot continue independently. The single-writer fencing guide explains that handover boundary. Restoring the same credentials onto two machines does not establish exclusive authority.
Keep a receipt another operator can use
Finish the drill with a short worksheet containing the failure tested, surviving dependencies, backup identifiers, per-dataset recovery points, timed milestones, unresolved order IDs and next permitted action. Add the current credential version, writer ownership and the person responsible for each exception.
Include a second test in which a required dependency is unavailable: the decryption service, a newer journal segment or venue history. The expected result is a visible hold with a reason. A procedure that advances only because every dependency happened to work has left its failure behavior untested.
After fixing a failed step, repeat that recovery path and record the new result. Rehearse again when a release changes schemas, storage, secret access or account routing. The useful artifact is a recoverable system plus an explicit decision about what it may do next. Keep the recovered worker paused until the receipt can name both the restored evidence and the current account state.