A reproducible crypto backtest preserves the data, strategy, software environment and random scenarios needed to repeat a run. A fixed random seed is only one part of that record. For a fair comparison, run both strategy versions against the same identified scenarios, then examine their paired differences. Repeating a result does not establish that its data, execution assumptions or trading logic are valid.
A seed is useful when the test includes randomness: sampled trading costs, resampled observations, randomized model initialization or a random search. A historical engine that contains no random operation should not need a seed to repeat its output. If such an engine changes between identical runs, investigate the inputs and event processing before adding a seed field.
What the random seed actually fixes
A pseudorandom generator turns an initialized state into a sequence of draws. The seed helps initialize that state. The number does not identify the market data, strategy configuration or meaning assigned to each draw.
NumPy's random stream compatibility policy makes reproduction conditional on matching the generator, seed, calls and arguments, build, environment and machine. Array size also counts as an argument. A saved seed alone therefore cannot promise identical draws after a library upgrade or a code change that consumes the stream differently.
The Monte Carlo trade-sequence guide explains what different sampling rules preserve and change. Record that method before its seed.
Keep the generator's explicit name alongside the seed rather than recording only “default random.” Preserve generated scenario files when exact replay matters. A scenario file records what the engine received, while the seed and generation code explain how that file was created.
Decide what equality means before rerunning. Exact scenario identifiers and trade decisions may be required, while a declared numerical tolerance may be appropriate for some calculated values. A tolerance should not quietly excuse a different entry, omitted fee or changed liquidation decision.
Compare the same scenario with the same scenario
Consider this original hypothetical fixture. Two frozen strategy versions, A and B, trade one invented spot instrument over the same interval. Their baseline P&L after declared fixed fees is 180 USDT for A and 220 USDT for B. Aggregate traded notional, counting buys and sells, is 10,000 USDT for A and 20,000 USDT for B.
We apply one additional adverse execution cost rate to that aggregate notional. These four rates are manually chosen fixture inputs, not exchange quotes, generated seed outputs or an estimated probability distribution. Assume the extra charge changes only cash P&L; it does not change fills, sizing or subsequent trades.
| Saved scenario ID | Extra cost rate | A: additional cost | A: final P&L | B: additional cost | B: final P&L | Paired B minus A |
|---|---|---|---|---|---|---|
| S1 | 10 basis points | 10 USDT | 170 USDT | 20 USDT | 200 USDT | +30 USDT |
| S2 | 20 basis points | 20 USDT | 160 USDT | 40 USDT | 180 USDT | +20 USDT |
| S3 | 40 basis points | 40 USDT | 140 USDT | 80 USDT | 140 USDT | 0 USDT |
| S4 | 50 basis points | 50 USDT | 130 USDT | 100 USDT | 120 USDT | −10 USDT |
Additional cost equals aggregate traded notional × basis points / 10,000. For S3, B pays 20,000 × 40 / 10,000 = 80 USDT, leaving 220 − 80 = 140 USDT.
Across this equally weighted fixture, A averages 150 USDT and B averages 160 USDT. The mean paired difference is (30 + 20 + 0 − 10) / 4 = 10 USDT. B's higher turnover explains why its advantage shrinks as the cost assumption rises.
Comparing B under S1 with A under S4 instead gives 200 − 130 = 70 USDT. That subtraction mixes a strategy change with a scenario change. Keeping the scenario ID attached prevents this accidental comparison.
The four rows do not establish a reliable advantage. They expose a cost dependency under a deliberately narrow model. Estimating uncertainty around a mean requires a justified sampling method, covered separately in crypto bootstrap confidence intervals. Changing strategy settings across nearby values answers another question, explored in parameter sensitivity.
Why the same seed can still break a paired comparison
Imagine that A consumes one random slippage draw per order. B adds an extra order after an early signal. With one shared sequential stream, B consumes an extra draw, shifting which later order receives each remaining value. Both runs can be perfectly repeatable with the same seed while later orders face different shocks.
One practical design is to generate the external scenario first and give each observation a stable key, such as scenario, venue, instrument, timestamp and cost component. Each strategy then reads the applicable observation. This is a proposed research design, not a claim about a KLINEO control.
Matching must remain economically meaningful. If A submits a market order and B posts a limit order, copying the same fill price into both does not create a fair execution model. A shared market state can be paired; order outcomes still need their respective rules. Unfilled orders and extra orders must remain visible.
For parallel simulations, assign random streams to stable scenario or job identities rather than whichever worker happens to finish first. NumPy documents SeedSequence spawning and deterministic job identifiers as ways to organize reproducible streams. Save the identity-to-stream mapping and collection order as well as the root seed.
Attach a manifest to the result
A manifest is a small record that lets another run identify its inputs. It should travel with the trade ledger and summary rather than live only in a notebook comment.
| Manifest field | What to save | Failure it helps diagnose |
|---|---|---|
| Data identity | Venue, instrument, dates, interval convention, source files and checksums | A later download replaced or corrected history |
| Strategy identity | Code revision, full parameters, preprocessing and model artifact if used | The displayed name hid a changed rule |
| Execution model | Fees, funding scope, fills, event ordering, sizing and rounding policies | The same signal produced a different ledger |
| Randomness | Generator, seed, scenario-generation revision, saved scenarios and checksums | A seed was reused with different draws or meanings |
| Environment | Dependency lockfile, interpreter, library build, operating system, hardware and relevant settings | A software or numerical change altered output |
| Comparison contract | Scenario IDs, metric definitions, equality rules and accepted tolerances | Two summaries silently describe different experiments |
A checksum establishes file identity, not correctness. Start with the backtest data-quality audit before treating a frozen file as trustworthy.
Keep the manifest free of exchange secrets. Research scenario identifiers and seeds do not require API credentials.
Diagnose the first divergence
Repeat a frozen run and compare intermediate artifacts in order: accepted input rows, prepared features, scenario observations, signals, submitted orders, fills, account ledger and final metrics. The first mismatch usually narrows the investigation more effectively than comparing final returns.
If scenarios match but signals differ, inspect preprocessing, model state and rule evaluation. If orders match but balances diverge, inspect fees, valuation, rounding and event ordering. Record the explanation before accepting a new reference result.
A successful replay earns a bounded statement: this recorded experiment repeated under these conditions. It does not repair look-ahead bias, optimistic fills or selection of favorable seeds. Preserve unfavorable seed results and declare the evaluation set before comparing outcomes. The backtest evidence worksheet addresses the assumptions that repetition alone cannot validate.
Method and limits
This article was prepared with AI assistance from the linked primary software documentation. The paired-cost fixture, manifest worksheet and diagnostic sequence are original educational constructions; arithmetic was checked independently. No market backtest, stochastic simulation or KLINEO feature test was performed. The scenario table explains comparison discipline and supplies no forecast of trading returns.