Crypto backtest data revisions can change a result even when the strategy code and date range stay fixed. Preserve each downloaded data vintage, compare changed observations and rerun frozen rules on both versions before attributing the difference to a strategy improvement. A reproducible snapshot identifies what a research run used; it does not prove that the same values were available at the historical trade decision.
A useful investigation begins with one question: did the experiment change because the rule changed, the history changed, or both? Keeping only the newest download makes those explanations difficult to separate.
The same historical URL can deliver a new experiment
Binance's public-data repository documents checksum files alongside its archive ZIP files. It also says archives can be updated after issues are discovered, and its update records identify replaced and replacement checksums. A stable archive path therefore does not establish stable contents. Binance archive integrity and update policy
Call each retained version a data vintage: a particular capture of a source over a defined interval. Keep the raw file, its retrieval time and identifying metadata together. Recording a checksum without retaining the corresponding object lets you detect a mismatch later, but cannot recover the missing version.
Separate three research questions:
| Question | Appropriate evidence | What it cannot establish alone |
|---|---|---|
| Can the old result be repeated? | Original input bytes, transformations, rules and execution assumptions | Whether those observations were correct |
| How does corrected history change the result? | Old and new vintages run through the same frozen pipeline | What an earlier live trader could have known |
| What could a historical decision use? | Observations with defensible availability times and the applicable cutoff | That fills or future returns would match a simulation |
The backtest data-quality audit addresses malformed, conflicting and missing records. Revision analysis starts after that audit: two internally valid versions can still disagree.
A one-bar correction can erase a rule comparison
Consider an original hypothetical spot fixture, with prices in USDT. V1 and V2 are two downloaded versions of the same history. Only the signal close changes. These are invented values, not Binance observations or KLINEO results.
| Observation, all times UTC | V1 | V2 |
|---|---|---|
| 10:01 finalized candle close | 102 | 104 |
| 10:02 next candle open | 105 | 105 |
| 10:03 following candle open | 103 | 103 |
Evaluate exactly one signal after the 10:01 candle has closed. Rule A buys one unit if that close is greater than 101; rule B buys one unit if it is greater than 103. An eligible order enters at the 10:02 open and exits at the 10:03 open. Assume sufficient cash, zero fees, zero slippage and no other trades. These deliberately simple execution assumptions isolate the revision's effect.
| Frozen rule | V1 decision and P&L | V2 decision and P&L | V2 minus V1 |
|---|---|---|---|
| A: close greater than 101 | Buy; −2 USDT | Buy; −2 USDT | 0 USDT |
| B: close greater than 103 | No trade; 0 USDT | Buy; −2 USDT | −2 USDT |
| B minus A | +2 USDT | 0 USDT | −2 USDT |
Each executed trade earns 1 × (103 − 105) = −2 USDT before costs. Under V1, changing A to B improves this fixture's P&L by 2 USDT because B avoids the loss. Under V2, both rules take the loss and the difference disappears.
The effect of changing the rule depends on the data vintage. Comparing A on V1 with B on V2 would mix two changes and conceal that interaction. Run all four cells before explaining a supposed improvement. One toy trade demonstrates attribution; it supplies no estimate of an edge or how frequently revisions matter.
For randomized execution assumptions, hold the saved scenarios fixed across these comparisons too. The random-seed reproducibility guide explains why the same seed alone may not supply the same economic scenario.
Compare observations before comparing P&L
Start with the raw hashes. A changed ZIP hash is a reason to inspect the contents, not proof that an OHLCV value changed: packaging metadata or serialization can differ while normalized observations remain equal.
Next compare parsed rows using a complete key: source, venue, market, instrument, interval and opening timestamp under the same convention. Report inserted rows, deleted rows and field changes separately. Retain exact numeric representations where needed; a display rounded to two decimals can conceal a threshold crossing.
Keep the parser and cleaning policy frozen while isolating a vendor revision. If an updated parser interprets timestamp units differently, that is another experimental change. Save raw and normalized hashes so you can distinguish changed source bytes from changed interpretation.
Then trace the first changed downstream artifact: prepared input, indicator state, signal, order, fill or valuation. A revised warmup candle may alter later indicator values even if the traded interval itself contains no changed rows. Do not limit the comparison to the final entry timestamp.
For portfolios, the revision may insert a previously absent asset observation and change allocation eligibility. Preserve the join and missing-asset rules described in multi-asset backtest alignment while making the vintage comparison.
A corrected volume can change eligibility too
Price is only one input. Suppose a separate hypothetical fill policy caps an order at 10% of the reference bar's reported base-asset volume. V1 reports 100 units, allowing up to 10 units; V2 corrects that volume to 60 units, allowing only 6. A requested 8-unit order passes under V1 and fails under V2 if the policy rejects oversized orders rather than clipping them.
The arithmetic is 100 × 0.10 = 10 and 60 × 0.10 = 6. Record which behavior applies; silently clipping would produce another ledger. The volume-participation guide covers capacity assumptions and the limits of using historical volume. This example only demonstrates their sensitivity to a revision.
A saved vintage is not automatically point-in-time history
A file downloaded today can faithfully reproduce today's research while containing corrections unavailable at an earlier decision. That distinction matters for revised on-chain metrics as well as exchange candles.
Glassnode documents point-in-time metrics, historical coverage limits and a delay between a datapoint's computed_at timestamp and its API publication. Its computation timestamp therefore should not automatically serve as a trader's receipt timestamp. Coverage begins when tracking starts for the metric; a modern PIT endpoint does not imply unlimited earlier PIT history. These statements describe Glassnode's product, not all crypto feeds. Glassnode point-in-time metric timing and coverage
For an as-observed test, retain local receipt times where available and explain how they relate to the historical decision cutoff. If the original vintage or availability evidence is missing, label that limitation. A later corrected archive may be useful for a corrected-history experiment without supporting a claim that it recreates an earlier information set. The look-ahead bias audit covers that decision-clock test.
Attach a revision receipt to the rerun
Use this worksheet whenever a historical download changes:
| Receipt field | Record for this comparison |
|---|---|
| Vintage pair | Original and replacement object locations, retrieval times, raw hashes and source notice if available |
| Interpretation | Schema, parser, units, normalization and cleaning versions; normalized hashes |
| Change scope | Added/deleted keys, changed fields, first/last affected times and warmup dependencies |
| Frozen conditions | Rule versions, costs, fill model, sizing, join policy and saved random scenarios |
| Four-cell result | Each rule on each vintage, with trade-count and ledger differences alongside summary metrics |
| Availability limit | Which observations have historical availability evidence and which do not |
Retain the previous result with its previous vintage. Publish a replacement result with a revision explanation rather than overwriting the evidence of how the old result arose. If a material original object is unrecoverable, the honest outcome may be “the prior result cannot be exactly reproduced.”
This AI-assisted article uses the linked primary documentation for provider behavior. Its vintage comparison, volume example and receipt worksheet are original educational constructions with checked arithmetic. No market archive or historical trading result was validated for this article.