Crypto calibration and Brier scores

Crypto probability calibration checks whether predicted event probabilities match observed frequencies; the Brier score measures their average squared error against binary outcomes. Use both: a forecast can match the overall event rate yet offer little separation between situations. Compare scores on the same unseen observations against a base-rate forecast fixed before evaluation, and keep probability quality separate from trading profitability.

A percentage needs a defined event before it can be scored. The confidence-score audit covers that interpretation problem. Here the question is narrower: once genuine probability forecasts exist, what should a reproducible scoring worksheet contain?

Fix the scoring contract

For an illustrative crypto task, predict whether the next completed hourly close on one specified spot market exceeds the current completed close. Record each forecast after receiving the current finalized candle and before the next outcome becomes known. A higher close is outcome 1; an equal or lower close is outcome 0. This is a direction label, with no implied fill or profit.

Keep the venue, instrument, interval, label rule, forecast timestamp and model version in the ledger. Record probabilities as fractions between zero and one. A value of 80 must not enter the calculation where 0.80 belongs.

For each valid observation, calculate (p − y)², where p is the predicted event probability and y is the binary outcome. The Brier score is the arithmetic mean of those losses. This article uses the binary convention ranging from zero to one; lower is better. Some implementations sum errors across both classes and produce twice that value. The official Brier score API documentation specifies these scaling conventions. Store the convention with the result.

The score is strictly proper: its expected value favors reporting the event probability honestly. If the true event probability were q, the expected loss would be (p − q)² + q × (1 − q). Only the first term changes with the reported probability, and it is minimized at p = q. This is an incentive property under the stated event model, not proof that an estimated probability is correct.

Thresholding first discards information. Forecasts of 0.51 and 0.99 may both become a positive signal at a 0.50 cutoff, but receive very different penalties when the event fails. Use precision and recall for the separate question of which labeled opportunities a chosen threshold captures or misses.

Two calibrated forecasts can have different scores

The following original hypothetical ledger has ten observations. It is deliberately tiny so every value can be checked; it is not market evidence or a KLINEO result. Assume both forecast procedures were fixed before these outcomes arrived. Group names describe their common observation sets, not categories selected after seeing outcomes.

Model A's two observed event frequencies are 0.20 and 0.80. They exactly match its probabilities in this constructed sample. Model B always predicts 0.50, and five of ten events occur. Its single forecast value also matches its observed frequency.

Yet their scores differ. In Model A's first group, four failures each incur 0.04 and one event incurs 0.64. The total is 0.80; the second group has the same total. Model A's Brier score is 1.60 ÷ 10 = 0.16. Every Model B observation incurs 0.25, giving a score of 0.25.

Model A separates situations with different observed frequencies; Model B does not. Calibration alone does not reward that separation. A finite sample's exact frequency match also does not establish population calibration. Five observations at each Model A probability provide very little evidence.

The scikit-learn calibration guide explains why Brier loss reflects reliability, resolution and outcome uncertainty together. It should not be labeled a pure calibration score.

Reconcile the discrete decomposition

For these exact repeated probability values, the familiar decomposition can be checked directly. Reliability is the count-weighted squared gap between each probability and its observed event frequency. Resolution is the count-weighted squared gap between each event frequency and the overall event rate. Uncertainty is the overall rate multiplied by one minus that rate.

For Model A, resolution is 0.5 × (0.20 − 0.50)² + 0.5 × (0.80 − 0.50)² = 0.09. The identity reconciles its raw score. This exact discrete calculation must not be assumed to hold unchanged after arbitrary binning of continuously varying probabilities.

Lock a benchmark before the test

Suppose a separate earlier development period contained 40 events among 100 eligible observations. A predeclared constant benchmark therefore predicts 0.40 for every later observation. On the ten-row evaluation ledger above, five events incur 0.36 each and five failures incur 0.16 each. Its Brier score is (1.80 + 0.80) ÷ 10 = 0.26.

Define Brier skill score as 1 − model score ÷ benchmark score, following the relative-score convention in the ECMWF forecast verification guide. Model A's skill score against this fixed benchmark is 1 − 0.16 ÷ 0.26, approximately 0.3846. Model B's is approximately 0.0385. Positive means lower squared error than that benchmark on these observations; zero means equal; negative means worse. It is not a percentage improvement in returns.

Using the evaluation period's realized 0.50 rate instead would produce a 0.25 constant-forecast score. That is a useful retrospective reference for the decomposition, but it was unavailable at the evaluation start. Keep its role separate from the forecastable 0.40 benchmark.

A rolling base-rate benchmark is also possible if its window and update timing are fixed and use only outcomes already available. Preserve every benchmark prediction, rather than reconstructing it from later data. The walk-forward testing guide explains that availability boundary.

If benchmark loss is zero, the skill-score ratio is undefined. Report the raw scores and the zero denominator. If there are no valid outcomes, even the mean Brier score is undefined; an empty evaluation is not a perfect score.

Make calibration bins auditable

Real models seldom emit just two probability values. For a conventional reliability table, declare equal-width or equal-count bins and their endpoint rules. Show count, mean predicted probability and observed event frequency in every nonempty bin. Use the mean prediction, rather than assuming the bin midpoint represents its contents.

Changing bin boundaries can change the apparent gaps without changing the original forecast losses. Wide bins can hide opposing errors; narrow bins can leave unstable estimates. The original research on stable reliability diagrams documents this sensitivity and proposes an alternative based on isotonic regression. Its evidence concerns probability evaluation, not crypto profitability.

Retain the unbinned pairs for scoring and show sparse bins honestly. Do not treat an empty bin as an observed zero event rate. For uncertainty, account for shared market episodes and overlapping horizons; the sample-size guide explains why a large row count can exaggerate independent evidence.

Keep a scoring receipt

A missing next-hour candle is a realistic failure case. It does not establish that the price was unchanged or that the event failed. Apply a predeclared unresolved-label policy, report exclusions for each model, and investigate whether missingness concentrates in difficult periods. The missing-candle and zero-volume guide distinguishes an absent record from an explicit no-trade interval.

Fit any recalibration mapping on separate past data, then evaluate the frozen mapping on later outcomes. Repeatedly changing bins, models or mappings after inspecting the test turns it into development data. These checks assess a probability forecast; position sizing, costs and execution require separate evidence. This AI-assisted educational worksheet follows the editorial policy; every numerical ledger here is hypothetical.