An AI trading confidence score is not a probability of profit unless the provider defines the predicted event and demonstrates calibration on relevant unseen outcomes. A review score may measure checklist agreement, explanation quality or model preference instead. Before using any percentage, ask what would count as success, over what horizon, and after which costs.
The distinction is practical. “82% confidence” can make two unrelated systems look comparable when they are measuring different things. This article builds an original, hypothetical score audit. It offers no current signal, performance estimate or recommended trading threshold.
First determine what the number describes
A score might represent the model's estimated probability of a next-bar direction. It might measure agreement among rules. It might summarize a reviewer's opinion of an existing trade plan. A language model might simply have generated a number in a response.
Those numbers cannot share a denominator by appearance alone. A score of 82 out of 100 for completeness means something different from an estimated 82% probability that a specified event occurs. Neither automatically gives an 82% probability of a profitable order.
Ask for a sentence that can be tested: “Among comparable forecasts assigned approximately 0.8, the stated event occurred approximately 80% of the time.” That describes calibration. Scikit-learn's documentation explains the relationship between predicted probabilities and observed event frequencies. Scikit-learn probability calibration guide.
Then identify who produced the score, the model or rule version, the source data and whether there was a separate calibration process. An unexplained percentage should be treated as an unlabeled output, not converted into a risk allowance.
Define the event before counting successes
Suppose a model predicts whether a spot market's next completed hourly close will exceed the current completed close. That event is not the same as reaching a target before a stop. A market can close higher after first falling through an exit level.
Define the venue, instrument, observation time, horizon and equality rule. If the two closes are identical, does that count as failure, success or an excluded observation? Decide before inspecting results. Record missing data separately; excluding difficult intervals can make a score appear better.
For a tradable outcome, specify the entry eligibility, fills, fees, exit rule and treatment of unresolved positions. The backtest evidence article shows why a candle alone can leave the order of price events unknown. Direction classification and executable strategy performance are different evaluation problems.
Also distinguish prediction time from outcome time. If a forecast is created halfway through the next hour, it cannot be evaluated as though it existed at the previous close. An apparently precise score loses its meaning when its observation boundary moves.
A worked calibration audit with four bins
The following table is invented. It is a teaching dataset of 100 forecasts with four deliberately uniform scores, not an observed study or a KLINEO result. Each outcome is binary under one fixed hypothetical definition.
| Assigned score | Forecasts | Successful events | Observed frequency | Difference from score |
|---|---|---|---|---|
| 0.55 | 25 | 13 | 52% | −3 percentage points |
| 0.65 | 25 | 15 | 60% | −5 percentage points |
| 0.75 | 25 | 16 | 64% | −11 percentage points |
| 0.85 | 25 | 17 | 68% | −17 percentage points |
Average assigned probability is (0.55 + 0.65 + 0.75 + 0.85) ÷ 4 = 0.70. The observed event rate is (13 + 15 + 16 + 17) ÷ 100 = 0.61. The sample is overconfident on average by nine percentage points.
The increasing event frequencies show some separation across bins in this construction, but the higher labels overstate their observed frequencies. That is useful diagnostic information. It does not establish how the next forecast will behave, and 25 observations per bin leave substantial uncertainty.
Do not repair the table by relabeling today's 0.85 forecast as a certain 0.68. The table is a sample estimate, not a transfer rule. A calibration method needs separate fitting data and evaluation data, along with checks for changed conditions.
Check a proper score without mistaking it for profit
For a binary event, the Brier score is the mean squared difference between a predicted probability and the realized zero-or-one outcome. Lower is better for this scoring rule. It combines aspects of forecast quality, so it should not be read as a pure calibration measure. Scikit-learn Brier score documentation.
Using the constructed table, the first bin contributes [13 × (0.55 − 1)² + 12 × 0.55²] ÷ 100 = 0.062625. The other three contributions are 0.060625, 0.060625 and 0.061625. Their sum is 0.2455.
A constant forecast of 0.61 on this same dataset would score 0.61 × 0.39 = 0.2379. That baseline uses the sample's realized overall rate, so it is an in-sample descriptive comparison, not an available prospective trading model. The exercise shows that appealingly high score labels can perform worse under a proper scoring rule than a simple benchmark.
A low Brier score still does not show a profitable trade. The event may have tiny favorable moves and large adverse moves. Execution costs may consume the edge. Keep probability assessment separate from payoff assessment.
Show how payoff can reverse a conclusion
Consider another invented example: a binary trade that gains 10 USDT on success, loses 30 USDT on failure and incurs two USDT of total costs either way. Assume, solely for arithmetic, that the event probability really is 75%.
Expected net outcome is 0.75 × 10 − 0.25 × 30 − 2 = −2 USDT. A high success probability does not rescue that payoff structure. Under these assumptions, break-even requires p × 10 − (1 − p) × 30 − 2 = 0, so p = 32 ÷ 40 = 80%.
Now change the loss to 10 USDT while keeping the gain and costs. Break-even becomes p × 10 − (1 − p) × 10 − 2 = 0, giving p = 60%. The relevant probability depends on the exact outcome and costs. Neither threshold is a suitable trading rule for readers; real fills and losses need not match the simplified binary assumptions.
This is why a confidence score should never be multiplied directly by account equity to choose position size. Loss severity, correlation, liquidity and uncertainty remain outside that arithmetic.
Keep calibration separate from model selection
If you choose the best calibration method after inspecting the final evaluation period, that period has influenced your choice. It no longer provides an untouched test of the selected process.
Keep a chronological record of training, calibration fitting and final evaluation. Describe any overlap explicitly. Adjacent trading observations may share data and outcomes; a large row count need not mean an equally large number of independent observations.
The original research paper On Calibration of Modern Neural Networks studies probability calibration and temperature scaling in classification settings. It supports treating calibration as something to measure, not something guaranteed by a neural model's label. Its experiments do not establish a crypto trading result. Guo and colleagues, original paper.
Changing a model, market, horizon or data process can also change calibration. Maintain version-specific evidence rather than pooling incompatible scores into one flattering chart. A historically measured association remains uncertain when conditions change.
Count the forecasts the system declined to make
Calibration tables describe the forecasts included in them. They do not reveal how often a system abstained, failed to return an output or encountered unsupported data. Add a coverage record beside the score audit rather than treating missing forecasts as invisible.
For another original illustration, suppose there were 200 predetermined decision opportunities. The system issued 100 forecasts, explicitly abstained 70 times and failed to produce a usable response 30 times. Forecast coverage is 100 ÷ 200 = 50%. The remaining 50% is not one homogeneous category: abstention and operational failure deserve different investigation.
Do not compare that model's selected 100 forecasts with another model evaluated on all 200 as though the opportunity sets were identical. Report the shared eligible set and each system's coverage. A model that avoids hard cases may have a useful role, but its performance claim must describe the cases it actually covers. This is an evaluation design issue, not evidence that abstention is always better.
A score audit you can actually request
Ask the provider for the event definition, the score's construction, evaluation dates, sample count, missing-outcome handling, reliability bins, a simple baseline and the separation between model selection and evaluation. Ask for cost-aware performance only if the provider is making a strategy claim.
If any field is unavailable, write “unknown” in the record. Do not substitute a marketing explanation. The AI bot evaluation guide helps turn these unknowns into an acceptance decision without inventing an overall safety score.
For KLINEO, an AI Trade Reviewer score describes a read-only critique of an available workflow run. It is not a calibrated probability of profit or proof that an order filled. Use AI analysis to support inspection, and review actual order parameters separately. The decision-card walkthrough connects observations, invalidation and modeled loss assumptions.
A useful response to an unexplained 82% can be to withhold judgment. A smaller number with a documented meaning is more interpretable than a larger number with none. Even a well-calibrated forecast leaves individual losses possible and says nothing by itself about suitability. This AI-assisted educational article follows the editorial policy, and all tables and payoff examples above are original hypothetical constructions.