An LLM can produce a plausible forecast and a detailed explanation in the same response. Evaluating that forecast requires a different task: saving the probability before the outcome is known, defining how the outcome will be judged, and comparing performance across an appropriate set of questions. A persuasive paragraph does not perform those checks for you.
For prediction market research, begin with an evaluation notebook alongside the research notebook. The first records predictions and scores; the second records evidence and reasoning. Our AI forecasting coverage explores the broader workflow. This guide explains how to tell whether a forecasting system's probabilities are informative and how much confidence the available evidence supports.
Freeze the question and the information cutoff
Every forecast needs an exact question, a resolution rule, a deadline, and a submission timestamp. Save the model's probability with the version of the prompt, the retrieved evidence, and the relevant system settings. If the system can browse, preserve enough of its research record to establish what information it used.
Use questions whose outcomes are unknown when predictions are submitted. The researchers behind ForecastBench designed a dynamic evaluation around future events that had not resolved at submission time. That design addresses a central evaluation problem: a system should be assessed on forecasting rather than access to an answer that already exists.
Historical exercises can still test parts of a workflow, but require careful qualification. Restricting retrieved documents to an earlier date does not prove the model lacks later information from training. Describe such results as retrospective unless the conditions support a stronger claim. Save the original question even when later wording becomes clearer.
Understand what calibration means
A calibrated probability has a frequency interpretation across repeated forecasts. Among a sufficiently large set of comparable events assigned approximately 70 percent probability, about 70 percent should occur. Calibration concerns a pattern across cases. One event assigned 70 percent can fail without showing that the forecaster is systematically miscalibrated.
Group forecasts into probability ranges and compare their average predicted probability with the observed outcome frequency. Also show how many forecasts fall into each group. A range containing three questions deserves a different interpretation from a range containing three hundred. Very narrow ranges may create a tidy-looking chart with too little evidence in each group.
Calibration alone does not establish useful discrimination. A system that always gives the overall base rate may be calibrated across a dataset while failing to distinguish easier from harder cases. Evaluate whether the model gives different probabilities when the evidence warrants it, and whether those differences improve its overall score.
Calculate a score with a clear definition
For a binary event, a simple Brier score is the squared difference between the forecast probability and the outcome: (p − y)². Here p is a number between zero and one, while y equals one if the event occurs and zero otherwise. Average that quantity across the evaluated forecasts. Under this binary convention, lower scores are better.
Consider these entirely hypothetical predictions. They illustrate arithmetic and do not describe a real model, market, or research result.
| Question | Forecast p | Outcome y | Squared error |
|---|---|---|---|
| A | 0.80 | 1 | 0.04 |
| B | 0.70 | 0 | 0.49 |
| C | 0.60 | 1 | 0.16 |
| D | 0.10 | 0 | 0.01 |
The mean score is 0.175: add 0.04, 0.49, 0.16, and 0.01, then divide by four. A constant 0.50 forecast would score 0.25 on each of these binary questions. This tiny example demonstrates the calculation; four selected questions cannot establish reliable forecasting skill.
Publish the scoring convention because implementations can scale or aggregate scores differently. Also distinguish an average over questions from an average over every saved update. A frequently updated question should receive extra weight only if your protocol deliberately assigns that weight.
Choose baselines that answer the comparison
Start with a simple baseline, such as a relevant historical base rate estimated without using the test outcomes. A constant 50 percent forecast is easy to interpret, but may be weak for a dataset dominated by events that rarely happen. State how the baseline was constructed and why it is appropriate.
If comparing against a market estimate, record the same contract, outcome, and observation time. Identify the price measure you use. A model forecast made before new information arrives cannot be fairly compared with a market quote collected afterward. Missing or ambiguous market observations should follow a predefined exclusion rule.
Human comparisons also need matched conditions. Describe the questions, information access, time budget, and deadline for each group. A single overall ranking can hide meaningful differences across topics and horizons. Report the comparison that the experiment actually supports, and preserve enough detail for readers to understand its limits.
Prevent hindsight from improving the record
Separate development questions from the final evaluation set. Use development results to adjust prompts, retrieval methods, and aggregation rules. Once the evaluation starts, keep those decisions fixed for that run. Repeatedly changing the system after viewing test outcomes turns the test set into another development set.
Predefine how unresolved, canceled, or ambiguously resolved questions will be handled. Preserve exclusions and their reasons in the published record. Removing difficult questions after seeing a disappointing score can create a misleading result even when every remaining calculation is correct.
Log corrections transparently. If a parsing error changed 0.08 to 0.80, retain the original entry, the correction, its justification, and its timing. Distinguish an administrative correction from a revised forecast. The goal is to let a reviewer reconstruct both the submitted record and the reported score.
Inspect uncertainty and related questions
A score difference is an estimate from a particular sample. Show the number of resolved questions, the evaluation period, and uncertainty around the comparison when your analysis supports it. Small improvements on a small dataset may disappear when the question mix changes. Avoid presenting a narrow numerical lead as a universal capability ranking.
Related questions can reduce the independence of the evidence. Several contracts about one election, for example, may depend on the same underlying event. Counting every contract as a wholly separate forecasting challenge can exaggerate how much has been learned. Group related questions when assessing uncertainty or show sensitivity to that grouping.
Break down performance by categories established before inspecting results, such as time horizon or topic. Use these views to locate weaknesses, and label exploratory findings as exploratory. A system that performs well on short-horizon questions may need a different research process for events months away.
Keep model changes and forecast updates distinct
A forecast update responds to new evidence about a question. A model change modifies the system producing forecasts. Record both explicitly. If retrieval, prompts, or model versions change midway through an evaluation, preserve the earlier results and identify the new run. Combining versions without explanation makes the overall score difficult to interpret.
Decide in advance how updates will be scored. You might evaluate one forecast at a fixed horizon, one daily forecast, or a defined average over time. Each choice answers a different question. Use the same rule for the model and its baselines, and include the effects of missing submissions.
A calibration adjustment should be learned from separate development data and then tested on new questions. Rewriting probabilities after seeing the evaluation outcomes produces a description of the past. To demonstrate improvement, the adjusted system must commit its probabilities before the next set of outcomes is known.
Make the final claim proportional to the evidence
A useful evaluation report states what was tested, when forecasts were submitted, how many questions resolved, what was excluded, and how scores compared with the baselines. Include calibration results and material weaknesses. Link the analysis back to the saved evidence process described in our AI research brief guide.
Forecast accuracy and trading returns are different measurements. A market implementation introduces prices, fees, available liquidity, execution timing, and operational behavior. A favorable forecasting score alone establishes none of those outcomes. Our AI prediction bot risk controls article considers the additional operational questions.
The strongest result is one another researcher can examine and reproduce under the stated conditions. Clear timestamps, preserved forecasts, appropriate baselines, and visible uncertainty make that possible. They also make a disappointing result useful: it can reveal which part of the research process deserves the next improvement.
