Methodology
How a vanta probability is made — and what it is not.
Every question runs through seven independent reasoning modules. Five estimators produce probabilities with weights; the skeptic tries to break the emerging consensus; synthesis combines what survives. The debate is stored verbatim — every forecast on this site shows its full internal argument.
- 1Research Agent
Weighs qualitative evidence for and against the event; tilts the market prior in log-odds space by the net signal.
- 2Quant Agent
Matches the question against a corpus of resolved events (similarity-weighted hit rate), then runs a Beta-posterior Monte Carlo for the credible interval.
- 3Market Agent
Reads the prediction-market consensus and decides how much to trust it from liquidity and volume.
- 4Sentiment Agent
Public mood and momentum. A weak standalone predictor, so it enters the pool at low weight.
- 5Historian Agent
Category base rates. Long horizons pull the market's current read toward the long-run frequency.
- 6Skeptic Agent
Never estimates. Attacks the interim consensus — strongest opposing evidence, structural risks — and haircuts confidence by disagreement and evidence thinness.
- 7Synthesis Agent
Pools every weighted estimate as a weighted average of log-odds, shrinks toward the base rate, and calibrates confidence from inter-agent agreement.
Log-odds pooling. Agent estimates combine as a weighted average of logits — a logarithmic opinion pool. Unlike averaging raw probabilities, this respects how evidence compounds near the extremes.
Base-rate shrinkage. The pooled estimate is pulled mildly toward the category's long-run frequency, correcting the overconfidence that plagues both crowds and models.
Monte Carlo uncertainty. Evidence strength sets the pseudo-sample size of a Beta posterior; sampling it yields the 90% credible interval and the probability that the true value sits above the market.
Confidence, not just probability. The 0–10 confidence score rises with inter-agent agreement and decisiveness, and falls with the skeptic's haircut. A 70% at confidence 8 and a 70% at confidence 3 are different claims.
Scoring. Resolved questions are scored on directional accuracy, Brier and log scores, and binned into the calibration curve on the accuracy page. Each agent's own call is frozen at resolution and scored on the agent leaderboard. Forecasts are only as good as their track record.
Leave-one-out replay over the 40 resolved reference events: the analog engine ventured a forecast on 38 (95% coverage, abstaining elsewhere) and scored a Brier of 0.380 against the always-predict-the-base-rate benchmark of 0.244 · directional accuracy 42%. Run it yourself: GET /api/quant/backtest.
The seeded corpus claims no forecasting edge by construction — simulated vanta estimates derive from the market signal, never the outcome. This deployment runs on a seeded demo corpus: market prices, evidence, and the resolved track record are deterministic fixtures that illustrate the system, not live data. The math is real; the edge is not yet earned. And when an LLM key is configured it writes the agents' prose — never the numbers.