Skip to content
VANTA

Methodology

How a vanta probability is made — and what it is not.

The pipeline

Every question runs through seven independent reasoning modules. Five estimators produce probabilities with weights; the skeptic tries to break the emerging consensus; synthesis combines what survives. The debate is stored verbatim — every forecast on this site shows its full internal argument.

  1. 1
    Research Agent

    Weighs qualitative evidence for and against the event; tilts the market prior in log-odds space by the net signal.

  2. 2
    Quant Agent

    Matches the question against a corpus of resolved events (similarity-weighted hit rate), then runs a Beta-posterior Monte Carlo for the credible interval.

  3. 3
    Market Agent

    Reads the prediction-market consensus and decides how much to trust it from liquidity and volume.

  4. 4
    Sentiment Agent

    Public mood and momentum. A weak standalone predictor, so it enters the pool at low weight.

  5. 5
    Historian Agent

    Category base rates. Long horizons pull the market's current read toward the long-run frequency.

  6. 6
    Skeptic Agent

    Never estimates. Attacks the interim consensus — strongest opposing evidence, structural risks — and haircuts confidence by disagreement and evidence thinness.

  7. 7
    Synthesis Agent

    Pools every weighted estimate as a weighted average of log-odds, shrinks toward the base rate, and calibrates confidence from inter-agent agreement.

The math

Log-odds pooling. Agent estimates combine as a weighted average of logits — a logarithmic opinion pool. Unlike averaging raw probabilities, this respects how evidence compounds near the extremes.

Base-rate shrinkage. The pooled estimate is pulled mildly toward the category's long-run frequency, correcting the overconfidence that plagues both crowds and models.

Monte Carlo uncertainty. Evidence strength sets the pseudo-sample size of a Beta posterior; sampling it yields the 90% credible interval and the probability that the true value sits above the market.

Confidence, not just probability. The 0–10 confidence score rises with inter-agent agreement and decisiveness, and falls with the skeptic's haircut. A 70% at confidence 8 and a 70% at confidence 3 are different claims.

Scoring. Resolved questions are scored on directional accuracy, Brier and log scores, and binned into the calibration curve on the accuracy page. Each agent's own call is frozen at resolution and scored on the agent leaderboard. Forecasts are only as good as their track record.

A checkable claim — analog-engine backtest

Leave-one-out replay over the 40 resolved reference events: the analog engine ventured a forecast on 38 (95% coverage, abstaining elsewhere) and scored a Brier of 0.380 against the always-predict-the-base-rate benchmark of 0.244 · directional accuracy 42%. Run it yourself: GET /api/quant/backtest.

Honest limits

The seeded corpus claims no forecasting edge by construction — simulated vanta estimates derive from the market signal, never the outcome. This deployment runs on a seeded demo corpus: market prices, evidence, and the resolved track record are deterministic fixtures that illustrate the system, not live data. The math is real; the edge is not yet earned. And when an LLM key is configured it writes the agents' prose — never the numbers.