Nate Silver · 2012

The Signal and the Noise

Why So Many Predictions Fail—but Some Don't

Rough AI truth score

75/100

The book's central case is sound: prediction improves when forecasters quantify uncertainty, test models out of sample, update with evidence, and respect domain-specific limits. Its Bayesian framing is useful, though some examples risk making successful practice look cleaner and more portable than it is, especially where regimes change or rare events dominate.

Based on three central claims · medium-high confidence

The three claims

01Supported

Many prediction failures come from overfitting noise, neglecting uncertainty, and failing to test forecasts against new data.

Forecast-evaluation research strongly supports calibration, out-of-sample validation, proper scoring, and comparisons with transparent baselines. Models can fit historical data while failing under distribution shift, and point estimates conceal decision-relevant uncertainty.

Assessing Calibration of Probabilistic Predictions NOAA Model Verification

02Mostly supported

Bayesian updating is a broadly useful discipline for combining prior knowledge with new evidence.

Bayes' rule is the coherent mathematical update for stated probability models, and explicit priors can expose assumptions. In practice, results depend on model specification, likelihood choice, data quality, and whether the environment is stable enough for yesterday's priors and mechanisms to remain useful.

Good Judgment Project: Identifying Superforecasters Independent Reality Check on Superforecasting

03Mixed

The book's lessons transfer reliably across elections, economics, sports, weather, pandemics, and rare-event risks.

Calibration and validation travel well, but forecastability does not. Weather and repeated games provide abundant feedback; geopolitical shocks, pandemics, and financial tails have sparse samples, changing mechanisms, strategic actors, and fat-tailed losses. Domain knowledge and horizon remain decisive.

Fat-Tailed Forecasting Review Forecast Evaluation Under Structural Change

Other claims worth checking

Prediction markets can aggregate dispersed information under favorable conditions.
Base rates should constrain compelling causal stories.

What this number means. It is an AI-generated first-pass judgment of three central factual or causal claims—not a rating, exhaustive fact-check, or human peer review. Claim credits are 100% for supported, 75% for mostly supported, 50% for mixed, and 25% for weak, then averaged and rounded. Lower confidence means the score should move more as better evidence arrives.

Method three-central-claims/0.1.0 · checked 2026-09-01 · 3/3 selected claims assessed · method and source audit