Verdict: Crash-Odds Model (M-020) is rated “NOT RELIABLE” in its validation review dated 4 Sep 2026.
Key result: Purged out-of-fold AUC (the honest number): 0.5517 vs 0.50 = coin flip.
Key result: Gate AUC vs volatility-only baseline: 0.7218 vs 0.7250 — baseline won.
How it was tested: training data not stated, test data 2024-09→2025-08; out-of-sample design (how it was checked on data it never saw while being built): holdout.
Horizon: 252 trading days (~12 months) forward, measured on daily closes only (not intraday lows). Coverage: Point-in-time monthly panel, matured 2003-2025 formations (667,002 rows, 116,525 confirmed crashes) for the final fit; scored cross-section 2026-08-31 = 8,279 names across 20 markets.
Stage: Retired. Version v1; the hub record is as of 6 Nov 2026.
The verdict
NOT RELIABLE: purged out-of-fold AUC 0.552 (Brier within 0.0016 of the constant base rate) shows no discriminative power. The gate step's apparent pass (AUC 0.722) had no 12-month embargo and failed once applied. Retired at v1.
Development summary: Once look-ahead is purged the model has no usable power: gate-step AUC 0.722 lost to a vol-only baseline (0.725) and on precision@20; purged year-block OOF AUC 0.552, Brier 0.1426, within 0.0016 of a constant base rate. Retired at v1.
What the review allows
The purged OOF numbers (AUC 0.5517, Brier 0.14259) as the honest record that price/volume-only features do not predict >=40% drawdowns over 12m on this panel — useful negative evidence for scoping a v2.
as a reproducibility artifact / regression fixture for a future v2 harness — not as a live signal.
What the review rules out
Treat p_crash_12m as predictive of anything — read it nowhere.
Cite the gate step's AUC 0.7218 / precision@20 0.2923 as validated performance — superseded by the purged OOF result and very likely inflated by a non-purged split.
Assume the volatility-only baseline is validated either — it was only compared on the same non-purged gate window, not purged itself.
Extend p_crash_12m's survivorship-biased base rate to any absolute crash-probability claim — it is a measured lower bound.
What it predicts
Item
Detail
Question it answers
What is the probability that a name suffers a -40% or worse peak-to-trough drawdown within the next 12 months, on price/volume/regime information alone?
Method
RandomForest + purged isotonic
How it works
RandomForest (depth 6, leaf 200, 400 trees, max_features 0.5) on a point-in-time monthly panel; chronological gate vs a vol-only baseline, then purged year-block out-of-fold isotonic calibration.
p_crash_12m — isotonic-calibrated probability of a >=40% drawdown in the next 252 trading days, per symbol per formation date; plus p_crash_raw and rank_pct.
Horizon
252 trading days (~12 months) forward, measured on daily closes only (not intraday lows)
Coverage
Point-in-time monthly panel, matured 2003-2025 formations (667,002 rows, 116,525 confirmed crashes) for the final fit; scored cross-section 2026-08-31 = 8,279 names across 20 markets
Markets
US, India, Japan, Korea, Hong Kong, Taiwan, Canada, Singapore, UK, Germany, France, Switzerland, Netherlands, Italy, Spain, Sweden, Denmark, Norway, Poland, Finland
Model type
Model · ML/Stat
Stage
Retired
Version
v1, first recorded 4 Sep 2026
Validation evidence
Training data: not stated. Test data: 2024-09→2025-08. Out-of-sample design: holdout.
Headline results from the model's development record, as the hub publishes them.
Measure
Result
Compared with
Note
Gate AUC vs volatility-only baseline
0.7218
0.7250 (baseline)
baseline won; the gate split lacks the 12-month embargo the purged OOF stage below applies
year-block OOF, 12-month embargo each side, folds 2016-2025 — this is the honest number
Purged out-of-fold Brier score
0.14259
~0.1442 = naive Brier at the 17.47% full-sample base rate (116,525/667,002)
within 0.0016 of predicting the unconditional rate for every name; base rate not logged separately for the OOF-specific fold population
The hub's standard metrics for this model class (only those the record measured).
Metric
Sample / window
Compared with
Value
Range (CI)
AUC (headline)
purged OOF 2016–25
0.5
0.5517
n/a
Precision at k
gate 2024-09→25-08 (unpurged)
vol-only 29.49%, base 13.46%
29.23%
n/a
Brier score
purged OOF 2016–25
0.1442
0.1426
n/a
What the validation review found
Is the design sound? The harness (point-in-time monthly panel, rolling gate cutoff, purged year-block OOF isotonic calibration, RandomForest) is a direct, appropriate mirror of growth-model's validated Moonshot Odds Model applied to the opposite tail — same problem class, same overfitting risk — and the target definition (path-based max drawdown from daily closes, censoring-aware label) is the right read of '>=40% drawdown within 12 months' for a risk lens. The gap between the two evaluation stages is itself conceptually informative: it demonstrates why the embargo step exists rather than casting doubt on the harness design.
Data quality. Features are price/volume/regime only, correctly excluding fundamentals (not point-in-time before 2026-08-22) and short interest (single undated snapshot) rather than testing them and hoping. Survivorship was measured directly on, not assumed: 0 of 10,988 symbols end more than 180 days early, 23 (0.21%) end more than 30 days early — the panel is today's listed universe with history attached, so measured crash rates are a lower bound, in the same direction and for the same reason Moonshot's absolute-return upper bound is documented.
Backtest. Two-stage: a chronological gate (train strictly before a 12-cohort out-of-time window, 108 matured cohorts, 498,407 train / 85,732 test rows, 32 features) against a trailing-rvol_12m baseline on identical rows, then a purged year-block (12-month embargo each side) out-of-fold isotonic calibration on the all-cohorts final fit (667,002 rows, 116,525 positives), folds 2016-2025. The gate reported AUC 0.7218 (baseline 0.7250, baseline won) and precision@20 0.2923 (baseline 0.2949, base rate 0.1346); the purged OOF — the stage that actually embargoes the forward-looking label window — returned AUC 0.5517 and Brier 0.14259. The two numbers disagree sharply, and the purged one is the one to trust.
Calibration. Purged OOF Brier of 0.14259 is close to (0.0016 above) the Brier a constant prediction at the full-sample base rate (17.47% = 116,525/667,002) would score (~0.1442) — i.e. isotonic calibration on this fitted score adds essentially nothing over ignoring every feature and predicting the unconditional rate for every name. Consistent with that, p_crash_12m collapses to 34 distinct values with 60.2% of the scored cross-section (4,986/8,279 names) pinned at the single floor value 0.1545 — the calibrated output is not a spectrum for most of the universe.
Limitations
Purged out-of-fold AUC is 0.5517 — barely above 0.50 and the only number in this card computed on a properly embargoed split. Nothing downstream should treat p_crash_12m as predictive.
The gate step's apparent pass (AUC 0.7218, prec@20 0.2923 — both of which actually lost to the baseline) used a chronological-only split without the 12-month embargo the final OOF calibration applies; given forward-looking 252-day labels, that gap plausibly explains why gate looked competitive and purged OOF did not. Any future gate must apply the same embargo before its verdict is trusted.
SURVIVORSHIP: measured P(crash) is a lower bound on the true historical crash rate — delisted-for-cause names are almost entirely absent from the panel (not present-then-truncated), and they are disproportionately the true crashes.
No fundamentals (not point-in-time before 2026-08-22) and no short interest (single undated snapshot) — both excluded on leakage/PIT grounds, not tested and rejected; a v2 with fundamentals would need >=12 months of valuation history accrual, earliest 2027-08, the same constraint M-002 documents.
p_crash_12m is degenerate: isotonic calibration collapses the cross-section to 34 distinct values, and 60.2% of scored names (4,986/8,279) share the single floor value 0.1545 — consistent with the near-chance AUC; most of the universe gets one bucket, not a spectrum.
No hyperparameter search and no VIF/correlation feature pruning were run for v1 (inherited from Moonshot's config, a documented scope decision) — flagged as v1.1 follow-ups in the module docstring itself; given the OOF result, re-tuning is unlikely to be the fix.
Questions
Is the Crash-Odds Model validated?
Its validation review dated 4 Sep 2026 rated it NOT RELIABLE: purged out-of-fold AUC 0.552 (Brier within 0.0016 of the constant base rate) shows no discriminative power. The gate step's apparent pass (AUC 0.722) had no 12-month embargo and failed once applied. Retired at v1.
What does the Crash-Odds Model predict?
It answers: What is the probability that a name suffers a -40% or worse peak-to-trough drawdown within the next 12 months, on price/volume/regime information alone? Output: p_crash_12m — isotonic-calibrated probability of a >=40% drawdown in the next 252 trading days, per symbol per formation date; plus p_crash_raw and rank_pct. Horizon: 252 trading days (~12 months) forward, measured on daily closes only (not intraday lows). Coverage: Point-in-time monthly panel, matured 2003-2025 formations (667,002 rows, 116,525 confirmed crashes) for the final fit; scored cross-section 2026-08-31 = 8,279 names across 20 markets.
How was the Crash-Odds Model tested?
Training data not stated, test data 2024-09→2025-08, out-of-sample design: holdout. Purged out-of-fold AUC (the honest number): 0.5517 vs 0.50 = coin flip; Gate AUC vs volatility-only baseline: 0.7218 vs 0.7250 — baseline won; Purged out-of-fold Brier score: 0.14259 (~= naive base-rate-only Brier of ~0.1442).