Key result: OOT AUC (WF 2007-23): 0.71 vs 0.62 for the 11-input logistic.
How it was tested: training data 2000-01→2023-12, test data 2007-01→2026-09; out-of-sample design (how it was checked on data it never saw while being built): walk-forward yearly + hold-out.
Horizon: 21 sessions. Coverage: US market state (one number per day), 2000-01 → today.
Stage: Context-only (validated FIT WITH CONDITIONS; scored nightly since 2026-10-17, not yet shown on a tab). Version v1; the hub record is as of 6 Nov 2026.
The verdict
FIT WITH CONDITIONS as a context probability (champion B1). M1 and the HMM/mixture models are NOT FIT as predictors; the HMM-3 state is descriptive only.
Development summary: No regime model (HMM 2/3, mixture 2/3) beats the logistic baseline out of sample; a 2-input persistence logistic beats all (skill 0.12, AUC 0.71, slope 1.04) and is published.
What the review allows
B1's p_riskoff_21d as a calibrated context read of the 21-session risk-off odds as defined.
The walk-forward and hold-out comparison as the honest record of the regime-model race.
What the review rules out
Read it as a drawdown or crash forecast (SPY-leg AUC 0.59).
Size positions or gate strategies from it, or from the HMM state.
Quote M1–M5 probabilities as forecasts; apply it outside US SPY/VIX.
What it predicts
Item
Detail
Question it answers
Odds that SPY is 5% lower, or VIX above its 1-year 80th percentile, 21 sessions from now.
VIX percentile in trailing year, SPY 21-session return, HMM-3 state (descriptive)
Output
Probability 0–1 (history 0.07–0.59), level-valid; state label descriptive
Horizon
21 sessions
Coverage
US market state (one number per day), 2000-01 → today
Markets
US
Model type
Model · ML/Stat
Stage
Context-only (validated FIT WITH CONDITIONS; scored nightly since 2026-10-17, not yet shown on a tab)
Version
v1, first recorded 16 Oct 2026
Validation evidence
Training data: 2000-01→2023-12. Test data: 2007-01→2026-09. Out-of-sample design: walk-forward yearly + hold-out.
Headline results from the model's development record, as the hub publishes them.
Measure
Result
Compared with
Note
Persistence logistic (B1)
Brier 0.157, AUC 0.71
base rate 0.179
skill CI 0.06–0.18
11-input logistic (M1)
Brier 0.173, AUC 0.62
B1
B1 better, BH q 0.053
Best regime model (HMM-3)
Brier 0.169, AUC 0.55
M1
not better, BH q 0.79
Hold-out 2024-26 (B1)
skill 0.08, AUC 0.69
base rate
8 episodes, CI -0.04–0.17
The hub's standard metrics for this model class (only those the record measured).
Metric
Sample / window
Compared with
Value
Range (CI)
Brier score (headline)
WF 2007–23 (17 yearly folds)
0.1787
0.157
[0.1303, 0.1847]
AUC
WF 2007–23 (17 yearly folds)
11-input logistic 0.624; HMM-3 0.549
0.7118
[0.6537, 0.7595]
Brier skill
WF 2007–23 (17 yearly folds)
0
0.1209
[0.0556, 0.1813]
What the validation review found
Is the design sound? The label is relative (VIX against its trailing year) while HMM/mixture states are absolute fear levels; persistence is the right benchmark for a persistent label and it wins. Readout from filtered probabilities is causal.
Data quality. TradingView bars from the local store; same-session SPY/VIX, FRED +1 business day, FX next session; 100% coverage after 2000-01-03; S5TH (monthly pre-2007) replaced before fitting.
Backtest. Expanding yearly walk-forward with label purge, 17 development folds and a one-shot hold-out; block bootstrap (63), Newey-West (42), month clusters, largest p, BH.
Calibration. B1 ECE 0.022, slope 1.04, intercept -0.05: level-valid in 0.07–0.59 and rank-valid (AUC 0.71); M1–M5 neither (slopes 0.48–0.71).
Limitations
Mostly predicts the VIX leg; not a drawdown or crash forecast.
Probabilities live in 0.07–0.59; it cannot say 'very likely'.
Relative VIX leg can fire from calm markets after a quiet year.
Hold-out has 8 episodes; its CI includes zero.
HMM state is descriptive, not forward odds.
US only (SPY, VIX).
Questions
Is the Regime Model — P(risk-off, 21 sessions) validated?
Its validation review dated 16 Oct 2026 rated it FIT WITH CONDITIONS as a context probability (champion B1). M1 and the HMM/mixture models are NOT FIT as predictors; the HMM-3 state is descriptive only.
What does the Regime Model — P(risk-off, 21 sessions) predict?
It answers: Odds that SPY is 5% lower, or VIX above its 1-year 80th percentile, 21 sessions from now. Output: Probability 0–1 (history 0.07–0.59), level-valid; state label descriptive. Horizon: 21 sessions. Coverage: US market state (one number per day), 2000-01 → today.
How was the Regime Model — P(risk-off, 21 sessions) tested?
Training data 2000-01→2023-12, test data 2007-01→2026-09, out-of-sample design: walk-forward yearly + hold-out. OOT Brier skill (WF 2007-23): 0.12 (CI 0.06–0.18); OOT AUC (WF 2007-23): 0.71 vs 0.62 for the 11-input logistic; HMM / mixture vs logistic: none beats it (BH q ≥ 0.79).
Out-of-sample: results measured on data the model never saw while it was built or tuned.
Walk-forward: the model is refit on data up to a date and tested on the period after it, then the window rolls forward, so every test is out of sample.
Hold-out: a slice of data set aside and used once, for the final test.
AUC: area under the ROC curve: how well a score ranks cases that happened above cases that did not; 0.5 is a coin flip, 1.0 is perfect.
Brier score: the mean squared error of probability forecasts; lower is better.
Base rate: how often the outcome happens with no model at all.
Confidence interval: the range that likely contains the true value (95% unless stated); [a, b] on this page is that range.
SPY: the SPDR S&P 500 ETF, the US market benchmark used here.
BH correction: the Benjamini–Hochberg adjustment for running many tests at once, so chance hits are not counted as findings.
Calibration: whether predicted probabilities match how often things actually happen.