No. 04 · The exchange
Does more AI judgment help?
We replayed the live rules over 2017 to 2025, with and without extra AI judgment. The pass rule was fixed before any number existed.
- Status
- Open
- Design half
- Apr 2017 to Dec 2020
- Test half
- Jan 2021 to Feb 2025
- Verdict
- Fail: beat the rules, trailed SPY
One pass rule, written before the first result
The judgment book passes only if it beats two things over the test half.
- The rules alone. The live rules with no extra judgment, after costs.
- SPY. The S&P 500 with dividends, over the same window.
Anything else is a fail. Drawdown, evals and controls are shown but never change the verdict.
Fail: the judgment book beat the rules but trailed SPY
It returned +47.8% after costs. The rules alone made +42.1%. SPY made +70.5%.
- Judgment helped a little. It added 5.7 points over the rules alone, and traded less. Turnover 24% a year vs. 39%. In the market 98% of the time vs. 85%.
- The swap check approved nothing. Its book matches the rules alone exactly.
- The untuned switch rule hurt. It made +34.0%, 8.1 points behind the rules alone.
- The AI books beat the no-AI books. The controls made +28.3% and +26.9%. The design half showed the opposite.
- Memory may explain that edge. All profit came from files where the AI named the company. Treat the edge over no-AI books as an upper bound.
- Trailing stops cut losses and returns. The 10% stop held drawdown to -7.3%, but made +24.0%.
- Berkshire beat every book. BRK.B returned +124.9%.
| Book | What it does | Total return | Per year | Max drawdown |
|---|---|---|---|---|
| BRK.B | Berkshire Hathaway, buy and hold. | +124.9% | Not reported | Not reported |
| SPY | S&P 500, total return with dividends. | +70.5% | Not reported | Not reported |
| Judgment book | Rules plus switch, swap check and near-miss review. | +47.8% | 9.9% | -18.1% |
| Rules only (v4 baseline) | Today's live rules. | +42.1% | 8.8% | -18.8% |
| Swap check | Rules plus switches the AI approves. | +42.1% | 8.8% | -18.8% |
| Switch rule | Rules plus every switch, no AI check. | +34.0% | 7.3% | -16.6% |
| Every gate-passer, equal weight | No AI. About 140 names, no account limits. | +28.3% | 6.2% | -24.5% |
| 15 cheapest gate-passers | No AI. Top 15 by earnings yield. | +26.9% | 5.9% | -18.1% |
| Rules plus 10% trailing stop | Sells after a 10% fall from the high. | +24.0% | 5.3% | -7.3% |
| Rules plus 5% trailing stop | Sells after a 5% fall from the high. | +11.0% | 2.6% | -6.2% |
| Check | What it asks | Result |
|---|---|---|
| Verdict | Did the judgment book beat the baseline and SPY? | Fail. It beat the rules, not SPY. |
| Leak probe | Share of profit from files where the AI named the company. | 100% in both books |
| Rule checker | Did each book follow its own rules? | Judgment book failed 1 of 6. Rules only failed 3 of 6. |
| Grounded reasoning | Does each answer rest on figures in its file? | 29 of 35 passed. Judge not yet calibrated. |
| Ranking skill | Does a higher expected return predict a higher real return? | 0.16 at 1 year, 0.21 at 3 years |
| Swap check | Did the AI approve any switch? | None. That book equals the rules alone. |
| Near-miss scorecard | Did rescued companies beat the gate-passers? | Not reported |
| Loop pick | How did the tuned rule set do on these years? | Not reported |
| Adjusted for 33 tries | How much of the loop pick's lift survives the search? | Not reported |
Three judgment calls on top of the live rules
- 1. Switch rule
When cash runs out, sell the weakest holding for a clearly better candidate.
- 2. Swap check
Before each switch, the AI reads both files and says yes or no.
- 3. Near-miss review
A company failing one gate gets a second look. Is the failure a one-off?
The quality gates did the work, not the AI's judgment
The rules alone returned +97.1%. Buying every gate-passer with no AI returned +97.7%.
| Book | What it does | Total return | Max drawdown |
|---|---|---|---|
| Rules only (v4 baseline) | Today's live rules: five gates, 8% bar, conviction sizing. | +97.1% | -32.2% |
| Every gate-passer, equal weight | No AI. Buys every company that passes the gates. | +97.7% | Not reported |
| Switch rule, untuned | Sells the weakest holding for a better candidate. | +88.2% | -40.7% |
| 15 cheapest gate-passers | No AI. Top 15 by earnings yield, equal weight. | +70.6% | -32.4% |
| SPY | S&P 500, total return with dividends. | +69.9% | Not reported |
| BRK.B | Berkshire Hathaway, buy and hold. | +39.0% | Not reported |
| Baseline plus 10% trailing stop | Sells after a 10% fall from the high. | +31.9% | -25.9% |
Rules alone: 19.9% a year. Max drawdown -32.2%. Beta 0.84. Alpha about 6.5% a year.
The AI mostly said: change nothing
- Swap check. It approved 0 of 37 proposed switches. That book matches the rules alone.
- Near-miss review. It rescued 1 of 1,704 near-misses. That book is close to the rules alone.
- Switch rule, untuned. It made less (+88.2%) and fell deeper (-40.7%).
In 2017 the rules alone matched SPY
The cheapest gate-passers led. Tight trailing stops cost the most.
| Book | Note | Total return |
|---|---|---|
| 15 cheapest gate-passers | No AI. | +21.5% |
| BRK.B | Buy and hold. | +18.8% |
| Rules only (v4 baseline) | Live rules. | +15.2% |
| SPY | Total return with dividends. | +15.1% |
| Baseline plus 10% trailing stop | Sells after a 10% fall. | +8.0% |
| Baseline plus 5% trailing stop | Sells after a 5% fall. | +0.2% |
The tuned rules added little in the 2020 check
A tuning loop tried 33 rule sets on 2017 to 2019. It froze one on Oct 20.
- Conviction 3 size
- 7%
- Buy bar
- 7.5%
- Conviction 4 to 5 size
- 12%
- Information ratio, 2017 to 2019
- 0.63
- 2020 check
- +0.7%
Knobs that change the AI's judgment had no effect.
Discount for 33 tries. The best of 33 looks better than it is. Treat the 2017 to 2019 lift as an upper bound.
Why you can trust this
- Blind files. The AI saw figures from filings, with names, tickers and dates removed.
- Only what was known. Each judgment used filings and news already public that day.
- No web search. A search from 2019 would return pages from today.
- Same answers everywhere. One set of AI answers feeds every book.
- Every try reported. All 33 rule sets count, not only the winner.
Known limits
- Memory. The AI was trained on text from these years. All profit came from files where it named the company.
- Graders. The judge that grades reasoning is not calibrated yet. Read its scores as a first look.
- Gate settings. Two gates were set in 2026 with 2023 to 2024 in view.
- Effect. Gate settings flatter every rules book against SPY. Memory flatters AI books against no-AI books.
Method: books pay trading costs. Idle cash earns nothing. SPY is total return with dividends. BRK.B is buy and hold. Tuning used 2017 to 2019 only. 2020 was a check. The test half stays locked. Not investment advice. Back to the live account.