Where this started
In late 2025, Nof1's Alpha Arena gave six models $10,000 each to trade crypto perpetuals on Hyperliquid for 17 days with no human in the loop. Qwen 3 Max finished first, DeepSeek second, and they were the only two in profit. GPT-5 lost about 63%. The organisers themselves cautioned that the results might be luck.
Two things about that setup bothered me. The models traded continuously, so the result mixed judgment with how often each one fired off trades. And there was no way to see whether a model's reasoning was any good, only its P&L. I wanted a version with fewer decisions and a visible paper trail: one decision per model, made through a fixed research process, with every intermediate output logged.
I posted the draft on X in July as a game. This is the write-up of what the logs actually show.
The setup
Ten models were entered: GPT, Claude, Gemini, Grok and Qwen as the closed-weights group, and DeepSeek, GLM, MiniMax and Kimi as the open-weights group. Meta's Spark couldn't be reached from outside the US and never placed a trade, so nine portfolios ran. Each got $10,000 of paper money and identical prompts. Positions could be long or short in US stocks, ETFs or crypto, and anything left undeployed stayed in cash.
Every model went through the same six API calls:
Research calls used OpenRouter's web-search tool, with today's date injected and an instruction to verify current prices. That instruction was added after a dry run where MiniMax built a thesis on months-old prices. All calls ran at temperature 0.7. Entries were locked at the 15 July open, before anyone could see the day's trading, and positions were marked at one week and again at the 28 July close. Fills are at the open with no fees, slippage or borrow costs.
The harness logs every response, so each number below can be traced to a raw JSON file: 62 candidates, 186 analyst reports (62 × 3), 20 positions plus two cash holds, and 2,751,082 tokens.
What they bought
| Model | Book | Cash |
|---|---|---|
| GPTclosed | AMD $6,000 · short TSLA $4,000 | — |
| Claudeclosed | VLO $3,500 · short TSLA $2,000 · IBM $1,500 | $3,000 |
| Grokclosed | TSM $3,000 · XOM $2,000 · short TSLA $5,000 | — |
| Geminiclosed | PLTR $10,000 | — |
| Qwenclosed | GS $7,500 | $2,500 |
| DeepSeekopen | GS $3,000 · ETH $2,000 · NVDA $5,000 | — |
| GLM 5.2open | BTC $2,000 · NVDA $3,000 · XLE $5,000 | — |
| MiniMaxopen | MU $2,000 · OXY $4,000 · short APP $4,000 | — |
| Kimiopen | LCID $10,000 | — |
Two patterns stand out before any prices move. First, the books are not independent. Three models shorted Tesla into its 22 July earnings, and four held an energy name (VLO, XOM, XLE, OXY) on the same Middle East oil thesis. Seven of nine books contain at least one of those two trades. Second, sizing varied far more than stock selection did. Two models put everything into one stock, and two held a quarter or more in cash, all after reading similar analyst reports.
Results
| Portfolio | After 1 week | At 28 Jul close |
|---|---|---|
| Kimi | +46.1% | +67.7% |
| Grok | +6.3% | +10.3% |
| MiniMax | +4.4% | −2.1% |
| GPT | +3.7% | −1.9% |
| Claude | +2.4% | +5.1% |
| GLM 5.2 | +2.0% | −2.1% |
| DeepSeek | −1.7% | −6.7% |
| Qwen | −3.2% | −7.5% |
| Gemini | −6.9% | −8.1% |
| Field (equal weight) | +5.9% | +6.1% |
| Field without Kimi | +0.9% | −1.6% |
| S&P 500 (SPY) | −1.0% | −1.8% |
| Nasdaq 100 (QQQ) | −2.2% | −6.7% |
| Gold (GLD) | +1.7% | −1.4% |
The headline holds: as a group, the models beat the S&P 500 and the Nasdaq over both windows. It does not hold up once you take out one portfolio. Without Kimi, the field was 0.9 points ahead of cash after a week and roughly level with the S&P after two.
The middle of the table also depends on when you look. Three of the six portfolios that were green after one week were red a week later. Over the two-week window, GPT's return would have been anywhere from −8.0% to +7.7% depending on which day it was closed. Ranking models on a single short window mostly ranks the window.
Where the return came from
Excluding Kimi, the other eight books made $698 in the first week. Splitting that by theme:
| Source (week 1, ex-Kimi) | P&L |
|---|---|
| Three TSLA shorts (GPT, Claude, Grok) | +$1,125 |
| Four energy longs (VLO, XOM, XLE, OXY) | +$731 |
| Every other position | −$1,157 |
| Total | +$698 |
Everything outside those two crowded trades lost money. Both themes were the most heavily covered stories that week: a Tesla earnings date and an oil spike tied to Iran. My reading is that the models are good at finding the consensus catalyst and acting on it, and that this happened to pay in a falling market. One week of one market regime can't tell skill apart from a shared bet that worked.
Hit rate tells the same story. Twelve of 20 positions closed the week in the right direction, but two of those were NVDA at +0.03%. Count only positions that moved meaningfully and it is 10 of 20.
Did the analysts know anything?
This is the question the harness was built to answer. Each candidate has a bear, neutral and bull score from sub-agents that never saw each other's work. For 47 of the 62 candidates I have a return, because some model held that ticker or it is a benchmark. That lets me score the analysts on ideas that were rejected as well as ideas that were bought.
| Signal | ρ, all 47 | ρ, without LCID (45) | p (clustered), without LCID |
|---|---|---|---|
| Bear score | +0.30 | +0.44 | 0.008 |
| Neutral score | +0.11 | +0.11 | 0.61 |
| Bull score | −0.19 | −0.25 | 0.15 |
| Gut rank | +0.04 | −0.05 | — |
Spearman correlation between each score and the position's direction-adjusted one-week return. The p-values come from a permutation test that shuffles returns across ticker-direction pairs rather than rows, since ETH alone appears eight times.
The only lens that tracked outcomes was the skeptic. Where the bear analyst was less worried, the position tended to do better. The bull's enthusiasm pointed slightly the wrong way, and the model's own first instinct carried no information at all.
I checked the obvious confound: the bear was kinder to shorts (average 5.1 versus 3.4 for longs), and shorts did better in a falling market. Restricted to the 33 long positions, the bear correlation is still +0.37, so it isn't only direction. But this is one week, 22 independent ticker-direction pairs, and four scores. I tested four signals, so a p of 0.008 is suggestive, not established.
The scores were also skewed. The average bull score was 6.6, the average bear score 3.7. That is by design, since each lens was told what to argue. It means the raw numbers can't be read as probabilities, only as rankings within each lens.
Did the models listen? Mostly. In eight of nine books, the top-scoring candidate (or one tied for top) was bought. Selected positions averaged 17.3 out of 30 combined, rejected ones 14.3. The pipeline did shape the books. It just shaped them around a combined score that was mostly noise, when the only informative part was the bear column.
The model that overruled its analysts
Kimi's analysts ranked a QQQ short first at 20 out of 30. LCID came fifth at 12, and its bear analyst gave it 1 out of 10, the lowest score in the whole draft. Kimi put the entire $10,000 into LCID anyway, citing a denied bankruptcy rumour and 47% short interest. It was the only model to buy its analysts' bottom half, and the only one to go all in against them.
It returned +46% in a week and +68% in two, more than every other portfolio combined. GLM had shorting LCID on its own candidate list and passed on it.
The rivals were unanimous before the result. Claude's roast:
"$10,000 all-in on LCID — the only trader here whose exit strategy is bankruptcy court either way."
I don't read this as skill. A short-squeeze thesis on a heavily shorted stock is a high-variance bet, and one sample of a high-variance bet says almost nothing about whether the next one pays. What the episode does show is that the sizing stage is where an agent's behaviour diverges most. Every model saw a balanced research packet. One treated it as a constraint and one ignored it, and that choice moved the field result more than all the research did.
Cost
| Model | API cost (6 calls) | Return, week 1 |
|---|---|---|
| Claude | $5.28 | +2.4% |
| Grok | $2.46 | +6.3% |
| GPT | $0.86 | +3.7% |
| Gemini | $0.69 | −6.9% |
| Qwen | $0.41 | −3.2% |
| DeepSeek | $0.39 | −1.7% |
| Kimi | $0.36 | +46.1% |
| GLM 5.2 | $0.29 | +2.0% |
| MiniMax | $0.21 | +4.4% |
| Total | $10.95 |
Claude cost 25 times as much as MiniMax for the same six calls, mostly from longer research outputs. There is no visible relationship between spend and return. The open-weights group as a whole cost $1.25. Their group average was ahead (+12.7% versus +0.5% for closed), but without Kimi it was +1.6%, close enough to the closed group that I wouldn't call a winner.
What this doesn't show
- It's one draft. Each model ran once at temperature 0.7. The persuasion experiments used 50 runs per condition; this used one. A rerun of the same model on the same morning could easily produce a different book.
- The positions aren't independent. Seven books share the TSLA short or the energy trade. The effective sample is closer to a handful of themes than 20 positions.
- The windows are short and overlapping. One week and two weeks, in one market regime (a falling Nasdaq and an oil spike). The ranking changed between them.
- Paper fills are generous. No fees, slippage or borrow costs. Shorting a stock with 47% short interest costs real money, and nobody actually shorted it here, but the TSLA and APP shorts would have.
- 15 candidates have no return. Tickers that no model bought and that aren't benchmarks (MSTR, SOUN, ASML, GME and others) were never priced. The analyst correlations cover 47 of 62.
- Search changes the task. Models with better search use had fresher prices. Some of what looks like judgment may be retrieval quality.
One property does work in the experiment's favour: it can't be contaminated. Every decision was locked before the outcome existed, so no model could have seen the answer in training.
What's next
The useful finding isn't who won. The pipeline produced one weakly informative signal, the bear analyst, and then averaged it away with two that weren't informative. The sizing stage turned identical research into books ranging from 30% cash to all in on a single stock. Both are fixable, and both can be measured.
The next version keeps the same harness and changes the design:
- Run each model 20 times per week rather than once, and score the distribution of books it produces, not a single draw.
- Price every candidate, so rejected ideas are scored as carefully as bought ones.
- Report returns net of market beta and of the shared themes, so a crowded trade working isn't counted as nine separate successes.
- Test whether weighting the bear score more heavily, or letting the bear veto, holds up out of sample.
This also links to the persuasion work. Both are about agents given authority over money. A trading agent that overrules its own analysts on a squeeze story is the same failure surface as a buyer that accepts a seller's anchor: a decision step where the reasoning before it is sound, and the action doesn't follow from it.