Ornith-1.5: Open Weights That Go Toe-to-Toe With Claude Opus 4.8
Ornith-1.5: Open Weights That Go Toe-to-Toe With Claude Opus 4.8
Scrolling my feed this morning I saw the "Aloha!" — the signature greeting the Ornith team opens every launch with. It's been quiet since June 25, when they dropped Ornith-1.0, a self-improving open-weight coder that made most of the open-source crowd do a double take.
This time it's Ornith-1.5, and the benchmark chart it brought faces a much scarier name than last time: Claude Opus 4.8.
So I did the thing I always do — I checked whether it was real, and pulled the model cards. Verdict: it's real, the weights are live under an MIT license, and the 397B flagship genuinely trades blows with Opus 4.8 — beating it in some spots, losing in others.
What is Ornith-1.5?
Ornith is the open-source LLM family from DeepReinforce (the five-person team behind the GrandCode and CUDA-L2 reinforcement-learning papers). What made 1.0 stand out wasn't just the numbers — it was the training recipe: instead of handing the model a fixed testing harness, they train it with reinforcement learning to write its own task scaffolds (memory, routing, the scaffolding an agent leans on) while it solves the task. It's "self-improvement" conditioning — the model learns the harness and the answer together, with anti-reward-hacking safeguards bolted on.
Ornith-1.5 is that recipe's second act. It comes in three sizes:
| Variant | Type | Ready for |
|---|---|---|
| Ornith-1.5-9B | Dense | Edge, laptops, Apple Silicon |
| Ornith-1.5-35B-A3B | MoE, 3B active | Gaming GPUs, DGX Spark class |
| Ornith-1.5-397B | MoE | Multi-GPU / enterprise |
All three are MIT-licensed (no revenue cap, no branding clause — genuinely open weights), post-trained on Qwen 3.5 backbones, with a 262K-token context window, multimodal input, and a reasoning/tool-call format that drops straight into vLLM, SGLang, and OpenAI-compatible clients. GGUF, FP8, NVFP4 and MLX quants shipped alongside.Region note: the 397B model card carries a region:us tag — another reminder that even "open" models can carry regional strings attached, so check before planning a deployment.
The headline: 397B vs Claude Opus 4.8
All figures below are DeepReinforce's official vendor numbers, averaged over five independent runs (full harness notes in the model cards). Here's how the flagship stacks up against Opus 4.8 and the other heavy hitters:
| Benchmark | Ornith-1.5 397B | Claude Opus 4.8 | DeepSeek V4 Flash | GLM-5.2 | vs Opus |
|---|---|---|---|---|---|
| Terminal-Bench 2.1 (Terminus-2) | 86.1 | 85.0 | 82.7 | 81.0 | WIN |
| Terminal-Bench 2.1 (Claude Code) | 85.2 | 78.9 | 81.8 | 82.7 | WIN |
| SWE-bench Verified | 86.0 | 85.8 | 81.6 | 83.0 | WIN (tiny) |
| SWE-bench Pro | 65.1 | 68.0 | 64.4 | 62.1 | loss |
| SWE-bench Multilingual | 79.6 | 75.7 | 77.9 | 78.4 | WIN |
| DeepSWE | 56.0 | 59.0 | 54.4 | 46.2 | loss |
| NL2Repo | 59.5 | 69.7 | 54.2 | 48.9 | loss |
| Toolathlon-Verified | 71.2 | 76.2 | 70.3 | 48.2 | loss |
| BrowseComp | 86.6 | 84.3 | 84.8 | 85.6 | WIN |
| WideSearch | 80.8 | 72.9 | 77.3 | 79.0 | WIN |
| ClawEval | 81.4 | 80.2 | 77.6 | 78.8 | WIN |
| GPQA Diamond | 92.8 | 93.6 | 91.4 | 91.2 | loss (tie) |
Read the wins/losses honestly and the picture is the tweet's exact claim: "similar performance to Claude Opus 4.8." Not "beats it." Not "trails it." It wins on terminal work, multilingual SWE, web/compute browsing, and agentic coding benchmarks; it loses on NL2Repo, Frontier-Bench, and the longest-horizon DeepSWE tasks — which is exactly where a 397B MIT model has no business competing. And it clears DeepSeek V4 Flash and GLM-5.2 on nearly every row.
How to read this: these are vendor-reported figures from models that went live ~2 hours ago. Weights are double-checkable, but no independent leaderboard has been publicly measured yet. Treat every number as "says DeepReinforce."
The part that matters more: the 1.0 → 1.5 jump
The most striking single number in the whole release isn't the 397B. It's DeepSWE: Ornith-1.0 scored 8, Ornith-1.5 scored 56. That's not a refresh — that's a different model.
Why that matters: DeepSWE punishes models that can't hold a long multi-file task together. Ornith-1.0 essentially couldn't (8). Ornith-1.5 is suddenly competitive with the frontier there (56, vs Opus 4.8's 59). When a "self-scaffolding" RL claim shows up as a 7× change instead of a marketing bullet, the training thesis gets a lot more credible.
The sizes people can actually run
The 397B is the billboard, but two of these are the models most of us can actually run:
| Variant | Terminal-Bench | SWE-bench Ver. | GPQA | Where it fits |
|---|---|---|---|---|
| Ornith-1.5-35B-A3B | 67.8 | 79.0 | 89.2 | Gaming GPU / DGX Spark class — 3B active |
| Ornith-1.5-9B | 46.2 | 70.6 | 86.4 | Laptops, Apple Silicon, edge |
The 35B-A3B is the interesting one: 3 billion active parameters hitting a 79 SWE-bench Verified and 89 GPQA. And a 9B dense scoring 70.6 on SWE-Verified is genuinely unusual for its size. Between them, the family covers "runs on my laptop" to "belongs in a data center," all under one MIT roof.
My take
Last month I wrote that Ornith-1.0's real test was whether the self-scaffolding approach could scale into a second generation instead of being a one-model trick. Ornith-1.5 just answered: the DeepSWE 8→56 jump is exactly the kind of evidence that turns a clever training trick into a credible research program.
The honest caveats are real — vendor-reported, hours old, some benchmarks judged by an Opus-family model, and the 397B still needs multi-GPU hardware to serve. But I'm watching the 35B-A3B. If an independent run confirms a 3B-active model around 79 on SWE-Verified, that's the model that changes what "open agentic coder" means for people with one gaming card.
I'll rerun the numbers once real benchmarks land.
Related on this blog: Nvidia Is Buying the Home of Open Weights in a $12.9B Deal · MiniMax Music 3: Five-Minute Songs, Open Weights, One $20M Catch
Comments
Post a Comment