Ornith-1.5: Check the Results for the Size You Can Run
MODELS & COSTS
Ornith-1.5: Check the Results for the Size You Can Run
The flagship's comparison with Opus tells a reader little about the smaller Ornith checkpoint without its own results. The family shares a release, not identical capability. For a local deployment, start with the exact artifact, license and tests for the size the machine can host.
In this article
The headline comparisons are the team's own evaluations. They include gains over Ornith-1.0, but the flagship's score does not transfer automatically to the 35B-A3B release.
The model cards list MIT-licensed weights. The 397B results lead Opus 4.8 in some rows and trail it in others; this is a task-specific vendor comparison, rather than an overall independent ranking.
What is Ornith-1.5?
Ornith is the open-source LLM family from DeepReinforce (the five-person team behind the GrandCode and CUDA-L2 reinforcement-learning papers). What made 1.0 stand out wasn't just the numbers — it was the training recipe: instead of handing the model a fixed testing harness, they train it with reinforcement learning to write its own task scaffolds (memory, routing, the scaffolding an agent leans on) while it solves the task. It's "self-improvement" conditioning — the model learns the harness and the answer together, with anti-reward-hacking safeguards bolted on.
Ornith-1.5 is that recipe's second act. It comes in three sizes:
| Variant | Type | Ready for |
|---|---|---|
| Ornith-1.5-9B | Dense | Edge, laptops, Apple Silicon |
| Ornith-1.5-35B-A3B | MoE, 3B active | Gaming GPUs, DGX Spark class |
| Ornith-1.5-397B | MoE | Multi-GPU / enterprise |
The three releases use Qwen 3.5 backbones and list MIT licenses, 262K context and multimodal input. Published conversions include GGUF, FP8, NVFP4 and MLX; actual support still depends on the selected artifact and runtime. The 397B card's region:us tag is metadata, not by itself evidence of a regional license restriction. Read the license text and hosting terms for those conditions.
The 397B vendor comparison with Opus 4.8
All figures below are DeepReinforce's official vendor numbers, averaged over five independent runs (full harness notes in the model cards). Here's how the flagship stacks up against Opus 4.8 and the other heavy hitters:
| Benchmark | Ornith-1.5 397B | Claude Opus 4.8 | DeepSeek V4 Flash | GLM-5.2 | vs Opus |
|---|---|---|---|---|---|
| Terminal-Bench 2.1 (Terminus-2) | 86.1 | 85.0 | 82.7 | 81.0 | WIN |
| Terminal-Bench 2.1 (Claude Code) | 85.2 | 78.9 | 81.8 | 82.7 | WIN |
| SWE-bench Verified | 86.0 | 85.8 | 81.6 | 83.0 | WIN (tiny) |
| SWE-bench Pro | 65.1 | 68.0 | 64.4 | 62.1 | loss |
| SWE-bench Multilingual | 79.6 | 75.7 | 77.9 | 78.4 | WIN |
| DeepSWE | 56.0 | 59.0 | 54.4 | 46.2 | loss |
| NL2Repo | 59.5 | 69.7 | 54.2 | 48.9 | loss |
| Toolathlon-Verified | 71.2 | 76.2 | 70.3 | 48.2 | loss |
| BrowseComp | 86.6 | 84.3 | 84.8 | 85.6 | WIN |
| WideSearch | 80.8 | 72.9 | 77.3 | 79.0 | WIN |
| ClawEval | 81.4 | 80.2 | 77.6 | 78.8 | WIN |
| GPQA Diamond | 92.8 | 93.6 | 91.4 | 91.2 | loss (tie) |
Read the wins/losses and the picture is the tweet's exact claim: "similar performance to Claude Opus 4.8." Not "beats it." Not "trails it." It wins on terminal work, multilingual SWE, web/compute browsing, and agentic coding benchmarks; it loses on NL2Repo, Frontier-Bench, and the longest-horizon DeepSWE tasks — which is exactly where a 397B MIT model has no business competing. And it clears DeepSeek V4 Flash and GLM-5.2 on nearly every row.
How to read this: these are vendor-reported figures from models that went live ~2 hours ago. Weights are double-checkable, but no independent leaderboard has been publicly measured yet. Treat every number as "says DeepReinforce."
Reported changes from Ornith-1.0
The most striking single number in the whole release isn't the 397B. It's DeepSWE: Ornith-1.0 scored 8, Ornith-1.5 scored 56. That's not a refresh — that's a different model.
Why that matters: DeepSWE punishes models that can't hold a long multi-file task together. Ornith-1.0 essentially couldn't (8). Ornith-1.5 is suddenly competitive with the frontier there (56, vs Opus 4.8's 59). When a "self-scaffolding" RL claim shows up as a 7× change instead of a marketing bullet, the training thesis gets a lot more credible.
Smaller variants and deployment requirements
The 397B is the billboard, but two of these are the models most of us can actually run:
| Variant | Terminal-Bench | SWE-bench Ver. | GPQA | Where it fits |
|---|---|---|---|---|
| Ornith-1.5-35B-A3B | 67.8 | 79.0 | 89.2 | Gaming GPU / DGX Spark class — 3B active |
| Ornith-1.5-9B | 46.2 | 70.6 | 86.4 | Laptops, Apple Silicon, edge |
The 35B-A3B is the interesting one: 3 billion active parameters hitting a 79 SWE-bench Verified and 89 GPQA. And a 9B dense scoring 70.6 on SWE-Verified is unusual for its size. Between them, the family covers "runs on my laptop" to "belongs in a data center," all under one MIT roof.
The reported DeepSWE increase from 8 to 56 is substantial. Reproducing it requires the same task version, scaffold and scoring rules. It supports further investigation of the training approach, without isolating which change caused the gain.
For a smaller machine, inspect the 35B-A3B artifact and its own test results. Three billion active parameters do not mean three billion parameters to store. A reported SWE-Verified score near 79 still needs independent reproduction and a measured memory budget before it becomes a single-GPU recommendation.
Related on this blog: Nvidia Is Buying the Home of Open Weights in a $12.9B Deal · MiniMax Music 3: Five-Minute Songs, Open Weights, One $20M Catch
Comments
Post a Comment