What Jev Is Actually Good For: 379 Calls, One Cent
What Jev Is Actually Good For: 379 Calls, One Cent
I posted about Jev's launch numbers a couple of days ago — the 444x headline and the 32x that people actually measured. Then I stopped reading about it and started pointing it at real work: a front page, a comment thread, my own post archive, and the draft you are reading. Everything in this post ran through the API today, and the bill for all of it, 379 requests, was $0.0104.
The thing to get straight first is that Jev is not a model you talk to. It cannot write a sentence, and it refuses to pretend: point a chat completion call at the Jev model name and the API answers 400 — is a decisions model and cannot be used with the chat/completions endpoint. This is why the "run your coding agent on Jev" videos are wrong, and it is also why I did not expect to get much out of it at first.
What it does instead is answer typed questions about a piece of text or JSON — all of them in one pass, no token-by-token loop — and hand back probabilities your code can branch on. So the useful question is not "is it smart" but "which decisions in my day are cheap, closed questions asked hundreds of times". I found four worth keeping, and here is what they cost.
| What I pointed it at | Items | Questions each | Wall clock | Cost |
|---|---|---|---|---|
| A Hacker News front page, gated for relevance | 30 stories | 3 | 11.1 s | $0.00074 |
| The Jev launch thread, read and tallied | 129 comments | 5 | 9.8 s (8 at a time) | $0.00442 |
| My own 68-post archive, searched | 68 titles | 3, in one call | 0.6 s | $0.00028 |
| This post's own claims, checked against sources | 16 citations | 2 | 1.7 s | $0.00064 |
What the answers actually look like
Three question types carry everything, and they are simpler than the marketing language suggests. Noul is a revived old word for yes/no and comes back as one probability. Choice picks one option out of a list you define (up to 255 of them, which matters later). Score puts the state somewhere along an ordered rubric you write. You can mix all three in one request and they all see the same state.
That confidence number is the most misread part of the API. It is a statistic about the shape of the probability distribution, not a promise the answer is right, and the docs say so plainly: calibration holds across groups of predictions, never per-answer. The pattern the docs recommend, and the one I ended up using, is to keep the thresholds in my own code — high confidence acts, middling confidence gets flagged, low confidence goes to me.
answer.confidence on every answer will crash on the yes/no questions. (screenshot: docs.typesafe.ai)Job one: gate a front page before it eats your context
I keep a long list of feeds, and the expensive part is not reading them — it is deciding what deserves reading. So I pulled the 30 current front-page stories and asked three questions per story: which topic is this, would a reader of this blog want it, and how much substance is behind the headline.
Thirty requests, 11 seconds, $0.00074. Four stories cleared my 0.7 relevance line: self-hosted inference orchestrators compared, a Jev build running offline on Apple silicon at 45 decisions per second, Qwen-Image-2.1, and the unfortunately named "I turned Jev into a (lousy) chatbot". One item scored 0.01 — a GameCube decompilation — which is the kind of thing I would normally have opened out of curiosity. Nothing about that run needed a language model's prose; it needed a sorting decision, and the answer cost less than the electricity of my monitor being on.
Job two: read 129 comments in ten seconds
The launch thread on Hacker News is now 506 comments deep, and I wanted the shape of it rather than the vibe of it. I grabbed every top-level comment and its first replies with some actual text in it — 129 of them — and asked five questions per comment: is the author speaking from their own use, what is their stance, what is the single main topic, how promotional is the writing, and is there anything here worth quoting.
python jev.py --batch comments.jsonl --questions-file q.json --out answers.jsonl
# 129 rows, 8 workers, 9.8s wall, 105,110 input tokens, $0.00442, zero errorsWhat came back is the chart below: the biggest single topic is what people could use it for (39), closely followed by what it cannot do (29), with benchmark skepticism third (12). Stance skews to "sees real value, names real problems" (49) over straight positive (38), and the promotional-writing score sat at a median 0.69 out of 2, which is roughly "opinion with reasoning".
Two of those comments are interesting enough to quote, and they point the same direction. A builder: "We've already started using it for some pretty powerful decision tree stuff... The one downside is that the context window is very small (32k.)" And a team that got access early: "Having a second form of verification, where you can ask multiple questions (in the form of Nouls) raised our confidence in the outputs of other models... works incredibly well in concert with LLMs, not as a replacement." (Both from the Hacker News thread.)
The number I did not expect: 2 of 129 substantive comments described first-hand use. That is genuinely different from the launch-week tweet census I wrote about before, where 23% of relevant tweets came from people who had run it themselves — and the difference is probably the room, not the product. X filled up with builders showing screenshots; a Hacker News launch thread fills up with people arguing about the claims. It is a good reminder that "what is the community saying" has a different answer on every platform, and that reading a thread is not the same as measuring one.
Did the labels hold up?
This is the part where a triage tool usually stops being useful, so I checked it by hand. I took ten comments from the batch — every thirteenth — read each one myself, wrote my own stance and topic for it, and compared without letting Jev see my answers.
Stance agreed on 9 of 10. Main topic agreed on 7 of 10, and all three misses were the same boundary: comments attacking the launch's speed and cost comparisons, which I filed under "doubts the benchmarks" and Jev filed under "what it can't do". That is a category design problem on my side, and the docs' Choice page already has the fix — when two options keep getting confused, stop describing them as one-line strings and give each one fields for what it covers and not_for, plus example inputs. I wrote the option list in five minutes over coffee. It shows.
The confidence number did real work here. Three of my four disagreements came back with a confidence under 0.7 on that question, so a threshold of "auto-accept above 0.7, send the rest to a human" would have flagged them for me. One did not: a commenter's topic came back wrong at confidence 0.97. Confident and wrong is the failure mode to plan for, which is why the last step of this pipeline is still me reading a sample.
Job three: search 68 posts without an embedding in sight
Every time I write a post I want to link something older, and I usually end up scrolling my own archive. So I numbered the 68 published post titles, sent them as one state, and asked two separate Choice questions — which post to link from a hands-on piece about cheap decision models, and which post is about running a model on local hardware — plus a Noul question asking whether a local-model post exists at all.
One request, 0.6 seconds, 6,750 tokens, $0.00028. The first question came back at confidence 0.99 with my own Jev launch post at probability 1.000 — which is exactly the link this post needed, and it is the link I used. The second was a different story: the top candidate (a Mac mini RAM post) only reached 0.37, second place 0.22, and confidence dropped to 0.35 — while the Noul check said 0.99 that such a post exists. That pairing is the whole trick, and it comes straight out of the docs' line-by-line search recipe: ranking always returns a best candidate even when nothing fits, so you ask a second question about whether an answer exists at all. Rank where to look, check whether there is anything to find.
The docs push this pattern to 218 lines of a terms-of-service document in one request, which is a practical way to get semantic search without a vector store, a chunking strategy or an index to keep warm. Past 255 candidates you split it into two passes — one Choice picking a window, a second ranking inside it.
Job four: check this post's own claims against their sources
This is the use I have been waiting for someone to build for years: an automatic check that every claim in a draft is actually supported by the thing it cites. So I ran it on this post. I built 16 citations — 13 real claims from the sections above, each with the quote and the source I was leaning on, plus three deliberate failures: one where the quote is not in the source at all, one where the source says the opposite, and one where the source never addresses the claim.
My first attempt asked Jev to do everything: does the source contain the quote word for word, and how does it relate to the claim. It caught all three planted failures — and then it flagged four of my thirteen good citations as fabricated. The quotes were in the sources. I went back to the docs and found the answer in the cookbook this recipe comes from: the fabricated-quote check there is a string match in code, not a model question, precisely because it is the one part of the job with an exact answer that code cannot get wrong. So I rebuilt it that way — normalize whitespace, look for the quote, and only hand the quote's context to Jev to judge whether it supports the claim.
The second run verified all 13 real citations, caught all three failures (contradiction at confidence 0.99, unsupported at 1.00, and the missing quote stopped at the string match before any Jev call was spent), and flagged one claim at confidence 0.66 — the one about TypeSafe's own Hermes skill cookbook. I re-read the source line myself: it is supported, and it still went into the human pile, which is what the threshold is for.
Six questions cost 46% more tokens and 4% more time
Everyone repeats the same advice about this API — ask many narrow questions per call, because they run in parallel — so I measured it. Same state, same data, once with one question and once with six, five repeats each. The one-question calls had a median of 0.359 s and 553 billed tokens. The six-question calls: 0.375 s and 808 tokens. Six times the answers for 4% more wall clock.
The cost model follows from that: you pay for the state once per request, no matter how many questions you attach. Which means the expensive mistake is not asking too many questions — it is calling Jev once per question, or sending it a state it does not need. My comment run averaged 815 input tokens per comment for five answers; the same five answers from a chat model would have meant five prompts with the whole comment pasted into each.
Four rules carried over from a day of this, and they are all in the docs rather than invented here. Keep one judgment per question and combine the answers in code, because a broad question hides several decisions behind one number. Split anything you would describe with "and". Keep the policy in your code — thresholds, escalation, retries — and let the model only produce the reading. And use it for selection rather than generation: find the candidate values in code, let Jev pick one, then copy the chosen value verbatim, which is how you get extraction that cannot invent a value that was never there.
Where it breaks
The vendor's own jaggedness page is refreshingly blunt, and my day matched it. It reads instructions literally and answers the question you wrote rather than the one you meant. It cannot count or do arithmetic — keep the math in your code. It treats dates as text. Its accuracy drops as you add irrelevant detail to the state, which is a real cost when your instinct is to paste in everything. And my personal favourite line in the whole documentation set: "State is data, and jev-1.13 does not treat it as hostile by default." Prompt injection inside the state can move an answer. If a decision gates something expensive, that is your problem to solve in code.
It also cannot explain itself. You get a number, never a reason, so when a label is wrong you debug your question, not its reasoning — which is a different skill from prompt engineering and takes a few rounds to get used to. Non-English input is handled but, in the docs' own words, not equally well; test before you rely on it.
And then there is the part I got wrong, which is worth more than the part I got right. My first version of the "first-hand user" question produced 1 hit in 129 comments. I rewrote it to be explicit about what counts — ran it, built with it, got access — and got 2. Same data, same model, one question rewritten, and the number moved by a factor of two. Neither of those numbers is "the truth"; both are my question being read literally against a sample I then had to check by hand. Thresholds and labels are triage, not verdicts, and the only reason I can state 2 of 129 with a straight face is that I read ten of them myself.
My take
I set myself a rule while running these jobs, and I have kept to it: if the same judgment repeats over eight or so items — classify, route, rank, score, filter, verify — it goes to Jev instead of my main model, and if it is one question about one thing, it does not. That rule has already earned its place in three spots: gating feeds before they reach my context, reading threads I would otherwise skim, and the draft check below.
The draft check is the one that surprised me. Before publishing this, I ran all 32 paragraphs of this post through the same sort of request — four questions per paragraph, looking for unsourced numbers, hype, jargon and anything that needs verifying. It came to $0.00074, and it flagged 17 of the 32 paragraphs for something, which is exactly the failure mode I described further up: the criterion asks whether the source is named in that same paragraph, and half of this post cites everything in the footer instead. I read the list, acted on the two flags that were right (one paragraph named numbers without its source nearby, one was boosterish about the docs), and ignored the rest. A triage list is for deciding what to look at, not for deciding what is true.
The reason none of this feels like a toy to me is the arithmetic. My whole day of work here — 379 requests, 247,059 input tokens, a front page, a 129-comment thread, my own archive, and a claim check on this draft — came to $0.0104, with no failed calls and a median response of 0.55 seconds. At that price a decision stops being something you batch up and starts being something you ask inline, mid-sentence, the way you would call a function.
So the honest summary of what it is good for: not thinking, not writing, not the interesting part of anything. Just the small closed judgments that software makes millions of times a day and that we currently pay full generation prices to ask. I am wiring the feed gate and the comment reader into a scheduled job next, with the thresholds and the escalation in my own code, and I will post what a few weeks of that looks like — including the parts that turn out to be a bad idea.
Sources and method: all measurements are my own runs on 2026-09-21 through OpenRouter (typesafe/jev-1.13-20260917) at the published rate of $0.042 per million input tokens with output tokens free — the API's per-call usage fields match that arithmetic. Response times are wall clock from this machine and include network. Comment data is Hacker News thread 49717558, collected the same day; quotes are attributed by platform, kept verbatim, and no individual is named. The front-page batch used the Hacker News public API at the same timestamp. The three planted citation failures are mine, not the vendor's. TypeSafe documentation referenced: primitives · Choice · Noul · state · models and pricing · double-checking citations · line-by-line search · jaggedness. Earlier post on the launch numbers and the field data: Jev: 444x Cheaper on the Homepage, 32x in Real Runs.
Comments
Post a Comment