In an independent head-to-head, TypeSafe's Jev answered classification calls for a twelfth to a twenty-seventh of what Claude Haiku 4.5 cost. The accuracy is something you build, though, not something you buy. In that reproducible benchmark, Jev scored 62.6% on a 2,000-email phishing test when asked one question, against Haiku's 81.3%. Split into five narrow questions, with weights fitted on 1,000 labelled examples, it reached 95.0% on the other 1,000. Its probabilities, the feature that is supposed to let software act on its answers without a human, came out overconfident on some question types and underconfident on others. So the move this quarter is a shadow eval on your own labelled decision log, which costs cents, before a single production decision moves.
Jev launched on September 15. Within 24 hours, by Vercel's own count, roughly 13% of its paid AI Gateway teams were calling it. The independent evidence arrived over the next five days. Here is what it says.
What Is Jev, and What Did TypeSafe Ship?
Jev is a decision model. You send it text (the "state") plus a set of typed questions, and it sends back a Choice, a Score or a yes/no answer, each with a probability attached. It never writes a sentence. A decision model is a model that picks from a fixed set of answers you define in advance and tells you how sure it is, instead of generating free text for your code to parse.
TypeSafe came out of stealth on September 15 with a $40 million seed round led by DCVC. Its CEO and co-founder is Diogo Almeida, a former OpenAI researcher who DCVC credits as a co-inventor of InstructGPT and ChatGPT. TypeSafe's launch post calls the category "System One models", after Kahneman's fast, intuitive mode of thinking. It lists end-to-end latency of 70-500ms and supports up to 255 options on a single question.
The pricing is the hook. TypeSafe's models page lists jev-1.13.0 at $0.042 per million input tokens, with output free. The same page gives a 64k-token budget per request, 32k of it for the state plus the longest question, text input only, and English as the language it's best at. The listed rate limits are 1,200 requests a minute and 250,000 tokens a second, and TypeSafe warns they may change without notice while it scales.
The ecosystem moved faster than the evidence. Vercel put Jev on its AI Gateway through an experimental_evaluate call in the AI SDK, with zero data retention and no-training as per-request options. Its adoption numbers are 2x the GPT-5.6 family and more than 6x Fable 5.1 at the same 24-hour mark. Cloudflare lists jev-1.13.0 with a 32,000-token context. LangChain built routing middleware and a guardrail that blocks risky tool calls such as bash on it. Langfuse published a guide to using Jev as an eval judge.
Why you should care: routing, triage, guardrails and judging make up a big share of the calls in an agent stack, and right now you pay LLM output-token rates for every one of them. Jev is not a model router, which picks which LLM answers (see our model router buyer's guide). It replaces the LLM call on the decision itself.
Is Jev Really 444x Cheaper Than an LLM?
No. Not in any sense you should put in a business case. TypeSafe's headline claim, "193.6x faster, 444.6x cheaper", comes from workflow evals where the correct answer was defined by two other models. So it measures how cheaply Jev agrees with frontier LLMs, not how often Jev is right.
The launch post says so plainly: "We use the average of GPT-6 Astra and Fable 5.1 as the reference answer," and it says it expects its figures to sit at the higher end of real-world gains. It also explains the 0% hallucination rate in its charts: "Our number is not empirical. Schema matching is guaranteed, thus we can confidently add 0% into the plots." That is a type guarantee. Jev cannot return an answer outside your schema. But it can still return the wrong answer inside your schema, and that is the failure you actually care about. The post publishes no calibration metric at all. There's no expected calibration error and no reliability curve, even though the pitch rests on "calibrated" probabilities from a new training method TypeSafe calls RLCD.
The strongest case for TypeSafe's method: agreement with a frontier model is how most teams already run LLM-as-judge when they have no labels. On that basis the numbers are striking. Langfuse reports that in Good Start Labs' test of 6,003 rubric checks, Jev cost $160 per million verdicts and matched Claude's decisions 91.5% of the time, while Claude Fable 5.1 cost $33,000 per million.
But the same Langfuse table carries the counterweight. DeepSeek V4.1 Flash cost $260 per million verdicts and agreed with Claude 93.5% of the time. A cheap LLM at 1.6x Jev's price agreed more often. So the comparison that matters is not Jev against a frontier model. It's Jev against the cheapest LLM that already clears your accuracy bar, and against ground truth you own.
Why Jev Lost to Haiku Until the Question Was Split
On a fully reproducible head-to-head, Jev lost to Claude Haiku 4.5 when both were asked a single question. It matched Haiku only after a person broke the task into five narrow signals and fitted weights on labelled data.
The jev-phishing-bench run, published September 17, fed 2,000 PhishNChips v5.2 emails to both models, with Haiku in non-thinking mode at temperature 0. The email bodies are synthetic and the labels come from URL reputation feeds, not a human reader, so most of the signal sits in the links and the sender. Asked "is this phishing?", Jev scored 62.6% and Haiku 81.3% (McNemar p < 0.0001). Jev caught only 43.2% of the phishing emails, flagged 18.0% of the legitimate ones, and had a calibration error of 0.154 on that single verdict.
Then the author asked five atomic questions in one call, such as whether the links point to shortened URLs or free hosting, and whether the sender uses a free email address while claiming to be an organisation. A logistic regression was fitted on 1,000 emails and scored on the other 1,000. Jev reached 95.0%, Haiku 93.2%, and a two-line regex 91.8%. The gap between Jev and Haiku was not statistically significant (p = 0.063). A 1.8-point difference on 1,000 items rarely is, which is why you should size the eval set before you trust the gap.
The cost gap, on the other hand, is large and real. Per 1,000 emails at list price, Jev cost $0.038. Haiku cost $0.462 for the single verdict and $1.02 for the five signals. That's the 12x and 27x. Median latency from France was 239ms against 687ms. XenoSpectrum's write-up of the benchmark adds one more detail: splitting the question helped only Jev. Haiku's best single signal, at 94.2%, beat its own five-question composite.
Read that carefully. The 95% is not Jev. It is Jev plus your labelled data plus a regression you maintain. Once you have 1,000 labels, you have options. A commenter on the benchmark trained Qwen3-4B with LoRA on 1,000 of the benchmark's emails and got 97.4% accuracy with a calibration error of 0.010 on a disjoint 500, in 18 minutes on one RTX 4070 SUPER. Training was the cheap part of that, and serving it is the real bill. But it's the old BERT-era lesson: with stable criteria and labels in hand, a small trained classifier is hard to beat.
Jev's real advantage is the one XenoSpectrum names: you set the task at runtime through the text of the question. If your triage categories change every quarter, there's no retraining step. The case for Jev is agility, not accuracy.
The fairest counter-evidence comes from a pre-registered evaluation run on September 20: 5,721 calls, 21 experiments and 50 predictions timestamped before any data was collected. On its 400-item benchmark, Jev scored 95.9% zero-shot, against 77.2% for hand-written keywords and 66.0% for a supervised TF-IDF model. The authors admit their own bias: "18 of our 21 misses were in the same direction — we kept predicting failure and kept being wrong." They also found the failure mode that matters most for your rollout: "Wrong criteria descriptions are catastrophic: 16.7 %, below the 25 % random floor." With Jev, the question text is the program. Get it wrong and the model does worse than guessing.
Can You Trust Jev's Probabilities?
Not out of the box. The probability on each answer is what makes a decision model worth wiring into software. Two independent calibration studies say you cannot set a single threshold on it and walk away.
A model is calibrated when the answers it gives at 0.9 probability are right about 90% of the time. Expected calibration error (ECE) is the average gap between the confidence a model states and the accuracy it actually delivers. Zero is perfect.
The out-of-distribution calibration study, run September 19 through Vercel's gateway, tested Jev on 900 freshly generated synthetic support tickets. It measured an ECE of 0.107, 4.4x the 0.024 noise floor of a perfectly calibrated model. The errors ran in opposite directions depending on question type. Yes/no answers were underconfident (refit temperature 0.66). Choice and score answers were overconfident (3.29 and 3.40). On a ticket-priority question whose rule came from an internal policy that never appeared in the ticket text, Jev was right 44.7% of the time while assigning its answers an average probability of 0.74. It was confidently wrong on a task nobody could have answered. Meanwhile, the same study found Jev well calibrated on public benchmarks (ECE 0.024-0.032), which is exactly where the authors expect training-set contamination. It also found that Jev's separate confidence field "was never better than the max probability."
The pre-registered study saw the same shape from another angle. Accuracy above the threshold stayed flat from 0.50 to 0.95, then jumped to 100% at 0.99, covering 60.2% of traffic. It also found that "0 of 30 out-of-scope messages were flagged — at 0.99 confidence." Langfuse describes the underlying behaviour: "A forced binary with no unknown or needs_review option makes Jev pick the least wrong answer instead of saying it does not know." The most favourable result so far, a 60-case test on agent tool-call risk, measured an ECE of 0.05-0.07 and found every answer given at 1.0 confidence correct. Its author notes that 83% of predictions sat in the top bin and that 60 cases is too few to settle it. TypeSafe's own failure-mode documentation concedes there is no guarantee that thresholds transfer between question types. It's the same confident-confabulation pattern we found when we audited Apple's on-device model.
In practice, that means three things. Calibrate per question, not per model. Give every Choice an explicit "none of these" option, as the pre-registered authors recommend. And treat the 0.99 band as the only candidate for automation until your own data says otherwise.
What Stops Jev From Running a Production Decision Today
Jev is in early access and served from a single region. Its rate limits change without notice, and the vendor's own documentation says it trusts whatever you put in the state. That's fine for a shadow eval, but not yet for a regulated decision path.
- Region and latency. The launch post says the service is currently based on the West Coast. The pre-registered study, which routed through OpenRouter, measured a ~430ms floor and advises against sub-300ms loops from Europe.
- Data terms. TypeSafe's models page says Jev is not trained on customer requests or responses, and offers zero data retention to enterprise customers. On Vercel, those protections are flags you set per request. A per-request flag is not a contract term. TypeSafe's models page lists training and retention terms, but no SOC 2 or ISO 27001 attestation. Ask for one before any regulated workload goes near it.
- Prompt injection. The failure-mode page says: "State is data, and jev-1.13 does not treat it as hostile by default." Injected instructions can steer the answer. That matters most in exactly the use LangChain showcased, gating an agent's tool calls, because the text being judged is often text an attacker can write. Put a dedicated injection detector in front of it.
- Version drift. The models page shows
jev-latestandjev-previewboth pointing to jev-1.13.0 today. Pin the version, not the alias, or your calibration fit expires silently the day the alias moves. - Known weak spots. TypeSafe documents that Jev is "not a calculator". It struggles with date comparisons, double negatives, and a large state full of irrelevant detail. The pre-registered study found it better on numbers and negation than the docs admit, which is one more reason to test on your own data. Filter in code first and send only the fields the question needs.
What to Do Before a Production Decision Moves
This Week:
- Pick one high-volume decision you already have ground truth for. Ticket queue assignment, invoice routing and alert triage are good candidates. Pull 1,000-2,000 records where the label is the human's final disposition, not the LLM's first guess.
- Run Jev in shadow against it. The whole pre-registered study, 5,721 calls, cost $0.176 at list price. Your eval budget is a rounding error. Pin
jev-1.13.0, and give every Choice a "none of these" option. - Write the question text as carefully as you'd write code. Have the person who owns the decision review the criteria. A wrong description scored below random.
This Month:
- Fit calibration per question on half the set and score the other half. Find the confidence band where accuracy clears your bar, and measure what share of traffic it covers. That share, not the per-token price, is your real saving.
- Run the same set through your current cheapest LLM, and through a small fine-tuned model if you have the labels. Compare the cost of each correct decision, not the cost per token.
- Attack the state. Plant instructions inside the ticket or email text and see whether the answer moves, before Jev goes anywhere near a guardrail.
Before Renewal:
- Route only the confident band to Jev, and keep the LLM as the fallback. Get zero data retention written into the contract rather than set as a request flag, and ask TypeSafe for a SOC 2 report and a version-deprecation policy before the pilot becomes a dependency.
The Bottom Line
This is the pendulum swinging back. The industry went from task-specific classifiers (BERT, 2018, which XenoSpectrum notes still draws 47 million monthly downloads on Hugging Face) to prompting general LLMs for everything. Prompts were faster to change than retraining a model. Jev offers a third option: a classifier you program with the text of the question, priced so low the per-call cost drops out of the business case.
That part is real, and the adoption curve shows teams know it. What the first week of evidence also shows is that the accuracy and the calibration aren't in the box. They come from how you break the decision down, the labels you check it against, and the per-question fit you maintain. Teams that already have a labelled decision log will get the 27x and keep their accuracy. Teams that don't will get a fast, cheap model that is 0.99 sure about things nobody could know.
Jev prices a decision at a fraction of a cent. It doesn't price being wrong. Your labelled log does.