Jev vs LLMs, closed and open

The AI model that can't write a sentence beat Claude Opus 4.8 at reading contracts.

TLDR. I took three Texas government RFPs I had already labeled by hand in my Thinking Machines experiment: 438 total rules, and gave them to a new decision model called Jev.

Jev can't write. It only picks from options you give it. It still beat every model from my July bake-off, including Claude Opus and three fine-tuned open models, and it read each RFP for about two cents.

Then I built a bid/no-bid reader on top of it. I ran it on those three plus one more from July, and it said no to all four, and each time it named the rule that kills the bid, with the page number.

Its one real weakness is its confidence numbers. When Jev says 70%, believe roughly 20%.

What Jev is

Most models we've heard of are writers. We ask, they compose an answer word by word.

Jev, launched by it's makers TypeSafe AI , is a sorter. You hand it some text and a fixed set of slots, and it drops the text in one slot and tells you how sure it is.

It answers in about a fifth of a second. It charges only for what it reads, since it produces no text to bill for.

The catch is that it can't explain itself. There is no "because." You get a choice and a number.

That changes how you frame the task.

An LLM can be asked to "list every requirement." Jev can't list anything. So I split each RFP into sentences and asked Jev four questions about every one of them:

1. Is this a requirement?
2. What type: a hard "shall," a softer "should," or just information?
3. What category: work to deliver, eligibility, a form, a deadline?
4. Could failing it get the bid thrown out?

1,311 sentences across the three RFPs. 1.35 million tokens in.

Total bill: 5.7 cents. Wall time: under a minute.

The job

When a Texas government agency buys something, it publishes an RFP, a request for proposals. They usually run 25 to 120 pages. Buried in the legal text are a few hundred rules: things the vendor must do, must have, or must send.

Miss one and your bid can be thrown out before anyone reads your price.

Bid teams comb through these by hand and build a list called a compliance matrix. It's slow, dull, and it decides whether the rest of the work matters.

In July I turned that into a contest. I labeled three RFPs sentence by sentence, 438 rules in total, and made that the answer key.

Then I trained three open models on eleven other RFPs and pitted them against Claude. A fine-tuned Kimi won. I wrote that up at the time.

This month I got early access to Jev, from TypeSafe AI, and wanted to know one thing. Could a model that doesn't write at all do this job?

How I scored it, and what F1 means

Grading a rule-finder takes two numbers.

Recall asks: of the 438 real rules, how many did you find?

Precision asks: of everything you flagged, how much was actually a rule?

You can cheat either one. Flag every sentence in the document and you get perfect recall and terrible precision. Flag only the one most obvious rule and you get perfect precision and find almost nothing.

F1 blends the two into a single score from 0 to 1, and it punishes lopsidedness. The formula is 2 × precision × recall ÷ (precision + recall). On the smallest RFP, Jev's precision was 0.60 and its recall was 0.74:

2 × 0.60 × 0.74 ÷ (0.60 + 0.74) = 0.66

A plain average would give 0.67. F1 lands a bit lower, and the more lopsided the two numbers get, the harder it drags toward the weaker one. You can't buy a good F1 by maxing one side.

Macro-F1 just means I scored each RFP on its own and averaged the three, so the 74-page RFP doesn't drown out the small one.

Scoring was plain code, no model judging. A flagged sentence counts as a hit if it lands in the right section and shares enough words with a rule in the answer key.

Before running anything, I wrote down two bets that could fail:

• Jev's macro-F1 beats the July winner's 0.57.
• When Jev says it's very sure, it's right within 10 points of what it claims.

The scoreboard

It won on recall. Jev found 354 of the 438 rules, 81%. Claude found about 4 in 10. Kimi found about half.

What's the catch? It lost on precision. About half of what Jev flagged wasn't a rule. Claude was the careful one: on one RFP, 91% of its flags were right.

Which one matters more? F1 treats them as equal. Real world context changes.

A false flag costs someone ten seconds to cross off. A missed rule can cost the whole bid. For this job, I'd take Jev's trade every time.

It also sorted rules by type almost perfectly, 95 to 99% right. Category was weaker, 60 to 77%. The line between "work to deliver" and "eligibility" is blurry even for people.

Bet one held.

Where it broke: the confidence number

Bet two failed.

When Jev said it was about 94% sure, it was right about 82% of the time. That's a 12-point gap, just over the 10 I allowed.

The middle was far worse. When Jev said 50 to 80%, it averaged about 65%. The real hit rate in that band was 18%.

This matters the moment you put a slider on a screen. Any product that says "show me rules above 70% confidence" is quietly lying to its user.

The ranking is good, since higher really does mean more likely. The number itself isn't a probability.

The fix is cheap: pick the cutoff from measured results, never from the label. On this data the best cutoff was 0.8, which lifted macro-F1 to 0.71.

A bug in my code, fixing improved outcome

Jev can only pick sentences it's shown. So the code that cuts an RFP into sentences sets a ceiling on how well any sentence-picker can do.

My code had a bug. It treated numbered list items ("1.") and street addresses ("1100 West...") as section headings, which scrambled classification entirely.

With a perfect model, only 78% of the answer key was even reachable. After the code fix, it went up to 91%.

That was the single biggest gain in the project, and it came from plain code.

From finding rules to making a call

A list of 200 rules still leaves a person to decide. So I built the step a bid lead actually cares about: bid, no-bid, or look closer.

It works in three passes, all Jev:

1. Keep the sentences Jev is at least 80% sure are rules.
2. Check each work and eligibility rule against a capability library, 65 short write-ups of past projects and skills, and ask: met, partial, or gap?
3. For every rule that could get a bid thrown out, ask what kind of gate it is. Is it a qualification the firm must already have, like years in business? Paperwork it can go get, like a state certificate? Or just procedure text?

Then plain code, not the model, makes the call:

• A missing qualification, flagged as a gap at 70% confidence or higher, means no-bid. You can order a certificate by Friday. You can't order five years of history.
• Otherwise, covering half the required work means bid. Less means look closer.

Jev answers. Code decides.

I did that on purpose. The rules fit in eight lines of code a bid lead can read and argue with.

I ran it on four RFPs: the three scored ones, plus a live government bid from my July run. That fourth one has no answer key, so it gets a bid read but no score. All four came back no-bid, each for a reason I could check in seconds:

• A statewide IT contract (219): the vendor must supply acSoft's ServiceNow connector for Power BI, one named company's product. Page 9, section 2.2. Jev was 99% sure.
• A program-management contract (177): an account manager with five years of experience (page 21), and five years in business plus proof of nonprofit status (page 57).
• An HHS system contract: "Prior to award of the Contract, Respondent must obtain TX-RAMP Level 2 certification." Page 6.
• A live HHS contract: three years of relevant experience in a healthcare setting. Page 22, section 4.2.2.

That last one is the one I ran in July. The fine-tuned Kimi pipeline read it and reported 8 rules met, 29 partial, 152 gaps.

My conclusion then was "partner or pass." Correct, but vague.

Jev's version names the sentence. That's the whole difference between a report and a decision.

Each full read costs 3 to 17 cents and takes 7 to 23 seconds.

Jev vs Kimi on matching

For step 2 I had a second opinion on hand: the July Kimi pipeline had already judged the same 551 rules against the same library.

They agreed 85% of the time. That sounds great until you notice 84% of the rules are gaps, so two models that both say "gap" a lot will agree by accident. Adjusted for chance, the agreement is moderate at best.

So I had Claude Opus judge 30 of the disagreements blind. Kimi was right 16 times, Jev 13, neither once.

The pattern was more useful than the tally.

When Kimi said partial and Jev said gap, Jev was right 11 of 13 times. When Jev said partial and Kimi said gap, Kimi was right 13 of 13.

Each model over-calls "partial," on different rules. So "partial only when both agree" beats either one alone.

Also, Jev dodged a mistake Kimi made in July, calling the RFP's own posting date a requirement the firm had "met."

What makes this test unfair

Three things:

First, I built the answer key one sentence at a time, from the same splitter. Jev picks one sentence at a time. The test's shape suits Jev, and a paragraph-level answer key could close the gap.

Second, I found the splitter fix by measuring against the answer key. That's a mild leak. The fix is a real bug, but I saw it because I was looking at the key.

Third, the Opus judge in the matching round is a model, not a human with a red pen.

Lastly: Jev isn't perfectly repeatable. The same question twice can move its confidence by about 4 points. Its choice flips only when it was below about 40% to begin with.

What it costs to run, and why that matters when you design a system

Price per RFP was my headline. For anyone building a product, unit economics live in outcomes now.

Here is what each model read and wrote across the three scored RFPs.

Jev's numbers come from its own usage report. The LLM numbers are my estimates from the size of the RFP text and their saved answers, at about 3.8 characters per token. Claude also thought before answering, and those tokens aren't in the saved answers, so its real count is higher. I didn't log the LLMs' run times in July, so I won't guess them.

Jev reads about 20 times more than the LLMs do. Every call carries the question wording, the category definitions and the sentence before, so each sentence costs about a thousand tokens to read.

It still costs far less: 2 cents an RFP, against 22 cents for the small Qwen and $3.30 to $4.70 for Kimi and Claude. Reading is the cheap part. A model reads a whole page at once, in parallel. It writes one token at a time, and every token waits for the one before it. The LLMs spend their money and their time on the writing.

That leads to six things I'd want from any system built on this.

1. The output can't break. Jev answers from a fixed list, so there is no JSON to parse and no half-finished answer. In July, the untrained open models often thought out loud for 100,000+ characters and ran out of room before writing the answer. That's how Qwen scored zero on one RFP. The fine-tuned Qwen got stuck repeating one row hundreds of times on 5 of its 9 chunk runs, and I needed a guard that caught and reran those. Inkling's first fine-tune flagged 779 rows on an RFP that has 181, from the same kind of loop.
2. Failures stay small. One bad Jev call loses one sentence. One bad LLM call can lose a whole chunk of the RFP.
3. Cost is known before you start. Jev's bill follows the length of the document. An LLM's bill depends on how much it decides to write.
4. It's fast enough to sit behind a web page. The live bid read takes 7 to 23 seconds, with 12 calls running at once, each answered in about a fifth of a second.
5. Every decision can be checked. Each flag points to one sentence and one page, and eight lines of plain code make the bid call.
6. Context limits stop mattering. Kimi's window holds 32,768 tokens, so it has to read each RFP in pieces, and I trained all the open models on pieces for the same reason. Untrained Inkling read whole RFPs in its 65,536-token window, and two of its big answers were cut off mid-list. Jev never sees more than one sentence and the one before it.

It's not free lunch, here's what you give up.

• Lots of small calls. 1,311 calls instead of a dozen means you manage rate limits and retries. Mine ran 12 at a time.
• You must know the options up front. Jev can only pick from choices you wrote. The work of framing the question moves from the model to you, and my splitter capped the score until I fixed it.
• No reasons. You get a choice and a number, never a "because."
• The number needs checking. As shown above, its middle confidence band is badly off.
• It can't write. The proposal itself still needs a writer.

Where else this pattern fits

The pattern is simple. Cut the work into small pieces. Ask a multiple-choice question about each piece. Let plain code make the call. Save the LLM for writing at the end.

Anywhere a person reads many small things and sorts them, it fits:

• Security questionnaires. Check each question against your written policies: met, partial, or gap. Same shape as the capability match here, real time monitoring/ alert costs go down
• Support tickets. Route each ticket by product, urgency, and whether a human must see it.
• Invoices and expenses. Code each line to an account and flag the ones that break policy.
• Content moderation. Allowed, needs review, or removed, one post at a time, with a cost you can predict.
• Inside an AI agent. Use it for the routing step (which tool should handle this?) or the checking step (does this answer cite its source?), so the writer only runs when it has to.

What I'd want to look into

My position: for the reading-and-sorting half of document work, a decision model beats the writers, at a tenth of the cheapest one's price. Keep the LLMs for the half that writes.

To disprove this, I plan to label a few RFPs at the paragraph level, rerun everyone, and if fine-tuned Kimi beats Jev, the win was the test design, not the model.

I'm also curious where models like Jev actually fit in a security workflow?

1. Jev does the work itself, because it is by far the cheapest option.
2. Jev does the first pass and a big LLM handles the hard cases. Jev routes or filters, the LLM writes the finding.
3. Jev is not good enough, and we should just use an LLM and pay for it.

I plan to run an experiment to find out.

Where that leaves you with building

If you build with AI, look at your pipeline and find the steps that are really multiple choice: is this relevant, what kind is it, does it pass. Those steps may not need a writer at all.

If you use a model's confidence score anywhere a person sees it, measure it before you show it. A 70% that means 20% is worse than no number.

And before you swap models, check what the model is being fed. My biggest gain came from fixing the code in front of the model, not from the model itself.

The page is live at prithvidatla.com/evals/jev.

Originally published on LinkedIn, September 25, 2026.