Jev: A Model Built for Subjective Metrics

TypeSafe AI released a model this week called Jev. It has no text output. You send it some input and a list of questions, and it sends back decisions: which option fits, where something lands on a scale, or how likely a statement is to be true. Every answer comes with a probability attached.

I've been scoring content with LLMs since 2023, first in a song search demo and then in Klyde, a Chrome extension I built to run subjective metrics on anything in your browser. Klyde has since been retired. Reading the Jev docs, I recognized the idea right away. It's the same one I was building around, done at the model level instead of inside a prompt. It's good to see someone build it.

What Jev does

Jev takes a state and a set of typed questions. The state is whatever you're judging: a support ticket or a chunk of JSON from your app. The questions come in three types.

Choice picks one option from a list you define. Route this ticket to billing, technical, or sales. It supports up to 255 options.

Score places the input on an ordered scale of 2 to 10 levels that you describe. How frustrated is this customer, from "calm, just stating facts" to "very angry, strong language." The result can land between two levels.

Noul is TypeSafe's name for a yes/no question answered as a probability between 0 and 1. Is the customer explicitly asking for a refund? 0.95.

In their Python SDK, a ticket-routing call looks roughly like this:

response = client.system_one(
    state=ticket,
    questions={
        "department": Choice(
            instructions="Which team should handle this",
            criteria={
                "billing": "Payment or subscription issues",
                "technical": "Bugs or integration problems",
                "sales": "Pricing or account questions",
            },
        ),
        "frustration": Score(
            instructions="How frustrated the customer appears",
            criteria=[
                "Calm, just stating facts",
                "Frustrated but civil",
                "Very angry, strong language",
            ],
        ),
        "refund_requested": Noul(
            instructions="The customer is explicitly asking for a refund",
        ),
    },
)

Every question is answered in parallel in a single call. Answers can only be values you defined, so there's no malformed JSON to catch and no invented category to clean up. TypeSafe calls this a System One model, after Kahneman's fast, intuitive mode of thinking. The target is the kind of judgment a knowledgeable person makes in a second.

All three are subjective metrics

A Choice is a category. A Score is a rubric. A Noul is a yes/no check. Each one is a question with a fixed set of possible answers, about a quality that a rule can't decide and similarity search can't see. That's what I've been calling subjective metrics.

Klyde was built around this. You wrote prompt sets for a domain, and each prompt asked a frontier model to rate the content on one quality. The song set scored lyrics 0 to 10 on qualities like romantic, violent, political, and sad. The persuasion set checked articles for cherry picking, ad hominem, scapegoating, and false dichotomies. Every one of those prompts was a Score, a Noul, or a Choice with extra steps: a prompt template and a general-purpose model on one end, and parsing to pull a number back out of the text on the other.

Jev makes those three shapes the entire interface.

Classification is a big part of the job

A lot of what LLMs do in production is deciding. Which queue does this ticket go to. Is this safe to publish. Does this passage answer the question. Does a person need to look at this. The answer to each one is a word or a number.

Using a frontier chat model for those calls works, but you pay for a lot you don't use. You pay for the instructions telling it to return JSON and for every token it generates. When the JSON comes back broken, you pay again for the retry. Then you wait seconds for an answer that's one word long.

A model built for the job skips most of that. TypeSafe prices input at 4.2 cents per million tokens and doesn't charge for output at all. Agrici Daniel released an open-source SEO audit built on Jev the same day, and a full run on a real site made 78 Jev calls for about two cents. At that price, scoring every document in a corpus on five axes stops being a budget decision.

Confidence is the piece I didn't have

The part of Jev I'm most interested in is the calibrated confidence. In Klyde, if you wanted to know how sure the model was about a 7, you asked it, and the answer was another generated number. TypeSafe's launch post makes the same point: models asked for a confidence estimate tend to be overconfident and inconsistent.

Jev returns a confidence with every answer, and TypeSafe trains it so that higher confidence means higher accuracy. That turns a score into something you can route on. The SEO audit uses a simple split. A Noul above 0.80 counts as yes, below 0.20 counts as no, and anything in between gets flagged for a person to check.

That's the split I use in most of my automation work. Let the model handle the volume and send the unclear cases to someone who can make the call. A confidence number you can trust is what lets you do that without reviewing everything.

The metric is still the work

Jev only answers the questions you write, with the options you define. The better the categories, the better it performs. So the skill that matters is writing the metric.

The SEO audit has a clean example of this. The author ran an A/B test on 30 pages, comparing the original question wording against a sharper rewrite. Decisive answers on page questions went from 47% to 80%. The model and the pages didn't change.

TypeSafe's own guidance for Score levels is "describe situations, not degrees." Compare two ways of scoring the urgency of a support ticket:

urgency (weak)
  low
  medium
  high

urgency (better)
  General question, no deadline mentioned
  Blocking one user's work, workaround exists
  Blocking the customer's production system, no workaround

The first version makes the model guess what you mean by "high." The second tells it. It also tells the next engineer who reads the code.

A few rules I've landed on for writing metrics, all of which carry straight over to Jev:

1. Start from the decision. Know what happens downstream of the score. If nothing branches on it, you don't need it.
2. One quality per metric. "Clear and persuasive" is two metrics. Combine them and the score stops meaning anything.
3. Keep axes independent. Romantic and sensual will move together. Pick one or sharpen the difference.
4. Give the model a way out. A Choice without an "other" option forces every odd input into the wrong bucket. TypeSafe's docs recommend the same.

LLMs can help you draft these. Klyde shipped with a starter prompt that suggested metrics for whatever page you pointed it at, and it was a useful place to begin. What it couldn't do was tell you what you were actually testing for, because that depends on a decision only you know about. It's still just a metric. It's only as useful as the question behind it.

What I'm trying next

Jev is in early access and I haven't run it against my own work yet. The first thing I want to do is take Klyde's old prompt sets and rewrite them as Jev questions. Some will convert cleanly. Some will probably fall apart once I have to describe each level as a concrete situation instead of a number from 0 to 10. I suspect those were weaker metrics than I thought.