LLM as a Judge with Jev
Jev is a structured judgment model by TypeSafe. Ask boolean, choice, or score questions and get exact answers back, with up to 32 independent questions processed in parallel per request.
LLM as a Judge with Jev
Use Jev in agent and CLI workflows.
The Kollab Agent invokes Jev automatically when you ask for structured evaluations. You can also run it manually via the CLI for scripts and pipelines.
npm install -g @kollab/cli
kollab login
echo '{
"state": "Help! My payouts have been failing for 3 days.",
"questions": {
"is_urgent": { "type": "noul", "instructions": "Does this convey urgency?" },
"department": { "type": "choice", "instructions": "Which team should handle this?",
"criteria": { "billing": "Payments, invoicing, refunds", "technical": "Bugs, outages, integrations", "sales": "Pricing, upgrades, new accounts" } }
}
}' | kollab tool run jev --jsonPart of the Kollab model armory.
Access Jev alongside a complete suite of models for text, vision, video, and audio tasks.
LLM as a judge for moderation, scoring, and reranking.
Run specific evaluation tasks with Jev.
Performance benchmarks.
Fast execution and high consistency for batch classification tasks.
13 questions in one call: 0.27s vs 2.71s for 13 separate calls
TypeSafe cookbookCLERC legal dataset, 40 queries, BM25 top-30 shortlist
TypeSafe cookbook6,003 ratings checked · $160 vs $33,000 per million judgments
Good Start Labs0.35s median vs 8.83s · detected 6/7 seeded defects
Every.to5,733-email test set, close to a tuned TF-IDF baseline's 98.87%
bitnovus (independent)Real-world applications.
How developers and researchers are using Jev in production.
Spam and Auto-Reply Detection
Freek Van der Herten processes incoming helpdesk tickets with 3 parallel questions in 639ms. This achieves 48 requests per second at a monthly cost of $0.36.
Financial Research Consistency
Good Start Labs uses Jev to verify large sets of financial reports efficiently and consistently.
Writing Vibe Checks
Every.to evaluates drafted articles with Jev. The model flags seeded writing defects with a median speed of 0.35 seconds.
LLM Observability Scoring
Langfuse integrated Jev into their evaluation system to automate prompt scoring and feedback data.
Cloudflare Workers AI Integration
Cloudflare added the Jev model to their Workers AI platform for global edge inference.
Seed Funding Announcement
TypeSafe AI raised $40M to build specialized models for software applications.
Structured State Management
Jev powers an official demo playing Doom through structured game state outputs.
LLM as a judge: common questions.
What does LLM-as-a-judge mean?+
It means using a language model to evaluate text against a question you define, such as "Is this comment abusive?" or "Score this lead from 1 to 5." Teams often use it to grade another model's answers, and the same pattern works for tickets, comments, reviews, and search results. Jev is a structured judgment model that returns exact booleans, choices, and scores instead of conversational text.
How can I use an LLM judge correctly?+
Phrase each question so it has one clear answer: true or false, one of a few choices, or a score on a fixed scale. In Kollab, describe the task and the Agent calls Jev for you, or run `kollab tool run jev` from your terminal. Spot-check a sample of results against your own judgment before relying on them.
How effective is LLM-based judging?+
It depends on the task and how clearly the question is written. The benchmarks above list Jev's official and independent results with their sources, so you can compare them with your own case. Try a small labeled sample first.
What are the limits of an LLM judge?+
Jev's output probabilities are not strictly calibrated, so treat them as relative scores. Accuracy for Chinese, Japanese, and Korean is currently lower than for English. Jev reads plain text up to 80,000 characters, so transcribe audio or extract text from images first; other Kollab models, such as Grok STT, handle audio directly. For high-impact decisions in areas such as finance, law, or hiring, have a person review the results.
What should I look for in an LLM judge framework?+
Look for outputs you can parse, a way to run many questions at once, clear pricing, and a way to inspect confidence. Jev returns structured answers, processes up to 32 independent questions per request, and runs through the Kollab Agent or the CLI.
How is Jev priced?+
Input tokens cost $0.042 per million. Output tokens are completely free.
Start evaluating text with Jev.
Run high-volume classification and scoring tasks directly in your workspace.