Features · Evals
Make “better” a number.
Write down what a good answer looks like once. Then every change to a prompt can be checked against it.
Cases in, a score out.
An eval holds cases, each an input and what you expect. A method scores the answers from 0 to 1 and a threshold, 0.8 unless you set another, says whether the version passes. Run version 3 and version 4 on the same cases and compare.
- Methods: exact, contains, structural, judge or hybrid
- A threshold you set per eval
- Past runs are kept so you can see the trend
{ "tool": "evals", "action": "create",
"args": { "entity_type": "prompt", "entity_id": "<id>",
"title": "Reply stays within the rules",
"method": "contains", "threshold": 0.8,
"cases": [ { "id": "not-found", "input": "Where is order Z-9?",
"expected": "could not find" } ] } }Scoring
Five methods.
Exact
The answer must equal what you expect.
Contains
The answer must include what you expect.
Structural
The answer must have the shape you expect, such as valid JSON with certain keys.
Judge
A model reads the answer and scores it against your expectation.
Hybrid
A mix of the above for cases that need both a rule and a judgement.
Honest note
Who runs the model?
- Pre-betaYour own AI runs and scores
Your own AI runs and scores: it can read the saved cases over MCP, run the prompt, score the answers and tell you the result in the conversation.
Questions
About evals.
Is “better” an opinion here?
No. An eval scores every case from 0 to 1 and compares the result against a threshold you set, 0.8 by default. Run two versions on the same cases and the difference is a number.
Who runs the eval?
Your own AI. It reads the saved cases over MCP, runs the prompt, scores the answers and tells you the result in the conversation. The running is done by the AI you bring.