Scorers

Availability

Open core · the llm_as_judge scorer is commercial (Enterprise / Cloud)

A scorer turns a result and its ground-truth item into a numeric score. An evaluation run applies one or more scorers to every result. Most scorers are built into the source-available core; llm_as_judge ships in the commercial edition.

Built-in scorers

Scorer	What it measures	Needs context
exact_match	The result exactly equals expected_output.	No
rule_based	Match against configured rules (keywords, patterns).	No
tool_call_success	Whether the expected tool call succeeded.	No
tool_call_precision	Precision of the tool calls the agent made vs. expected.	No
precision_token_overlap	Token-level precision of the result against expected_output.	No
recall_token_overlap	Token-level recall of the result against expected_output.	No
groundedness	Whether the result is supported by the supplied context.	Yes
latency	The run's latency.	No
cost	The run's token cost.	No

Scorers that reference context — groundedness and the commercial faithfulness — require each dataset item to carry a context field. Without it they cannot score.

llm_as_judge (commercial)

llm_as_judge uses a model to grade results on faithfulness — whether the result is faithful to the supplied context rather than fabricated. It is registered by ee/scorers-pro through the splyntra.scorers entry point and is available in the Enterprise and Cloud editions; the Community core does not include it.

Like groundedness, faithfulness scoring requires each dataset item to include context.

The scorer catalog

GET /v1/scorers returns the catalog of scorers available to the running collector, including their identifiers and whether they require context. In the Community core the catalog lists only the built-in scorers; where ee/scorers-pro is installed it also lists llm_as_judge.

Next steps

• Evaluation — datasets, runs, baselines, and the leaderboard.
• CI regression gates — gate a build on scorer results.