Gate CI on evals

Availability

Open core · Cloud requires the Pro plan or higher

This guide takes you end to end: push a dataset, score a run against it, set a baseline, and then block a pull request whenever a change regresses quality. On Splyntra Cloud, evaluation and regression gates require the Pro plan or higher; self-hosted Community includes them. See Evaluation.

1. Push a dataset

A dataset is a set of items — an input, an expected_output, and optional context. Push it once; reference it by id thereafter.

Python

from splyntra import eval as ev

dataset_id = ev.push_dataset("refund-policy-qa", [
    {"input": "What's our refund window?", "expected_output": "30 days"},
    {"input": "Do we refund shipping?",     "expected_output": "No"},
])

TypeScript

import { pushDataset } from "@splyntra/sdk";

const datasetId = await pushDataset("refund-policy-qa", [
  { input: "What's our refund window?", expected_output: "30 days" },
  { input: "Do we refund shipping?",     expected_output: "No" },
]);

2. Run with a gate

Run your agent over the dataset, collect {input, actual} results, and score them. Set gate so the run fails when scores fall below the baseline; set the baseline once from a known-good run.

Python

from splyntra import eval as ev

results = [{"input": item["input"], "actual": run_agent(item["input"])}
           for item in dataset]

report = ev.run(
    dataset_id,
    results=results,
    scorers=["exact_match"],
    gate=True,           # fail if worse than the baseline
    set_baseline=False,  # set True once, on a trusted run
)

TypeScript

import { runEval } from "@splyntra/sdk";

const results = dataset.map((item) => ({
  input: item.input,
  actual: runAgent(item.input),
}));

const res = await runEval(datasetId, results, { gate: true });
if (!res.passed) process.exit(1);

Establish the baseline first: run once on main with set_baseline=True (Python) / the equivalent baseline option, so later runs have something to compare against. See CI regression gates and Scorers.

3. Wire the CLI into CI

For CI you usually don't need SDK code at all — the splyntra CLI runs a gated eval and exits non-zero on regression, which is all a build step needs. See the CLI.

splyntra eval run --gate

A non-zero exit fails the job.

4. Fail the build on regression (GitHub Actions)

Add a job that installs the SDK/CLI and runs the gate. A regression makes splyntra eval run --gate exit non-zero, which fails the check on the pull request:

name: eval-gate
on: [pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    env:
      SPLYNTRA_API_KEY: ${{ secrets.SPLYNTRA_API_KEY }}
      SPLYNTRA_ENDPOINT: https://ingest.splyntra.com
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install splyntra
      - name: Run gated evaluation
        run: splyntra eval run --gate

Store your ingest key as an encrypted repository secret. When the gate fails, the check turns red and the PR is blocked until scores recover or the baseline is deliberately updated.

Next steps

• Evaluation — datasets, runs, and the leaderboard.
• Scorers — built-ins and llm_as_judge.
• CLI — the full splyntra command reference.