Evaluation · CI/CD Regression Testing

Continuous evaluation, golden datasets & CI/CD regression gates

Ship agent updates with complete confidence. Turn edge-case production traces into test datasets with one click, run custom LLM-as-a-judge scorers, and gate pull requests in GitHub Actions if agent quality or safety regresses.

Start free on this feature →View documentation Book technical demo

.github/workflows/eval.yml

yaml

# Automated CI regression gate
- name: Run Splyntra Agent Evaluation Gate
  run: |
    splyntra eval run \
      --dataset "prod-edge-cases-v2" \
      --threshold-accuracy 0.95 \
      --max-risk-score 15 \
      --fail-on-regression

# ✓ 120 / 120 Test Cases Passed
# ✓ Accuracy: 98.2% (+1.4% vs main)
# ✓ P95 Latency: 1.2s (Within SLA)

1-Click

Trace to Dataset

Convert production failures into test fixtures.

LLM Judge

Scorers

Semantic accuracy, tone, safety, & tool correctness.

CI Gated

GitHub Actions

Block regressions before reaching production.

Version Diff

Benchmarking

Side-by-side prompt & model comparisons.

Engineered for high-throughput autonomous agents

Every capability is built into the OpenTelemetry streaming pipeline with sub-millisecond ingestion overhead.

01Dataset Curation

1-Click Production Trace to Golden Dataset

Effortlessly capture weird edge cases, customer escalations, and failed tool runs from production telemetry into versioned test datasets.

✓Filter production traces by high risk scores or user dissatisfaction
✓Synthesize negative test fixtures and adversarial jailbreak prompts
✓Version-controlled dataset schemas with JSON and CSV export

02Scoring Engine

Multi-Dimensional Evaluation Scorers

Combine deterministic assertions (schema validity, latency budgets) with LLM-as-a-judge evaluators (helpfulness, accuracy, policy compliance).

✓Pre-built scorers for hallucination, tool argument correctness, and toxicity
✓Custom enterprise scoring rubrics with few-shot guidance
✓Cost-effective evaluation using high-speed frontier models

03CI Gates

Automated CI/CD Quality Gates

Integrate agent benchmarks into GitHub Actions, GitLab CI, and CircleCI. Prevent pull requests from merging if agent accuracy drops.

✓CLI tool `splyntra eval` returns standard exit codes for CI runners
✓Automated PR comment with rich visual diff of test results
✓Historical benchmark trends tracked across all branches and releases

Frequently Asked Questions

How does the Splyntra CI gate work?▼

The `splyntra eval run` CLI runs your agent against a specified golden dataset, executes the evaluation scorers, compares the score against your baseline branch or configured thresholds, and exits with code 0 (pass) or 1 (fail).

Can I run evaluations locally before committing code?▼

Yes. You can execute `splyntra eval run --local` using your local agent code and LLM API keys.

Related Platform Features & Guides

ObservabilityTraces & Replay →GovernanceCompliance Reports →SolutionsFor AI Engineers →

Ready to monitor and secure your AI agents?

Get started in under 3 minutes with zero credit card required. Free tier includes up to 5 projects and community telemetry.

Start Free on Cloud Request Custom Enterprise Demo