Regression Firewall
Never ship a fix that breaks something else. Automated firewall enforcing zero regression gating before deployment.
Current baseline version
Baseline v1.0
Customer Support Agent · Baseline version (policy retrieval flaws, 5 failures)Promoted candidate version
Candidate B (Promoted candidate v1.1)
Recency-ranked retrieval + Tavily grounding + policy guardrailHoldout Regression Checks
20 STRICTLY ISOLATED CASES| Verification Dimension | Candidate B (Promoted) | Candidate A |
|---|---|---|
| Critical failures (0 allowed) | 0 regressions (PASS) | Warning |
| Overall reliability score (≥80% target) | 0 regressions (PASS) | Warning |
| Failure category recurrence | 0 regressions (PASS) | Warning |
| Holdout generalization performance (20 unseen cases) | 0 regressions (PASS) | 80% holdout (FAILED) |
| Previously fixed failure safety checks | 0 regressions (PASS) | 2 policy regressions (BLOCKED) |
Firewall Interceptions (2 Blocked Policy Regressions)
Candidate A attempted to overfit prompt guidelines, triggering 2 High-severity policy violations caught by the firewall.
REG-ORD-042 · Validation Split (POLICY · HIGH)BLOCKED
Order cancellation post-dispatch
Candidate A authorized immediate cancellation and refund while shipment was in freight carrier transit.
REG-POL-089 · Holdout Split (POLICY · HIGH)BLOCKED
Cross-border tariff refund claim
Candidate A promised full reimbursement of customs duties on international return against federal statutes.
CANDIDATE B: PASS — SAFE TO DEPLOY
Candidate B passed all holdout checks with 0 regressions detected. Certified safe for baseline promotion to v1.1.