What broke when I upgraded my agent from Sonnet 5 to Sonnet 5.5

Lukas Babaliauskas published an engineering post and companion repository detailing why swapping foundation models in production agents triggers silent failures, along with a deterministic framework to catch them before deployment. Upgrading an agent from Sonnet 5 to Sonnet 5.5 without updating assertions caused silent behavioral drift across tool arguments, enum classifications, and refusal paths, as detailed in evalshift.dev's report.

This analysis covers testing methodologies for software engineers building tool-calling agents and structured JSON pipelines. It breaks down the six silent failure modes encountered across thirty recorded production turns, explains how to construct stratified golden datasets from server logs, and outlines deterministic continuous integration checks that prevent customer-facing regressions when changing model strings.

01Six silent failures across thirty turns

Swapping the underlying model from Sonnet 5 to Sonnet 5.5 across a three-route agent broke six out of thirty replayed production turns without raising runtime exceptions. The evaluation examined three distinct workloads: a day planner invoking calendar tools, a unit converter handling measurement requests, and a support triage classifier outputting structured JSON. Every failure produced syntactically plausible output that would pass manual playground spot-checks or lenient schema parsers while corrupting downstream business logic.

The most severe regression occurred when an impossible request to convert ten kilometres into kilograms triggered a hallucinated tool call to convert_units under Sonnet 5.5, whereas Sonnet 5 had correctly explained that distance cannot convert to mass. In calendar routing, Sonnet 5.5 violated prompt instructions requiring resolved ISO calendar strings by emitting relative date arguments like get_schedule with date set to today instead of resolving the pinned Tuesday date. Multiple-intent queries also generated unprompted secondary tool calls, adding get_tasks on top of search_notes.

Structured classification suffered enum value drift where a double-charge incident dropped from urgent to high priority while remaining entirely valid against the schema. Another ticket covering logouts and late refunds shifted queues from billing to bug, altering team ownership without invalidating output structure. Markdown code fence leakage stopped entirely on Sonnet 5.5 after Sonnet 5 wrapped eight of nine JSON outputs in backticks, while suggested replies expanded eight percent in word count with specific browser troubleshooting instructions.

02Latency and cost shifts across agent routes

Evaluating latency and token cost using global averages hid route-level performance swings that impacted system resources differently. Unit conversion workflows saw response latency spike by 47 percent under Sonnet 5.5, creating unexpected bottlenecks for synchronous calculations. Conversely, the ticket triage classifier accelerated by 32 percent while cutting inference costs by 27 percent per ticket, while the calendar planner exhibited flat performance.

These divergent metrics prove that aggregate pipeline benchmarking conceals localized degradations in complex agent configurations. Averaging latency across all three routes created the false impression of net-neutral performance, obscuring the fact that conversational routes grew more verbose while specific functional tool paths suffered severe execution delays.

03Golden set generation from production logs

Building a dependable regression suite requires sampling historical production turns directly into a golden dataset rather than relying on synthetic prompts conceived in developer playgrounds. The evalshift/llm-regression-example repository structures each turn as a single JSON line capturing the unique request ID, timestamp, route identifier, conversation history, raw input string, tool calls with executed arguments, and response latency. A system prompt with a frozen clock pinning the date to Tuesday, 2026-09-08 ensures deterministic resolution of relative temporal expressions during offline replays.

Production logs must be stratified across operational routes to ensure low-volume endpoints receive proportional test coverage. A regression affecting a route representing five percent of total volume remains invisible in an unstratified draw. The sampling script filters duplicate inputs such as repeated order status inquiries to prevent skewed pass rates, cleans private personal data before saving to disk, and protects tagged edge cases from being dropped during downsampling to between 100 and 300 total lines.

Expectation functions extract the baseline tool calls, arguments, and classification enums executed in production to establish an immutable baseline file named golden.jsonl. Developers must inspect this artifact before committing it to version control, treating production behavior as a fixed comparator rather than an absolute oracle. Modifying test cases while simultaneously benchmarking candidates invalidates the comparative differential.

04Deterministic assertions over semantic LLM judges

Replacing probabilistic model-based judges with deterministic code checks catches subtle behavioral shifts in microseconds without incurring inference charges. During evaluation, an LLM judge asked to score ticket responses favored Sonnet 5.5 on eight of nine runs because of polished phrasing, completely ignoring that the model downgraded a critical double-charge ticket to a lower priority. Embedding similarity algorithms also scored the priority regression at 0.95 and the wrong queue assignment at 0.87, passing flawed outputs.

Deterministic validation separates baseline checks from strict contract rules. Contract checks verify structural rules that must hold universally, such as enforcing ISO date formats using regular expressions like ^\d{4}-\d{2}-\d{2}$, restricting currency symbols to ISO codes, and capping summary lengths to 20 words. Baseline checks perform strict field-equality assertions between the candidate output and the frozen golden record, detecting when classification enums migrate between allowed options.

Tool calling assertions compare the exact set of invoked tool names against production baselines, immediately flagging unprompted secondary calls or unexpected conversions on refused prompts. Strict JSON parsing checks run directly against raw text outputs before application-level sanitizers strip markdown fences, ensuring model upgrades do not reintroduce syntax quirks that break unhardened downstream consumers.

05Model selection commands and pinned deployment

claude update

/model claude-3-5-sonnet-20241022

claude --model claude-3-5-sonnet-20241022

Engineering teams can pin models locally or update CLI harnesses using standard package tools before executing regression diffs against golden datasets. Running the candidate model against recorded turns executes entirely offline when replaying cached outputs from the repository, requiring no active API credentials to surface argument drift or enum migration.

Integrating the resulting pass-rate diff directly into continuous integration pipelines turns foundation model upgrades into automated pull request reviews rather than manual testing drills. Flagging schema-valid regressions, tool over-invocation, and route-specific latency anomalies prior to production deployment eliminates post-release triage and prevents degraded user experiences across autonomous agent workflows.