Home / Blog / How Airbnb uses cost per task to operate coding agents on…
Tech News

How Airbnb uses cost per task to operate coding agents on Amazon Bedrock

Airbnb runs coding agents on Amazon Bedrock using cost per task, where shifting to high reasoning effort raised scores by 0.8 but cost 2.5 times as much.

By Dillip Chowdary • Nov 19, 2026 • Source: AWS Machine Learning Blog

How Airbnb uses cost per task to operate coding agents on Amazon Bedrock
Image: AWS Machine Learning Blog

At a glance

Platform tool
AirChat internal coding agent
Gateway platform
Amazon Bedrock on AWS
Evaluation suites
Seven engineering task suites
Adoption metric
Two thirds of requests on new models by mid-October
Effort cost impact
High effort increased task cost by 2.5 times
Effort score impact
High effort raised score by 0.8 points

Airbnb deployed an operational framework combining pre-rollout task evaluations and Amazon Bedrock CloudWatch telemetry to manage production costs, latency, and quotas for internal coding agents. According to AWS Machine Learning Blog's report, thousands of Airbnb engineers run coding agents weekly through an internal tool called AirChat. When eight models launched across five weeks, Airbnb used task-based pricing and runtime observability to steer two thirds of its agent traffic to the newer models.

This breakdown details how engineering organizations evaluate foundation models against real repository tasks, manage token quotas, and enforce reasoning effort limits on Amazon Bedrock. Platform leaders and infrastructure engineers will learn how to monitor CloudWatch runtime signals, calculate total token expenses per finished assignment, and reconcile offline benchmarks with live developer usage.

01Runtime signals in the Bedrock namespace

Diagnosing production agent behavior relies on core CloudWatch operational metrics collected from the AWS/Bedrock namespace. Infrastructure teams track InvocationLatency and TimeToFirstToken to catch responsive slowdowns, while monitoring OutputTokenCount and derived output tokens per second to inspect completion volume. These runtime metrics reveal real-time capacity and latency shifts whenever model releases alter how agents process code.

Throttling investigations leverage InvocationThrottles alongside request token reservations and the EstimatedTPMQuotaUsage metric across all Bedrock inference APIs. This quota metric tracks tokens per minute consumption, accounting for cache write tokens and output burndown multipliers across Converse, InvokeModel, ConverseStream, and InvokeModelWithResponseStream calls. Engineering teams use these measurements to set proactive alarms and request higher quota allocations before developer traffic experiences rate limiting.

Financial monitoring tracks four distinct token types that carry independent pricing rates across model providers. Bedrock reports uncached input, cache read input, cache write input, and generated output tokens for each invocation. Because cache reads cost less than uncached inputs while cache writes can carry premiums depending on the model, low cache hit rates directly increase expenses per API call.

02Dual workspaces and the proxy gateway

Airbnb developers run the AirChat agent directly on AirDev personal cloud development workspaces during daily programming. For asynchronous execution, engineers dispatch assignments to AirChat Remote, which coordinates autonomous cloud agents inside separate workspaces while the developer continues other tasks. Both client pathways direct model traffic through a centralized internal large language model gateway that connects directly to Amazon Bedrock.

The centralized gateway captures the model, client, team, and user identity for every agent request. Operational teams aggregate these records strictly for capacity planning and financial attribution across departments, matching logged caller data against CloudWatch error rates and latency numbers. Comparing live telemetry against offline task scores highlights discrepancies between expected developer workflows and actual production execution.

03Logging controls and tagged role prerequisites

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

Deploying Bedrock agents requires establishing specific AWS platform prerequisites before receiving developer traffic. Streaming calls through ConverseStream or InvokeModelWithResponseStream are necessary because Bedrock only publishes TimeToFirstToken metrics on streaming interfaces. Organizations must also activate AWS cost allocation tags on caller IAM roles before traffic begins, as Bedrock cannot backfill cost tags on historical consumption.

Note: Model invocation logging is disabled by default across AWS regions and records entire prompts, responses, and caller identities. Organizations must restrict read permissions, configure data retention periods, and isolate developer runtime sandboxes to protect corporate codebases from unauthorized exposure.

Internal testing setups require real engineering tasks built from historical code modifications, pull request reviews, and implicit ticket decisions. Each test task pairs an initial repository snapshot with an engineer prompt and an automated evaluation program. Teams configure Amazon Bedrock Guardrails to filter sensitive content and apply contextual grounding checks to flag unsupported model statements.

04Effort ceilings and full task pricing

Evaluating coding agents requires measuring list-price token costs across an entire finished assignment rather than pricing isolated requests. Because an agent makes multiple sequential calls that repeatedly process accumulating context, total expenditure aggregates uncached input, cache reads, cache writes, and output across all turns. Evaluation runs average these expenses across all attempts, including runs where the agent failed the underlying task.

Model reasoning effort adjustments introduce substantial performance and economic trade-offs for production deployments. In one evaluation run, increasing reasoning effort from low to high improved the task score by 0.8 points while increasing cost per task by 2.5 times. Platform administrators use these empirical ratios to establish default reasoning configurations and programmatically enforce maximum effort ceilings at the gateway.

05Seven task suites and rapid bakeoffs

Whenever foundation model providers release new architectures, automated evaluation campaigns process the models across seven dedicated task suites. A single request initiates hundreds of evaluation runs across distributed cloud workspaces, allowing graders to score assignments asynchronously as executions finish. In Airbnb tests, initial evaluations returned within hours, delivering complete cost and performance scores for organizational deployment decisions by the following morning.

Rapid evaluation depends on structured task directories containing all dependencies and execution parameters. Each test folder includes an instruction file detailing developer requirements, a task configuration specifying processor limits and memory, and a Dockerfile defining the isolated sandbox environment. Test scripts execute grader programs that recalculate correct solutions from frozen baseline data and assign quantitative rewards to candidate models.

In an excerpted machine learning bakeoff task named honest-holdout-bakeoff, candidate models evaluate two prediction systems to decide which version warrants production deployment. The agent analyzes frozen holdout metrics to determine whether either candidate justifies replacing the live system, producing an objective recommendation. Reference solutions ensure that graders accurately verify model reasoning against established engineering criteria before code changes reach developers.

Frequently asked questions

What is cost per task for coding agents?

Cost per task is the list-price cost of all tokens an agent consumes to complete a single engineering assignment. It sums uncached input, cache reads, cache writes, and output tokens across every sequential request required to finish the task.

How does reasoning effort impact model score and expense?

Raising reasoning effort from low to high increased the eval score by 0.8 points on a tested model. However, that score gain caused cost per task to increase by 2.5 times, prompting the use of gateway-enforced effort ceilings.

Which Amazon Bedrock metrics track quota consumption?

Bedrock publishes the EstimatedTPMQuotaUsage metric across all inference APIs to measure tokens per minute consumption, including cache write tokens and output burndown multipliers. Teams also track InvocationThrottles and custom request token reservations to alert before hitting limits.

Why are streaming APIs required for latency monitoring on Bedrock?

Amazon Bedrock only reports the TimeToFirstToken metric when using streaming interfaces like ConverseStream or InvokeModelWithResponseStream. Standard non-streaming calls report total invocation latency but do not isolate the initial token generation delay.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →