Artificial Analysis Language Model API Performance Benchmarking Methodology
Overview
Measuring LLM performance requires sending the LLM a prompt and measuring the characteristics of its output. We measure performance across several test workloads.
Workload Types
| Workload type | Description |
|---|---|
| 1k input token | A single request of approximately 1,000 input tokens |
| 10k input token | A single request of approximately 10,000 input tokens |
| 100k input token | A single request of approximately 100,000 input tokens |
| Multi-turn agentic | An agentic task over multiple turns, from our coding, knowledge work and automation evaluations |
Load
We measure every workload under concurrent load, to better represent how endpoints perform in production.
Testing Frequency
- We run the benchmark every 6 hours across a range of workloads
- We measure endpoints of deprecated models every 18 hours
Prompt Generation
Each run draws its prompts and tasks from a large, diverse set and sends the same ones to every endpoint in that run, so every endpoint is compared on the same work.
Single-request prompts combine long-form input content with a range of tasks, sized to fill each workload's target input length.
Prompt diversity matters for performance benchmarking because techniques like speculative decoding mean that output speeds vary with the type of output.
Multi-turn workloads use agentic tasks from our coding, knowledge work and automation evaluations, drawn at random in each run, so the task varies from run to run.
Measurement Representation
For each run, we take the median of each workload's samples: its requests, or its session for a multi-turn workload. An endpoint's value for a workload is the average of its run medians over the past 5 days.
An endpoint's headline output speed, time to first answer token and time to first token are the average of its workloads. Each workload counts equally. Where we have no measurement of a workload for an endpoint, such as a prompt longer than its context window or a tool-calling workload on an endpoint without function calling, the average covers the workloads we measured.
A model's speed and latency figures come from its representative endpoint, usually the model creator's own API, which is also the endpoint we use for its price. When a model has no representative endpoint, or we have no measurements of it in the past 5 days, we use the median across the model's measured endpoints.
Key Definitions
- Time to First Answer Token: The time in seconds between sending a request to the service or system and receiving the first answer token of the response. For reasoning models, this includes the time spent reasoning, so unlike time to first token, it measures the wait for an answer the same way for reasoning and non-reasoning models. For multi-turn workloads, we average it over a session's turns.
- Time to First Token: The time in seconds between sending a request to the service or system and receiving the first token of the response. For reasoning models which return reasoning tokens, this will be the first reasoning token. For multi-turn workloads, we average it over a session's turns.
- Output Speed (output tokens per second): The number of output tokens, including reasoning tokens, received per second over a request's full duration, from sending the request to receiving the final token, so it includes time to first token. For a multi-turn session, we divide its output tokens by the time the endpoint took to answer its requests, which excludes the time tools take to run between them.
- End-to-End Time: The time in seconds between sending a request and receiving the final token, measured on each request. For a multi-turn session, it is the total over the session's requests, which excludes the time tools take to run between them.
- Endpoint Error Rate: The share of an endpoint's requests in the past 5 days that failed on the endpoint's side. A multi-turn session counts as one request, which fails if any of its requests fails. We exclude failures we don't attribute to the endpoint, such as an invalid API key, exhausted credits or our account's quota, a request rejected as invalid or too long for the context window, a response our max_tokens limit cuts off before it answers, a safety block or a sandbox error. Errors on the endpoint's side count, including its rate limits, timeouts and requests still running at the benchmark deadline. Across workloads, we pool requests before dividing.
- Cache Hit Rate: The share of cacheable input tokens served from the provider's prompt cache, as reported by the provider's API during multi-turn agentic sessions. We report a representative cache hit rate for each endpoint based on its recent measurements.
Technical Details
Server Location: We run the benchmark on Google Cloud Run in the us-central1 region.
Test Accounts: We run tests using a combination of anonymous accounts, accounts with credits, and API keys provided explicitly for benchmarking. Where our primary benchmarking does not use an anonymous account, we register a separate anonymous account and validate that performance is not being manipulated.
API Libraries: For all providers claiming compatibility with OpenAI's API, we use the official OpenAI Python library to ensure consistency across tests. For providers without OpenAI compatibility, we use their recommended client libraries.
Cache Hit Rate Measurement: Cache hit rate is the share of cacheable input tokens read from the cache. We measure it from our multi-turn workloads.
API Parameters: We use the following API parameters:
temperature: 0.6for reasoning models andtemperature: 0for non-reasoning models, unless specified otherwise by the model creator.top_p: 1for single-request workloads, unless specified otherwise by the model creatormax_tokensof up to the endpoint's output token limit, capped at 65,536, or 128,000 for endpoints that often reason longer
Token Measurement: We count output tokens as reported by each provider's API where available, including reasoning tokens whether or not the API returns their text. When a provider reports none, we count them from the response text. The Artificial Analysis Intelligence Index evaluations also use the token counts reported by each model's API provider (including cached input, reasoning, and output tokens), which let us more accurately report the cost of running the Intelligence Index. When reporting cache hit rates and cost, we combine these with live measurements of each model's typical cache hit rate, rather than relying on the one-off measurement when the evaluation was run.
Failure Handling: A request that fails on the endpoint's side, such as a server error, a rate limit (429) or a timeout, counts as an output speed of 0. A multi-turn session that fails or does not finish within its scheduled run counts only the requests the endpoint answered, or 0 if it answered none. We exclude failures on our side, such as an invalid API key, exhausted credits or our account's quota, or a sandbox error.
Known Limitations
Quantization: Some models use quantization to reduce computational requirements and increase speed. However, quantization can also affect model quality. We are moving towards full disclosure of quantization methods used by the models we benchmark. To measure how serving choices such as quantization affect quality, we also run the Endpoint Accuracy Index on a select set of the most relevant open weights models served by many providers.
Server Location and Latency: Time to first answer token and time to first token include network latency, so they are sensitive to server location. We run our benchmarks in Google Cloud's us-central1 region, which may advantage or disadvantage certain providers based on their server locations. We are considering additional testing locations to mitigate this effect.
Version History
Version 3.0.0
3 November 2026
- Workloads: We added multi-turn agentic workloads with tool calls, from our coding and knowledge work evaluations, alongside the single-request workloads, and measure them all under the same sustained load. These replace the vision workload and the single and parallel prompt load scenarios.
- Headline output speed: Our headline output speed now averages the workloads we measured over the past 5 days (previously the median of the 10k input token workload over the past 72 hours).
- End-to-end output speed: Output speed now covers each request's full duration, including time to first token, and excludes the time tools take to run in multi-turn sessions.
- Latency: We now measure time to first answer token and end-to-end time on each request (previously estimated from average reasoning tokens and output speed).
- Token counting: We count output tokens as reported by each provider's API (previously tiktoken o200k_base).
- Endpoint error rate: We now report the share of each endpoint's requests that fail on the endpoint's side.
- Failures: Failures on the endpoint's side now count as an output speed of 0, and failures on our side are excluded.
- Frequency: We run the benchmark every 6 hours (previously every 3 hours).
- Prompt Options: We removed the Prompt Options pages. Per-workload charts show each workload's results.
Version 2.2.0
2 March 2026
- Updated prompts: We introduced a new set of prompts with broader, more varied content and a wider range of task types. Prompt diversity matters because techniques like speculative decoding mean that output speeds vary with the type of output.
- Default workload change: Default speed on our site now reflects results from 10k input token prompts (previously 1k) and we have deprecated performance measurements for 100 token input workloads. All 1k, 10k and 100k input token workloads are still visible on all pages by selecting the Prompt Options drop-down. The easiest way to compare output speeds for different workload shapes is using the Output Speed by Input Token Count chart.
Integrity Terms
Background
Artificial Analysis is an independent AI benchmarking and insights provider. Our benchmarks are consulted by millions of users and organizations.
We strive to ensure that all data published on Artificial Analysis is fair, transparent, and representative of the typical experience of developers and organizations using the models and endpoints we cover. Our inference benchmarking data is widely used as a reference standard for comparing inference providers across dimensions such as speed, latency, and quality. The value of these measurements depends on them reflecting the experience of an ordinary developer using a provider's standard, publicly available endpoint.
These terms formalize the approach Artificial Analysis has taken since we began benchmarking inference performance. They apply equally to all inference providers and platforms (together, "providers") that publish serverless endpoint data on Artificial Analysis, and they ensure every provider is measured on the same fair basis.
Provider Requirements
Traffic from Artificial Analysis must be served on the same publicly available configuration that any ordinary developer would receive when using the endpoint. In particular, providers must not:
- Detect, fingerprint, or otherwise identify Artificial Analysis traffic (whether by account, API key, IP address, request headers, payloads, or traffic patterns) and serve it differently from ordinary traffic.
- Route Artificial Analysis traffic to dedicated, reserved, or non-public resources, including separate hardware, capacity pools, priority queues, or geographic regions that are not generally available.
- Serve Artificial Analysis a model, quantization, context length, or endpoint configuration that differs from what is publicly advertised and generally available under the same name.
- Serve Artificial Analysis traffic at a batch size, concurrency, or load configuration that is not representative of what ordinary traffic on the same endpoint receives (for example, running benchmark requests at reduced batch sizes to increase per-request speed).
On request, providers must confirm the model version, precision or quantization, and context length serving Artificial Analysis traffic, and confirm that this matches the standard public offering. Providers must ensure that every employee, contractor, advisor, or affiliate who is aware that an endpoint is being measured by Artificial Analysis complies with this policy.
Performance features that are part of the standard public offering and available to all developers on equivalent terms (for example, autoscaling, generally available caching, or published tiering that any customer can access) are not a violation, provided they are not applied selectively to Artificial Analysis traffic.
Compliance Process
Artificial Analysis lists endpoints and providers at its sole discretion. We reserve the right to delist any endpoint or provider that does not comply with these terms, and to decline, withhold, or remove any listing for any reason.
We conduct frequent consistency checks on every endpoint we measure. As standard practice, we measure from our primary Artificial Analysis account alongside independent test accounts, each created and used in the same way an ordinary developer would. Where these return materially different results on identical payloads and the same endpoint, we investigate before drawing conclusions.
If we detect or reasonably suspect that Artificial Analysis traffic is being served differently from the standard public offering, we may:
- Withhold, remove, or delay publication of affected measurements.
- Re-measure on a standard, publicly created account and publish those results.
- Remove the endpoint or provider from performance leaderboards and listings.
- Suspend the provider across all leaderboards.
- Publicly disclose that a provider has served non-representative performance to Artificial Analysis.
By submitting endpoints to Artificial Analysis, providers acknowledge these terms and our shared commitment to maintaining benchmark integrity. Artificial Analysis may update this policy at any time.
We appreciate your cooperation. If you have questions or concerns, please contact our team.