Service tier comparison
Check Pricing for current per-model availability.
Service tier details
Default (ASAP)
Default, aka ASAP, is the standard low-latency tier for Sail’s core models. It is characterized by:- No queueing delay - TTFT is dependent on prompt size and model.
- High TPS - targeting 50-100 TPS. Your experienced TPS will vary by workload and model.
- Foreground-only - this tier is not compatible with background mode or batch requests.
Flex
Flex is characterized by:- A flexible start - the first turn in a run is guaranteed to start within 1 hour.
- Much faster subsequent turns - follow-up turns of an in-flight run see minimal (seconds) queueing delay.
- Low TPS - targeting 10-20 TPS. Your experienced TPS will vary by workload and model.
- Support for extreme scale by design - Flex offers the most generous capacity and highest rate limits.
- Background-only - Flex is not compatible with streaming or foreground APIs.
Sail uses longest-prefix matching to automatically recognize subsequent turns.
Balanced
Balanced is the middle ground between Flex and Default, characterized by:- A flexible start with a tighter bound than Flex - the first turn in a run should start within 5 minutes.
- Much faster - like Flex, follow-up turns of an in-flight run see minimal (seconds) queueing delay.
- Medium TPS - targeting 30-50 TPS. Your experienced TPS will vary by workload and model.
- Streaming, foreground, and background-compatible.
Multi-turn agents and rollouts with Balanced and Flex
Balanced and Flex are designed to support multi-turn agents and rollouts. For
the first turn in a run, Balanced and Flex have a “warm-up” queueing delay of
up to 5 minutes (for Balanced) and 1 hour (for Flex), respectively.
Afterwards, Sail automatically uses longest-prefix matching to recognize
subsequent turns of an in-flight run. These follow-up turns are prioritized
and incur minimal to no queueing time.
How to use service tiers
Via the service_tier field
Set service_tier to asap, balanced, or flex. Reserved TPM
reservations use the same field and values.
background=True or submit requests through the Batch API.
Via request header
Alternatively, if your client cannot setservice_tier in the request body,
you can set the service tier using the X-Sail-Completion-Window request
header.
Default behavior
For inference requests for core models, the default behavior (when no service tier is specified) is low-latency inference (asap).
The exceptions to this default behavior are:
- Batch inference requests, which default to Flex and do not support other service tiers.
- LoRA requests, which are Flex-only.
- Requests that use custom tools, which default to Balanced for core models because ASAP does not support them.
- Responses API requests with
background=Trueuse Balanced or Flex by default, since they do not support ASAP.
Troubleshooting
ASAP requests
- ASAP does not support custom tools (
toolswithtype="custom") on the Responses or Chat Completions APIs. Use Balanced, or omit the service tier to default to Balanced for core models. - ASAP does not support Responses requests with
background=Trueor the Batch API. Use Balanced or Flex for Responses requests withbackground=True. - Setting
service_tier="asap"with custom tools orbackground=Truereturns HTTP400with error codeunsupported_asap_request.
Balanced requests
- You can send Balanced requests synchronously or as Responses requests with
background=True. - A synchronous streaming Balanced request opens its stream right away and keeps the connection alive while it waits. Output starts when the model begins generating.
Flex requests
- Chat Completions and Messages reject Flex with HTTP
400, with or without streaming. Use ASAP or Balanced on those endpoints. - Responses requests using Flex reject an omitted
backgroundvalue,background=None, orbackground=Falsewith HTTP400. Setbackground=Trueexplicitly.
Request settings
- If you set both
X-Sail-Completion-Windowandservice_tier, the value in the request body takes precedence. - Type checkers can reject Sail’s tiers in
service_tier, because the OpenAI and Anthropic SDKs type the field with their own values. The SDKs still send the value as written. When your type checker rejects it, silence only that line: add// @ts-expect-erroron the line aboveservice_tierin TypeScript, or end the line with# type: ignorein Python. OpenAI’s SDKs already allowflex, so aflexrequest there needs neither. - The model must support the tier you choose. Check Pricing for availability.
Batch and Background
- Responses requests with
background=Truecannot usestream=True. Retrieve results by polling or using webhooks. - A polling timeout does not cancel a Responses request with
background=True. Keep polling the same response ID. See Retrying requests if you lose the connection before receiving an ID. - The Batch API does not use
X-Sail-Completion-Window. - Batch always uses Flex and runs in the background. Setting a subrequest’s
service tier to
asaporbalancedreturns HTTP400. - Batch rejects models that do not support Flex.
Capacity errors
-
For ASAP requests, Sail may wait briefly for capacity. If none becomes
available, it returns
503with codemodel_capacity_unavailablewithout queueing the request. -
If the service is overloaded, Sail can return
503with codeservice_overloadedbefore accepting a request in any service tier. -
If the response includes
Retry-After, wait at least that many seconds. See Retrying requests before resubmitting after a timeout or another server error, since the request may already have started.