Sending requests at scale

Best practices for submitting thousands of concurrent requests to the Sail API

For large workloads, submit a batch or send concurrent Responses API requests in background mode.

Batch API

Submit inference requests in a batch, then poll GET /v1/batches/{batch_id} until status is completed. The final response includes a results array, with each result matched to your input by custom_id.

Use an idempotency key when submitting so retries don’t create duplicate batches. To list your batches, use GET /v1/batches.

Batch requests always use the flex completion window. For low-latency workloads, use the Responses API instead.

Python example

First, install the requests library:

pip install requests

import time
import requests

BASE_URL = "https://api.sailresearch.com/v1"
API_KEY = "YOUR_KEY_HERE"
HEADERS = {"Authorization": f"Bearer {API_KEY}"}

# 1. Submit a batch
batch_requests = [
    {
        "custom_id": f"request-{i}",
        "params": {
            "model": "zai-org/GLM-5.3",
            "max_output_tokens": 2048,
            "input": [{"role": "user", "content": f"What is a fun fact about the number {i}?"}],
        },
    }
    for i in range(100)
]

resp = requests.post(
    f"{BASE_URL}/batches",
    headers=HEADERS,
    json={
        "endpoint": "/v1/responses",
        "label": "my-batch-job",
        "requests": batch_requests,
    },
)
resp.raise_for_status()
batch = resp.json()
batch_id = batch["id"]
print(f"Created batch {batch_id} with {batch['request_counts']} requests")

# 2. Poll until the batch finishes
while True:
    status_resp = requests.get(f"{BASE_URL}/batches/{batch_id}", headers=HEADERS)
    status_resp.raise_for_status()
    status = status_resp.json()
    counts = status["request_counts"]
    done = counts["completed"] + counts["failed"]
    print(f"Progress: {done}/{counts['total']}")

    if status["status"] == "completed":
        break
    time.sleep(5)

# 3. Read the results returned by the final poll
if status["results"] is None:
    # Fetch large batch results individually, as shown below.
    print(status["error"]["message"])
else:
    for item in status["results"]:
        custom_id = item["custom_id"]
        if item["error"] is not None:
            print(f"{custom_id}: {item['error']}")
            continue

        result = item["response"]["body"]
        output_text = "".join(
            c["text"]
            for output in result.get("output", [])
            for c in output.get("content", [])
            if c.get("type") == "output_text"
        )
        print(f"{custom_id}: {output_text[:100]}")

Reading the batch response

• status is in_progress until every request finishes, then completed, even if some requests failed or were cancelled.
• request_counts shows how many requests were submitted, completed, or failed.
• request_status lists each request’s status by custom_id. This list may be incomplete while the batch is running. Stop polling when the batch’s top-level status is completed.
• Each result has either an error or a response.body containing a Responses API object. Check the body’s status for incomplete output, and match results by custom_id.

See the endpoint reference for the full response shape.

Large batches and individual results

• Submit up to 100,000 requests per batch, with a maximum request body of 256 MiB.
• Inline results are limited to approximately 1 GiB per batch. If the results are too large, the response tells you to retrieve them using the individual batch results endpoint (see the reference code below).
• Inline results are one JSON response. If the connection ends before it finishes, retry the GET.

You can also fetch individual results as soon as they finish:

for custom_id, info in status["request_status"].items():
    if info["status"] != "COMPLETED":
        print(f"{custom_id}: {info['status']}")
        continue

    result_resp = requests.get(
        f"{BASE_URL}/batches/{batch_id}/{custom_id}", headers=HEADERS
    )
    result_resp.raise_for_status()
    result = result_resp.json()
    # Save or process this Responses API object before fetching the next one.
    print(custom_id, result["output"])

Responses API with background mode

You can also submit requests individually using the Responses API with background=True and setting the completion window to balanced or flex. For large-volume workloads (1,000+ requests), we recommend:

1. Use AsyncOpenAI with DefaultAioHttpClient(). The OpenAI SDK’s built-in aiohttp client is more efficient than the default httpx backend for high-concurrency workloads.
2. Limit concurrency with an asyncio.Semaphore. The semaphore caps how many connections your client has open at once (such as 200), so the machine sending the requests does not run out of connections.
3. Submit all requests concurrently with background=True, then poll. Send all submissions in parallel (limited by the semaphore), collect the response IDs, and poll for completions in a separate loop.
4. Use one idempotency key per submission. Reuse it if you need to retry that submission.

Python example

First, install tqdm and the OpenAI SDK with the aiohttp extra:

pip install tqdm
pip install 'openai[aiohttp]'

import argparse
import asyncio
import uuid

from tqdm import tqdm
from openai import DefaultAioHttpClient
from openai import AsyncOpenAI

async def main(
    num_requests: int,
    input_tokens: int,
    max_output_tokens: int,
    model: str,
):
    base_url = "https://api.sailresearch.com/v1"
    api_key = "YOUR_KEY_HERE"

    async with AsyncOpenAI(
        base_url=base_url,
        api_key=api_key,
        http_client=DefaultAioHttpClient(),
    ) as client:
        # List supported models
        models = await client.models.list()
        supported_models = [m.id for m in models.data]
        print("Supported Models:")
        print(supported_models)

        # Submit requests concurrently
        sem = asyncio.Semaphore(200)
        pbar_submit = tqdm(total=num_requests, desc="Submitting requests")

        submission_keys = [str(uuid.uuid4()) for _ in range(num_requests)]

        async def submit(i):
            filler = ""
            if input_tokens > 0:
                filler = " " + ("word " * input_tokens)
            content = f"TASK {i}: What is a fun fact about the number {i}? Then, find the word at index {i} in the following sequence of words: {filler}"

            async with sem:
                response = await client.responses.create(
                    model=model,
                    input=[{"role": "user", "content": content}],
                    max_output_tokens=max_output_tokens,
                    background=True,
                    metadata={"completion_window": "balanced"},
                    extra_headers={"Idempotency-Key": submission_keys[i]},
                )
            pbar_submit.update(1)
            return response.id

        response_ids = await asyncio.gather(*[submit(i) for i in range(num_requests)])
        pbar_submit.close()
        response_ids = list(response_ids)
        print(f"Created {len(response_ids)} response IDs")

        # Poll for completions
        usable = {}
        poll_sem = asyncio.Semaphore(200)
        pbar = tqdm(total=len(response_ids), desc="Polling responses")

        async def poll(response_id):
            for _ in range(3600):  # 1 hour timeout
                async with poll_sem:
                    response = await client.responses.retrieve(response_id)
                if response.status == "completed":
                    usable[response_id] = response
                    pbar.update(1)
                    return
                if response.status == "incomplete":
                    reason = getattr(response.incomplete_details, "reason", None)
                    if reason == "max_output_tokens":
                        usable[response_id] = response
                    else:
                        print(f"{response_id} is incomplete because of {reason or 'an unknown reason'}")
                    pbar.update(1)
                    return
                if response.status in {"failed", "cancelled"}:
                    print(f"{response_id} ended with status {response.status}")
                    pbar.update(1)
                    return
                await asyncio.sleep(1)
            print(f"Timed out waiting for {response_id}")

        await asyncio.gather(*[poll(rid) for rid in response_ids])
        pbar.close()

        for response_id in response_ids:
            resp = usable.get(response_id)
            if resp is None:
                continue
            output_text = "".join(
                c.text
                for item in resp.output
                for c in getattr(item, "content", []) or []
                if getattr(c, "type", None) == "output_text"
            )
            print(f"\n\n{response_id} {resp.status} with output:\n{output_text}")


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--num-requests", type=int, default=20000)
    parser.add_argument("--input-tokens", type=int, default=50)
    parser.add_argument("--max-output-tokens", type=int, default=4000)
    parser.add_argument("--model", type=str, default="zai-org/GLM-5.3")
    args = parser.parse_args()

    asyncio.run(
        main(
            num_requests=args.num_requests,
            input_tokens=args.input_tokens,
            max_output_tokens=args.max_output_tokens,
            model=args.model,
        )
    )

The semaphore limit of 200 is a good starting point. Lower it if you run into connection errors or timeouts. Raise it if your machine can hold more open connections and you want faster submission throughput.