Timeouts and circuit breakers for LLM APIs

LLM API circuit breaker patterns exist because LLM APIs fail in ways that older synchronous APIs did not. Specifically, a request can hang for 90 seconds before returning, complete in 200 milliseconds when your cache is warm, or come back with tokens still streaming when the client already gave up. Consequently, using the same timeout defaults you use for a Postgres query will page your on-call engineer at 3am.

This page covers client-side timeouts and circuit breakers for the three languages Ciralgo customers use most: Python (with httpx), Node (with fetch and AbortController), and Go (with context.WithTimeout). Each section has runnable code, reasonable defaults, and the specific failure modes each pattern prevents.

The goal is not to make your application faster. Instead, the goal is to make it fail cleanly when the upstream is slow, and to recover automatically when the upstream recovers. Ciralgo runs these patterns on its own dependencies. Furthermore, we recommend you run them on the calls you make to Ciralgo.

Two failure modes an LLM API circuit breaker prevents

Before the code, understand the two failure modes.

The first is slow-response degradation. The provider is up but responding slowly. Requests still return, but latency stretches from 800 milliseconds to 20 seconds. Without a timeout, your request queue backs up, thread pool exhausts, and the application starts refusing new requests entirely. Meanwhile, downstream users get 502s from your load balancer even though the LLM API itself is technically fine.

The second is total outage. The provider is down and every request fails immediately with a connection error. Without a circuit breaker, your application still spends CPU on retries and error handling for every incoming request. As a result, you burn budget and latency on requests that had no chance of succeeding.

An LLM API circuit breaker addresses the second case. A timeout addresses the first. In practice, you want both, wired together.

Python: httpx timeouts and a circuit breaker with tenacity

The httpx library supports a per-request timeout tuple and a total timeout. For LLM calls, use both.

import httpx

client = httpx.Client(
    base_url="https://api.ciralgo.com/v1",
    timeout=httpx.Timeout(
        connect=5.0,   # TCP connect must succeed within 5s
        read=90.0,     # streaming responses can take a while
        write=10.0,    # sending the prompt is fast
        pool=5.0,      # waiting for a free connection
    ),
    headers={"Authorization": "Bearer YOUR_CIRALGO_KEY"},
)

For the circuit breaker, pybreaker gives you fail-fast behaviour once the failure rate crosses a threshold. Furthermore, the breaker automatically half-opens on a schedule to test whether the upstream has recovered.

from pybreaker import CircuitBreaker

llm_breaker = CircuitBreaker(
    fail_max=5,         # trip after 5 consecutive failures
    reset_timeout=30,   # try again after 30 seconds
)

@llm_breaker
def call_llm(prompt: str) -> str:
    response = client.post(
        "/chat/completions",
        json={
            "model": "gpt-4o",
            "messages": [{"role": "user", "content": prompt}],
        },
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]

Consequently, once the LLM API circuit breaker trips, subsequent calls fail immediately with a CircuitBreakerError for the next 30 seconds. In practice, your application handles that as a specific degradation path (cached response, "please try again later", or a different provider through Ciralgo's fallback policy).

Node: fetch, AbortController, and opossum

Node 18 and newer ship with a built-in fetch. For timeouts, wire an AbortController with a setTimeout.

async function callLlmWithTimeout(prompt, timeoutMs = 90000) {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), timeoutMs);

    try {
        const response = await fetch("https://api.ciralgo.com/v1/chat/completions", {
            method: "POST",
            headers: {
                "Content-Type": "application/json",
                "Authorization": "Bearer " + process.env.CIRALGO_KEY,
            },
            body: JSON.stringify({
                model: "gpt-4o",
                messages: [{ role: "user", content: prompt }],
            }),
            signal: controller.signal,
        });

        if (!response.ok) {
            throw new Error("LLM API returned " + response.status);
        }

        const json = await response.json();
        return json.choices[0].message.content;
    } finally {
        clearTimeout(timer);
    }
}

For the LLM API circuit breaker itself, opossum is the standard Node choice. Additionally, it fires events when the breaker opens, closes, or half-opens, which is useful for wiring into your observability stack.

import CircuitBreaker from "opossum";

const options = {
    timeout: 95000,             // fail requests slower than 95s
    errorThresholdPercentage: 50, // trip when 50% of recent requests fail
    resetTimeout: 30000,        // try half-open after 30s
    rollingCountTimeout: 10000, // measure over a 10s window
    rollingCountBuckets: 10,
};

const breaker = new CircuitBreaker(callLlmWithTimeout, options);

breaker.on("open", () => console.warn("LLM API circuit breaker OPEN"));
breaker.on("halfOpen", () => console.info("LLM API circuit breaker HALF-OPEN"));
breaker.on("close", () => console.info("LLM API circuit breaker CLOSED"));

const result = await breaker.fire("Summarise this in one line.");

Go: context.WithTimeout and sony/gobreaker

Go's canonical timeout mechanism is context. Every LLM call takes a context, and cancelling the context cancels the request.

package main

import (
    "context"
    "encoding/json"
    "net/http"
    "strings"
    "time"
)

func callLLM(ctx context.Context, prompt string) (string, error) {
    ctx, cancel := context.WithTimeout(ctx, 90*time.Second)
    defer cancel()

    body := strings.NewReader(`{"model":"gpt-4o","messages":[{"role":"user","content":"` + prompt + `"}]}`)
    req, err := http.NewRequestWithContext(ctx, "POST", "https://api.ciralgo.com/v1/chat/completions", body)
    if err != nil {
        return "", err
    }
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Authorization", "Bearer "+os.Getenv("CIRALGO_KEY"))

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return "", err
    }
    defer resp.Body.Close()

    var parsed struct {
        Choices []struct {
            Message struct{ Content string } `json:"message"`
        } `json:"choices"`
    }
    if err := json.NewDecoder(resp.Body).Decode(&parsed); err != nil {
        return "", err
    }
    return parsed.Choices[0].Message.Content, nil
}

For the LLM API circuit breaker, sony/gobreaker is the standard Go implementation.

import "github.com/sony/gobreaker"

var cb = gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:        "ciralgo-llm",
    MaxRequests: 3,
    Interval:    10 * time.Second,
    Timeout:     30 * time.Second,
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        return counts.ConsecutiveFailures > 5
    },
})

result, err := cb.Execute(func() (interface{}, error) {
    return callLLM(ctx, "Summarise this.")
})

Consequently, once the breaker trips, cb.Execute returns gobreaker.ErrOpenState immediately for the timeout window. In practice, your application code branches on that error as its degradation path.

Reasonable defaults for LLM API circuit breaker configuration

Every application has its own tolerance, but the following defaults are a sensible starting point.

• Connect timeout: 5 seconds. If the TCP handshake takes longer, the network is broken. Consequently, retrying will not help.
• Read timeout: 90 seconds. Long enough for large completions with reasoning, short enough that a genuinely stuck response gets abandoned.
• Total timeout: 95 seconds. Slightly longer than the read timeout so it does not fire before the read timeout does.
• Breaker fail threshold: 5 consecutive failures OR 50% failure rate over a 10-second window. Whichever comes first.
• Breaker reset window: 30 seconds. Long enough for a transient provider blip to clear, short enough to recover quickly from a real degradation.
• Half-open concurrency: 1 request. Prevents a stampede when the breaker starts probing recovery.

Furthermore, if you route through Ciralgo, the proxy itself applies a hard 120-second ceiling on all requests. So configuring your own client timeout above 120 seconds provides no benefit.

Common mistakes that page an on-call engineer

• No timeout at all. The default in most SDKs is either infinite or several minutes. Neither is safe for a production LLM call.
• Timeout shorter than the model's typical response time. Setting a 5-second timeout on a chain-of-thought reasoning model means every legitimate request fails. As a result, users see errors even though the system is working.
• Retrying on every error. Retrying on 429 (rate limit) and 5xx (transient) is correct. However, retrying on 4xx (client error) is not; the request will fail identically no matter how many times you retry.
• No jitter on retry backoff. Exponential backoff without jitter causes retry storms: every failing client retries at the same intervals, hammering the provider once it comes back up.
• Circuit breaker not shared across processes. In a multi-process web server (uWSGI, Gunicorn, Node cluster), a per-process breaker means each process has to fail independently before the breaker opens. Furthermore, the breaker's memory of failures does not survive a restart. For high-traffic systems, a shared external state store (Redis) matters.
• Breaker trips permanently in dev. Setting fail_max=1 in a development environment where the LLM endpoint sometimes takes ten seconds because you are on hotel Wi-Fi means the breaker opens on your first slow call and stays open. Consequently, developer productivity drops for no operational reason.

Frequently asked questions about LLM API circuit breaker patterns

Should I retry when the LLM API circuit breaker is open?

No. That defeats the purpose. When the breaker is open, the correct behaviour is to fail immediately with a graceful degradation. Meanwhile, retries during the open state should be handled by the breaker's own half-open probe logic, not by application code.

How does this interact with Ciralgo's own routing fallback?

Ciralgo's LLM provider routing policy fallback and your client-side LLM API circuit breaker are complementary. Specifically, Ciralgo fails over to a secondary provider before returning an error to your client. Consequently, your client-side breaker only trips if every provider in the chain fails. As a result, transient single-provider outages become invisible to your application.

What is a reasonable retry policy on top of a circuit breaker?

Retry 429 and 5xx up to three times with exponential backoff (1s, 2s, 4s) plus jitter (0-500ms). Do not retry 4xx. In practice, if the breaker is closed and half the retries fail, the breaker will open on its own.

Do I need a circuit breaker per model, or one for all LLM traffic?

One per upstream endpoint is typical. Specifically, if you talk to two Ciralgo endpoints (e.g. one for OpenAI-shaped traffic and one for Anthropic-shaped), use two breakers. In practice, this lets one degrade without disabling the other.

How do I test the LLM API circuit breaker path in staging?

The cleanest approach is to point the SDK at a mock endpoint that returns configurable errors on demand. Specifically, tools like httpbin.org or a small Express/Flask stub let you force 429s, 500s, and timeouts. As a result, you can verify your breaker configuration end-to-end.

What about streaming responses that legitimately take minutes?

Increase the read timeout to match your longest-legitimate stream duration, and keep the connect timeout tight. In addition, use a keepalive on the HTTP layer so intermediate load balancers do not silently kill the connection.

Related reading

• Connecting to the Ciralgo proxy
• Provider routing governance
• AI audit logging under the EU AI Act
• What is an EU-hosted LLM proxy?

Last refreshed: 9 July 2026. This page is reviewed quarterly.