Building Resilient LLM API Clients in Go: Circuit Breakers, Retries, and Rate Limiting

Most LLM integrations still look like this: a raw HTTP call wrapped in an if err != nil, maybe a single retry, and a prayer that the provider stays up. That's fine for a prototype. It falls apart in production, because LLM providers fail differently, and more often, than the APIs most backend engineers are used to depending on.

This article covers the resilience patterns that actually matter for LLM API calls, implemented in Go: timeouts that match real inference latency, retries with backoff and jitter, circuit breakers tuned for LLM failure modes, and rate limiting that respects provider quotas. If you've read the earlier article on building microservices with Go, this is the same resilience philosophy, applied to a dependency that fails in unusually specific ways.

Why LLM APIs need different resilience patterns

A typical internal microservice either responds in milliseconds or is clearly down. LLM providers behave differently, and the difference matters for how you write the client code:

• Inference latency is highly variable. A simple completion might return in 2 seconds; a complex reasoning request might take 30. A generic 10-second timeout kills valid requests just as often as it catches genuinely stuck ones.
• Availability is meaningfully lower than typical cloud infrastructure. LLM providers commonly run at 99–99.5% uptime, which is noticeably worse than the infrastructure most backend services depend on worth designing for explicitly, not treating as a rare edge case.
• Rate limits are aggressive and multi-dimensional. You're usually bound by both requests-per-minute and tokens-per-minute simultaneously, and hitting either one produces the same 429 response.
• A single provider outage can cascade fast. If your retry logic isn't capped, a five-minute provider outage turns into thousands of hung requests piling up behind it — including into agent or multi-step workflows, where one failed call can stall the whole chain.

None of this means LLM calls need exotic infrastructure. It means the same reliability patterns you'd already apply to any flaky external dependency — timeouts, retries, circuit breakers, rate limiting need to be tuned for these specific failure characteristics.

Setting a realistic timeout

Start by giving each call a deadline that matches the kind of request you're making, using context:

func callLLM(ctx context.Context, prompt string) (string, error) {
	ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
	defer cancel()

	req, err := http.NewRequestWithContext(ctx, http.MethodPost, apiURL, buildBody(prompt))
	if err != nil {
		return "", err
	}
	req.Header.Set("Authorization", "Bearer "+apiKey)

	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		return "", err
	}
	defer resp.Body.Close()

	return parseResponse(resp)
}

30 seconds is a reasonable default for a standard completion; a more complex reasoning call may need 60. The point isn't the exact number it's picking a deadline deliberately, based on the kind of call being made, instead of reusing whatever timeout your HTTP client happened to default to.

Retries with backoff and jitter

Not every failure deserves a retry. A 429 or a transient network error is worth retrying; a 400 for a malformed request is not retrying it just wastes a call and adds latency for no benefit. Classify the error first, then retry with exponential backoff plus jitter, so retries from many concurrent requests don't all land at the same moment:

func withRetry(ctx context.Context, fn func() error) error {
	const maxAttempts = 3
	base := 500 * time.Millisecond

	var err error
	for attempt := 0; attempt < maxAttempts; attempt++ {
		err = fn()
		if err == nil {
			return nil
		}
		if !isRetryable(err) {
			return err
		}

		backoff := base * time.Duration(1<<attempt)
		jitter := time.Duration(rand.Int63n(int64(backoff) / 2))
		wait := backoff + jitter

		select {
		case <-time.After(wait):
		case <-ctx.Done():
			return ctx.Err()
		}
	}
	return fmt.Errorf("max retries exceeded: %w", err)
}

func isRetryable(err error) bool {
	var apiErr *APIError
	if errors.As(err, &apiErr) {
		return apiErr.StatusCode == http.StatusTooManyRequests ||
			apiErr.StatusCode >= 500
	}
	return errors.Is(err, context.DeadlineExceeded)
}

Without jitter, retries from many requests failing at the same time re-synchronize and hit the provider in the same bursts which is exactly the pattern that keeps a struggling provider from recovering.

Circuit breaker: fail fast instead of piling up

Retries handle a single failed call. A circuit breaker handles the case where the provider is down for longer than a retry loop should tolerate instead of every request timing out slowly, the breaker trips and fails immediately for a cooldown period:

type CircuitBreaker struct {
	mu           sync.Mutex
	failures     int
	threshold    int
	state        string // "closed", "open", "half-open"
	openedAt     time.Time
	cooldown     time.Duration
}

func NewCircuitBreaker(threshold int, cooldown time.Duration) *CircuitBreaker {
	return &CircuitBreaker{threshold: threshold, cooldown: cooldown, state: "closed"}
}

func (cb *CircuitBreaker) Call(fn func() error) error {
	cb.mu.Lock()
	if cb.state == "open" {
		if time.Since(cb.openedAt) > cb.cooldown {
			cb.state = "half-open"
		} else {
			cb.mu.Unlock()
			return errors.New("circuit breaker open: failing fast")
		}
	}
	cb.mu.Unlock()

	err := fn()

	cb.mu.Lock()
	defer cb.mu.Unlock()

	if err != nil {
		cb.failures++
		if cb.state == "half-open" || cb.failures >= cb.threshold {
			cb.state = "open"
			cb.openedAt = time.Now()
		}
		return err
	}

	cb.failures = 0
	cb.state = "closed"
	return nil
}

The impact of this is easy to underestimate until you run the numbers. During a five-minute outage handling 100 requests per minute, a client without a circuit breaker leaves 500-1,000 requests hanging for a full timeout each; with a breaker tripping after roughly 10-15 failures, the remaining requests fail in microseconds instead of stacking up behind a dead endpoint. For a user-facing feature, that's the difference between a fast, clear error and a frozen UI.

Start conservative: a threshold around 5 consecutive failures and a 30-60 second cooldown is a reasonable baseline, tightened later based on what you actually observe in production.

Rate limiting: respect the provider's limits before they reject you

Rather than waiting for 429s and reacting to them, a token-bucket limiter can throttle outbound calls to stay under quota in the first place. Go's standard library already has one:

import "golang.org/x/time/rate"

var limiter = rate.NewLimiter(rate.Limit(50), 10) // 50 req/s, burst of 10

func callWithRateLimit(ctx context.Context, prompt string) (string, error) {
	if err := limiter.Wait(ctx); err != nil {
		return "", err
	}
	return callLLM(ctx, prompt)
}

If your provider limits by both requests-per-minute and tokens-per-minute, a single request-count limiter isn't enough a large prompt can exhaust your token budget well before it exhausts your request count. Track estimated token usage alongside request count if your traffic includes highly variable prompt sizes.

Putting it together

Composed, the call path looks like: rate limiter → circuit breaker → retry with backoff → the actual HTTP call with its own timeout. Each layer handles a different failure mode the limiter prevents self-inflicted 429s, the breaker stops hammering a provider that's already down, and the retry logic recovers from the transient blips that happen even when the provider is healthy.

func RobustLLMCall(ctx context.Context, breaker *CircuitBreaker, prompt string) (string, error) {
	var result string

	err := breaker.Call(func() error {
		return withRetry(ctx, func() error {
			if err := limiter.Wait(ctx); err != nil {
				return err
			}
			res, err := callLLM(ctx, prompt)
			result = res
			return err
		})
	})

	return result, err
}

Try it yourself

The full, runnable version of everything above — split into a clean resilience package (circuit breaker, retry, client, rate limiting), an OpenAI-compatible provider implementation, and an Observer that logs each retry and breaker state change so you can actually watch it work — is available on GitHub:

github.com/jutionck/resilient-llm-go

It ships with three ready-to-run commands: a simulated flaky provider (no API key needed), a real OpenAI call, and a free option using DeepSeek V4 Flash Free through OpenCode Zen so you can see the circuit breaker actually trip and recover without spending anything:

git clone https://github.com/jutionck/resilient-llm-go
cd resilient-llm-go
go mod tidy
go run ./cmd/demo

A note on fallback providers

If your application can tolerate switching models, wiring a secondary provider behind the same circuit breaker gives you graceful degradation instead of an outright failure: when the primary breaker is open, route to the fallback rather than surfacing an error to the user. This adds real complexity differing response formats, different latency and cost profiles so it's worth reserving for calls where availability matters more than using one specific model.

Conclusion

LLM APIs are external dependencies, and they deserve the same defensive engineering you'd apply to any other third-party service arguably more, given how much more frequently they fail compared to typical cloud infrastructure. Timeouts matched to real inference latency, retries with jitter to avoid synchronized thundering herds, a circuit breaker to fail fast during outages, and rate limiting to stay under quota in the first place together, these turn "the LLM provider is having a bad day" from an incident into a handled, expected case.

None of this is exotic. It's the same resilience thinking already covered for service-to-service calls in Go microservices, pointed at a dependency that happens to fail more often and more unpredictably than most.

Have you had an LLM provider outage take down more of your system than it should have? What was missing timeouts, a circuit breaker, or something else? Share it in the comments.