# LLM APIs

> Call the Anthropic and OpenAI APIs: models, streaming, tool use, structured output, caching, rate limits, errors and key handling.

Canonical: https://www.wiki.jodisand.me/llm-apis/
Reviewed: 2026-09-24
Related: [Python](https://www.wiki.jodisand.me/python/index.md), [HTTP and curl](https://www.wiki.jodisand.me/http/index.md), [jq](https://www.wiki.jodisand.me/jq/index.md), [Vault](https://www.wiki.jodisand.me/vault/index.md), [TypeScript](https://www.wiki.jodisand.me/typescript/index.md)


## Cheatsheet

| Task | Anthropic | OpenAI |
| --- | --- | --- |
| API key variable | `ANTHROPIC_API_KEY` | `OPENAI_API_KEY` |
| Endpoint | `POST https://api.anthropic.com/v1/messages` | `POST https://api.openai.com/v1/responses` |
| Auth header | `x-api-key: $ANTHROPIC_API_KEY` plus `anthropic-version: 2023-06-01` | `Authorization: Bearer $OPENAI_API_KEY` |
| SDK call | `client.messages.create(model=..., max_tokens=..., messages=[...])` | `client.responses.create(model=..., input=...)` |
| System prompt | Top-level `system=` parameter, not a message | `instructions=` parameter |
| Output cap | `max_tokens` (required) | `max_output_tokens` |
| Stream | `with client.messages.stream(...) as s: for t in s.text_stream:` | `stream=True` |
| JSON output | `output_config={"format": {"type": "json_schema", ...}}` or `client.messages.parse(output_format=Model)` | `text={"format": {"type": "json_schema", ...}}` or `client.responses.parse(text_format=Model)` |
| Count tokens before sending | `client.messages.count_tokens(model=..., messages=[...])` | |
| Rate limit headers | `anthropic-ratelimit-*`, `retry-after` | `x-ratelimit-*`, `retry-after` |
| Request ID for support | `request-id` header, `resp._request_id` in the SDK | |
| Cheap repeated context | Prompt caching (`cache_control`) | Automatic prompt caching |
| Work that can wait | Message Batches API, 50% off | Batch API |

## Current Anthropic models

| Model | API ID | Input / output per MTok | Context | Max output | Notes |
| --- | --- | --- | --- | --- | --- |
| Claude Fable 5.1 | `claude-fable-5-1` | $10 / $50 | 1M | 128K | Most capable. Thinking always on, no forced tool use |
| Claude Opus 5.5 | `claude-opus-5-5` | $4 / $20 | 1M | 128K | Anthropic's default recommendation. Thinking always on, default effort `medium`, no forced tool use |
| Claude Opus 5 | `claude-opus-5` | $5 / $25 | 1M | 128K | Previous Opus, still served |
| Claude Sonnet 5 | `claude-sonnet-5` | $2 / $10 | 1M | 128K | Speed and cost balance |
| Claude Haiku 4.5 | `claude-haiku-4-5-20251001` (alias `claude-haiku-4-5`) | $1 / $5 | 200K | 64K | Fastest and cheapest. Uses `budget_tokens` thinking, no `effort` |

Dateless IDs from the 4.6 generation onwards are pinned snapshots, not moving aliases. List what your key can use, with context window (`max_input_tokens`) and output cap (`max_tokens`), from the Models API:

```sh
# List models available to this key (read-only)
curl -s https://api.anthropic.com/v1/models \
  -H "x-api-key: $ANTHROPIC_API_KEY" -H 'anthropic-version: 2023-06-01' \
  | jq -r '.data[] | [.id, .max_input_tokens, .max_tokens] | @tsv'
```

Prices and IDs change. Check the [models overview](https://platform.claude.com/docs/en/about-claude/models/overview) and [pricing](https://platform.claude.com/docs/en/about-claude/pricing) before committing to a budget. For OpenAI, see the [OpenAI models page](https://developers.openai.com/api/docs/models).

## Request shape

Both APIs are stateless. Every request carries the whole conversation you want the model to see, and you pay for all of it on every call.

```python
import anthropic

client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY

resp = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system="You are a precise assistant. Answer with commands, not prose.",
    messages=[
        {"role": "user", "content": "How do I list pods that are not running?"},
    ],
)
text = next(b.text for b in resp.content if b.type == "text")
print(text)
print(resp.stop_reason, resp.usage.input_tokens, resp.usage.output_tokens)
```

Read text by filtering for `type == "text"` rather than taking `content[0]`: on models with thinking on, the first block can be a `thinking` block.

```python
from openai import OpenAI

client = OpenAI()                       # reads OPENAI_API_KEY
resp = client.responses.create(
    model="gpt-6-sol",
    instructions="Answer with commands, not prose.",
    input="How do I list pods that are not running?",
    max_output_tokens=1024,
)
print(resp.output_text)
```

| Parameter (Anthropic) | Effect |
| --- | --- |
| `max_tokens` | Hard cap on output tokens, thinking included. Hitting it truncates the answer |
| `system` | Role and constraints. A top-level parameter, not a message |
| `stop_sequences` | End generation when a marker string appears |
| `thinking` | `{"type": "adaptive"}` on current models. Fable 5.1 and Opus 5.5 always think; omit the field |
| `output_config.effort` | `low`, `medium`, `high`, `xhigh`, `max`. Trades depth for tokens and latency |
| `stream` | Server-sent events instead of one JSON body |
| `temperature`, `top_p`, `top_k` | Rejected with a 400 on Opus 4.7 and later, Sonnet 5 and Fable. Still accepted on Haiku 4.5 and 4.6 models |

`stop_reason` tells you how the turn ended:

| `stop_reason` | Meaning | Action |
| --- | --- | --- |
| `end_turn` | Finished normally | Use the answer |
| `max_tokens` | Cut off at the cap | Raise `max_tokens` or ask for less |
| `stop_sequence` | Hit one of `stop_sequences` | Expected |
| `tool_use` | Wants a tool result | Run the tool, send a `tool_result`, call again |
| `pause_turn` | A long server-tool turn paused | Send the response back unchanged to continue |
| `refusal` | Declined by a safety classifier (HTTP 200) | Read `stop_details.category`; do not treat `content` as an answer |
| `model_context_window_exceeded` | Input plus output reached the context window | Trim or summarise history |

> [!IMPORTANT] Prefill and sampling parameters were removed
> A conversation ending with a partial assistant message (prefill) returns a 400 on Claude 4.6 and later models. Use structured output instead. `temperature` and `top_p` return a 400 on Opus 4.7 and later, Sonnet 5 and Fable; control behaviour with the prompt and `effort`.

## Streaming

```python
with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=16000,
    messages=[{"role": "user", "content": prompt}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()   # full Message, same shape as a non-streaming call
```

Streaming changes perceived latency more than model choice does. It also avoids HTTP timeouts on long outputs: the SDKs refuse non-streaming requests they expect to run past 10 minutes, so use streaming for large `max_tokens` (up to 128K). Closing the stream early stops generation.

Errors can arrive after the HTTP 200 as an `error` event inside the stream, so a stream handler needs its own error path.

## Structured output and tool use

For JSON, constrain the output with a schema instead of asking for JSON in the prompt.

```python
from pydantic import BaseModel

class Incident(BaseModel):
    severity: str
    service: str
    summary: str

resp = client.messages.parse(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": transcript}],
    output_format=Incident,
)
incident = resp.parsed_output            # validated Incident instance
```

Without Pydantic, pass `output_config={"format": {"type": "json_schema", "schema": {...}}}` to `messages.create` and `json.loads` the text block. The schema needs `required` and `"additionalProperties": False`. The older top-level `output_format` parameter on `messages.create` is deprecated.

Tools let the model request an action. Set `"strict": True` on a tool so its `input` always validates against `input_schema`:

```python
tools = [{
    "name": "get_pod_status",
    "description": "Return the status of pods in a namespace.",
    "strict": True,
    "input_schema": {
        "type": "object",
        "properties": {"namespace": {"type": "string"}},
        "required": ["namespace"],
        "additionalProperties": False,
    },
}]

messages = [{"role": "user", "content": "What is failing in my-namespace?"}]
while True:
    resp = client.messages.create(model="claude-sonnet-5", max_tokens=4096,
                                  tools=tools, messages=messages)
    messages.append({"role": "assistant", "content": resp.content})
    if resp.stop_reason != "tool_use":
        break
    results = [
        {"type": "tool_result", "tool_use_id": b.id, "content": run_tool(b.name, b.input)}
        for b in resp.content if b.type == "tool_use"
    ]
    messages.append({"role": "user", "content": results})   # all results in one message
```

The loop appends the full `resp.content`, not just the text, because thinking and tool-use blocks must go back unchanged. Return every `tool_result` from one turn in a single user message, and report a failed tool with `"is_error": True` rather than dropping it. Validate tool input on your side: a schema constrains shape, not intent.

Forcing a specific tool with `tool_choice={"type": "tool", "name": ...}` or `{"type": "any"}` works on Sonnet 5 and Opus 5 but returns a 400 on Opus 5.5 and Fable 5.1. Use `auto` with `strict` and an instruction in the prompt, or structured output.

OpenAI's equivalents are `client.responses.parse(text_format=Model)` and `text={"format": {"type": "json_schema", "name": ..., "schema": ..., "strict": True}}` on `responses.create`. See the [OpenAI structured outputs guide](https://developers.openai.com/api/docs/guides/structured-outputs).

> [!WARNING] Treat model output as untrusted input
> Never execute generated commands, SQL or code without review or a sandbox, and never interpolate model output into a shell. Documents and web pages you pass in can carry instructions aimed at your system (prompt injection). The defence is limiting what the output can do, not a better prompt.

## Prompt patterns that earn their place

| Pattern | Use |
| --- | --- |
| Explicit output contract | State the exact format and show one example |
| Few-shot examples | Classification and extraction where the boundary is fuzzy |
| Delimiters around data | `<document>...</document>` so instructions and data cannot be confused |
| Structured output | Fixed JSON shape, replacing the old prefill trick |
| Refusal path | Say what to output when the input does not fit |
| Static first, variable last | Makes the prefix cacheable |

## Cost

Cost is input tokens plus output tokens, each at its own rate, and output costs five times input on current Anthropic models. Thinking tokens bill as output. Token counts are model-specific, so measure rather than estimate:

```python
count = client.messages.count_tokens(
    model="claude-sonnet-5",
    messages=[{"role": "user", "content": prompt}],
)
print(count.input_tokens)
```

Do not use `tiktoken` for Claude. It is OpenAI's tokenizer and undercounts Claude tokens.

| Lever | Effect |
| --- | --- |
| Prompt caching | Cache reads bill at 10% of base input (5% on Opus 5.5, 2.5% on Fable 5.1) |
| Message Batches API | 50% off every token, results within 24 hours |
| Lower `effort` | Fewer thinking and output tokens on routine work |
| Smaller model | Haiku 4.5 or Sonnet 5 for extraction and classification |
| Trim history | Summarise old turns instead of resending them |
| Cap `max_tokens` | Bounds the expensive half of the bill |

Mark the end of the stable prefix with `cache_control`. The render order is `tools`, then `system`, then `messages`, and any byte change before a breakpoint invalidates everything after it.

```python
resp = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": long_static_context,
        "cache_control": {"type": "ephemeral"},   # 5-minute TTL; {"type": "ephemeral", "ttl": "1h"} for 1 hour
    }],
    messages=[{"role": "user", "content": question}],
)
print(resp.usage.cache_creation_input_tokens, resp.usage.cache_read_input_tokens)
```

A request allows at most four breakpoints. Prefixes shorter than the model minimum (512 to 4096 tokens depending on model; 1024 on Sonnet 5, 4096 on Haiku 4.5) silently do not cache. See [prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching).

## Errors and rate limits

| Status | Type | Retry? | Response |
| --- | --- | --- | --- |
| 400 | `invalid_request_error` | No | Fix the request. Also returned when a spend limit you set is reached |
| 401 | `authentication_error` | No | Check which key the process actually loaded |
| 402 | `billing_error` | No | Fix billing in the Console |
| 403 | `permission_error` | No | Key lacks access to the model or workspace |
| 404 | `not_found_error` | No | Wrong path or model ID not available to this org |
| 413 | `request_too_large` | No | Messages requests are capped at 32 MB. Use the Files API for large documents |
| 429 | `rate_limit_error` | Yes | Wait for `retry-after`. No `retry-after` means the monthly spend cap was hit |
| 500 | `api_error` | Yes | Back off; include `request-id` if it persists |
| 504 | `timeout_error` | Yes | Stream long requests |
| 529 | `overloaded_error` | Yes | Back off; consider a less loaded model |

The SDKs already retry connection errors, 408, 409, 429 and 5xx twice with exponential backoff and honour `retry-after`. Raise the count on the client instead of wrapping calls in a second retry library, which multiplies attempts:

```python
client = anthropic.Anthropic(max_retries=5, timeout=120.0)   # timeout in seconds (Python)
```

If you do add your own retry, retry only `anthropic.RateLimitError`, `anthropic.InternalServerError` and `anthropic.APIConnectionError`. Catching `APIStatusError` also retries 400s, which never succeed.

Anthropic rate limits are per model and measured as requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM), using a token bucket that refills continuously. Cache reads do not count towards ITPM on current models, so caching raises effective throughput. `max_tokens` does not count against OTPM. Short bursts can trip a per-minute limit enforced per second, and a sudden jump in traffic can trigger acceleration limits, so ramp up gradually. See [rate limits](https://platform.claude.com/docs/en/api/rate-limits).

## Prompt patterns in practice

A prompt is configuration, so treat it like one: version it in the repository, keep the static part identical between requests so it caches, and test it against a fixed set of inputs before changing it. The patterns below are the ones that survive contact with production; each has a concrete shape.

```python
system = """You are an on-call assistant for a Kubernetes platform team.
Answer with commands first, one short sentence of explanation after each.
If the question cannot be answered from the supplied context, reply exactly: INSUFFICIENT_CONTEXT.
Never invent flags. Only cite files that appear in <context>."""

user = f"""<context>
{runbook_excerpt}
</context>

<question>
{question}
</question>"""
```

The system prompt states the role, the output contract and the refusal path; the user turn carries data inside delimiters so the model cannot mistake a line in the runbook for an instruction. Put the long, unchanging material first (context, examples) and the short, per-request question last, which is also the cache-friendly order. Few-shot examples work best as prior conversation turns (`user` example, `assistant` answer) rather than pasted into one block, because the model treats them as demonstrated behaviour rather than described behaviour. For classification, list the allowed labels and use structured output with an `enum`, not a request to "reply with one word". For extraction, describe each field's meaning in the schema's `description` rather than in prose. Avoid negative-only instructions ("do not be verbose") without a positive target ("answer in at most three sentences"); the model follows the latter more reliably. On Fable and Opus 5 models, prompts written for older models are often over-prescriptive and reduce quality, so cut step-by-step scaffolding and let `effort` control depth.

## Tool use in Go and curl

The loop is the same in every language: send tools, read `stop_reason`, run each `tool_use` block, return every `tool_result` in one user message, repeat. In Go the SDK converts the response into history with `ToParam()`; the raw `Input` is JSON to unmarshal, never a parsed map.

```go
import (
    "context"
    "encoding/json"

    "github.com/anthropics/anthropic-sdk-go"
)

podTool := anthropic.ToolParam{
    Name:        "get_pod_status",
    Description: anthropic.String("Return the status of pods in a namespace."),
    InputSchema: anthropic.ToolInputSchemaParam{
        Properties: map[string]any{"namespace": map[string]any{"type": "string"}},
        Required:   []string{"namespace"},
    },
    Strict: anthropic.Bool(true),
}
tools := []anthropic.ToolUnionParam{{OfTool: &podTool}}
messages := []anthropic.MessageParam{anthropic.NewUserMessage(anthropic.NewTextBlock("What is failing in my-namespace?"))}

for {
    resp, err := client.Messages.New(ctx, anthropic.MessageNewParams{
        Model: "claude-sonnet-5", MaxTokens: 4096, Tools: tools, Messages: messages,
    })
    if err != nil { return err }
    messages = append(messages, resp.ToParam())                 // thinking and tool_use blocks go back unchanged
    if resp.StopReason != anthropic.StopReasonToolUse { break }

    var results []anthropic.ContentBlockParamUnion
    for _, block := range resp.Content {
        if tu, ok := block.AsAny().(anthropic.ToolUseBlock); ok {
            var in struct{ Namespace string `json:"namespace"` }
            if err := json.Unmarshal([]byte(tu.JSON.Input.Raw()), &in); err != nil {
                results = append(results, anthropic.NewToolResultBlock(block.ID, "invalid input: "+err.Error(), true))
                continue
            }
            out, err := getPodStatus(ctx, in.Namespace)          // validate in.Namespace against an allow-list first
            results = append(results, anthropic.NewToolResultBlock(block.ID, out, err != nil))
        }
    }
    messages = append(messages, anthropic.NewUserMessage(results...))
}
```

The same second request in curl, with the assistant's `tool_use` block replayed and the result attached:

```sh
curl -s https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' \
  -d '{
    "model": "claude-sonnet-5", "max_tokens": 1024,
    "tools": [{"name": "get_pod_status", "description": "Return the status of pods in a namespace.", "strict": true,
               "input_schema": {"type": "object", "properties": {"namespace": {"type": "string"}}, "required": ["namespace"], "additionalProperties": false}}],
    "messages": [
      {"role": "user", "content": "What is failing in my-namespace?"},
      {"role": "assistant", "content": [{"type": "tool_use", "id": "toolu_01A", "name": "get_pod_status", "input": {"namespace": "my-namespace"}}]},
      {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "toolu_01A", "content": "api-7d9f CrashLoopBackOff (3 restarts)"}]}
    ]}' | jq -r '.content[] | select(.type == "text") | .text'
```

OpenAI's Responses API uses the same loop with different names: tools are `{"type": "function", "name": ..., "parameters": ..., "strict": true}`, the model returns `function_call` items with a `call_id` and an `arguments` JSON string, and you append `{"type": "function_call_output", "call_id": ..., "output": ...}` to the `input` list along with the model's output items.

```python
resp = client.responses.create(model="gpt-6-sol", tools=tools, input=input_list)
input_list += resp.output                                       # keep the model's items in history
for item in resp.output:
    if item.type == "function_call":
        args = json.loads(item.arguments)
        input_list.append({"type": "function_call_output", "call_id": item.call_id, "output": run_tool(item.name, args)})
resp = client.responses.create(model="gpt-6-sol", tools=tools, input=input_list)
```

## Structured output with curl

The schema goes in `output_config.format`; the text block is then guaranteed to parse and validate. Keep every property in `required` and set `additionalProperties: false`, or the request is rejected.

```sh
jq -n --rawfile t transcript.txt '{
  model: "claude-sonnet-5", max_tokens: 1024,
  messages: [{role: "user", content: $t}],
  output_config: {format: {type: "json_schema", schema: {
    type: "object",
    properties: {
      severity: {type: "string", enum: ["sev1", "sev2", "sev3"]},
      service:  {type: "string"},
      summary:  {type: "string", description: "One sentence, no more than 30 words"}
    },
    required: ["severity", "service", "summary"], additionalProperties: false
  }}}}' \
| curl -s https://api.anthropic.com/v1/messages \
    -H "x-api-key: $ANTHROPIC_API_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- \
| jq -r '.content[] | select(.type == "text") | .text' | jq .
```

Check `stop_reason` before parsing: a `max_tokens` stop yields truncated JSON, and a `refusal` yields no answer at all. The OpenAI equivalent is `"text": {"format": {"type": "json_schema", "name": "incident", "schema": {...}, "strict": true}}` on `/v1/responses`, read from `output_text`.

## Streaming in Go and against OpenAI

Go has no `get_final_message`; accumulate events into a `Message` and read text deltas as they arrive.

```go
stream := client.Messages.NewStreaming(ctx, anthropic.MessageNewParams{
    Model: "claude-sonnet-5", MaxTokens: 64000,
    Messages: []anthropic.MessageParam{anthropic.NewUserMessage(anthropic.NewTextBlock(prompt))},
})
var msg anthropic.Message
for stream.Next() {
    ev := stream.Current()
    msg.Accumulate(ev)                                          // builds the complete Message as events arrive
    if d, ok := ev.AsAny().(anthropic.ContentBlockDeltaEvent); ok {
        if td, ok := d.Delta.AsAny().(anthropic.TextDelta); ok { fmt.Print(td.Text) }
    }
}
if err := stream.Err(); err != nil { return err }             // includes error events delivered mid-stream
fmt.Println(msg.StopReason, msg.Usage.OutputTokens)
```

```python
with client.responses.stream(model="gpt-6-sol", input=prompt) as stream:   # OpenAI
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
    final = stream.get_final_response()
```

Both APIs stream Server-Sent Events. Anthropic emits `message_start`, `content_block_start`, `content_block_delta`, `content_block_stop`, `message_delta` (with `stop_reason` and output usage) and `message_stop`; OpenAI emits typed `response.*` events with a `sequence_number`. Tool input streams as `input_json_delta` fragments that are only valid JSON once the block stops, so buffer them per `index` before parsing. Behind a reverse proxy, disable response buffering for the streaming path (`proxy_buffering off` in nginx) or the client sees nothing until the end.

## Message Batches

The Batches API takes up to 100,000 requests (256 MB) in one call, processes them asynchronously within 24 hours (most finish within an hour), and bills every token at half price. Any Messages request, including tools, caching and structured output, is valid as a batch item. Results come back in arbitrary order, keyed by `custom_id`, and stay available for 29 days.

```sh
# Submit: one requests[] entry per prompt, each with a custom_id you can join on later
jq -n '{requests: [inputs | {custom_id: .id, params: {model: "claude-haiku-4-5", max_tokens: 512,
        messages: [{role: "user", content: .text}]}}]}' items.jsonl \
| curl -s https://api.anthropic.com/v1/messages/batches \
    -H "x-api-key: $ANTHROPIC_API_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- \
| tee batch.json | jq '{id, processing_status, request_counts}'

# Poll until processing_status is "ended", then stream the JSONL results
batch=$(jq -r .id batch.json)
until [ "$(curl -s https://api.anthropic.com/v1/messages/batches/$batch -H "x-api-key: $ANTHROPIC_API_KEY" -H 'anthropic-version: 2023-06-01' | jq -r .processing_status)" = ended ]; do sleep 60; done
curl -s "https://api.anthropic.com/v1/messages/batches/$batch/results" -H "x-api-key: $ANTHROPIC_API_KEY" -H 'anthropic-version: 2023-06-01' \
| jq -r 'select(.result.type == "succeeded") | [.custom_id, (.result.message.content[] | select(.type == "text") | .text)] | @tsv'
```

```python
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

batch = client.messages.batches.create(requests=[
    Request(custom_id=item.id, params=MessageCreateParamsNonStreaming(
        model="claude-haiku-4-5", max_tokens=512, messages=[{"role": "user", "content": item.text}]))
    for item in items
])
while client.messages.batches.retrieve(batch.id).processing_status != "ended":
    time.sleep(60)
for r in client.messages.batches.results(batch.id):
    if r.result.type == "succeeded":
        store(r.custom_id, r.result.message)
    elif r.result.type == "errored" and r.result.error.type != "invalid_request":
        retry_later(r.custom_id)                                # server-side errors are safe to resubmit; invalid_request is not
```

OpenAI's Batch API is file-based: write one JSON request per line with a `custom_id`, `method`, `url` (`/v1/responses` or `/v1/chat/completions`) and `body`, upload it with `client.files.create(file=..., purpose="batch")`, then `client.batches.create(input_file_id=..., endpoint="/v1/responses", completion_window="24h")` and download `output_file_id` when `status` is `completed`. Batch requests have their own rate limits, so they do not compete with interactive traffic. Use batches for evaluation runs, backfills, nightly classification and anything a human is not waiting on.

## Cost control

Measure before optimising: log `usage` from every response with the model, route and a request ID, and compute cost per completed task rather than per call, because a cheaper call that needs two retries is not cheaper. The levers, in the order they usually pay off:

| Lever | What it does | Typical saving |
| --- | --- | --- |
| Prompt caching | Stable prefix billed at a fraction of input price on hits | Most of the input bill on chat and agent loops |
| Trim input | Drop boilerplate, old turns and unused tool definitions; summarise history | Proportional to what you remove |
| Batches | Same models, 50% off, asynchronous | Half of any non-interactive workload |
| Lower `effort` | Fewer thinking and output tokens on routine requests | Large on output-heavy routes; measure quality |
| Smaller model per route | Haiku or Sonnet for classification, extraction and routing | 5 to 10 times on those routes |
| Cap `max_tokens` per route | Bounds the expensive half; pairs with structured output | Stops runaway outputs |
| Stop sequences and tight contracts | Model stops when the answer is complete | Small but free |

Put a spend limit on each workspace in the Console so a bug loops against a cap rather than a credit card, and alert on daily spend from your own `usage` logs, not the invoice. For agentic loops, cap the number of tool turns and the cumulative output tokens per task, and stop when a tool result repeats. A cache hit rate below expectations is nearly always a byte of variability before the last breakpoint: a timestamp in the system prompt, unsorted JSON keys, or a tool list that changes per request. Caches are model-scoped, so routing requests across models forfeits cache reuse; test the newest model at low effort before building a multi-model cascade, because it often beats an older model at high effort for less.

## Evaluation

A prompt change without an evaluation is a guess. The minimum useful eval is a file of real or representative inputs with an expected outcome per row, a runner that calls your application's actual entry point (not a re-implementation of the API call), and a grader that turns each output into a number. Run it before and after every prompt, model or parameter change and compare.

| Grader | Use when | Cost |
| --- | --- | --- |
| Programmatic check | Output is a label, number, JSON shape, or passes tests | Free, deterministic |
| Pairwise judge | Comparing two versions of open-ended output; randomise A/B order, allow `tie` | One judge call per case |
| Rubric judge | Absolute score on open-ended output against concrete, checkable criteria | One judge call per case |
| Human spot-check | Criteria are hard to write down | Limits how often the eval runs |

```python
import json, statistics

cases = [json.loads(l) for l in open("evals/triage.jsonl", encoding="utf-8")]   # {"input": ..., "expected": "sev2"}
scores, costs = [], []
for c in cases:
    resp = client.messages.parse(model=MODEL, max_tokens=512, system=SYSTEM,
                                 messages=[{"role": "user", "content": c["input"]}], output_format=Incident)
    scores.append(resp.parsed_output.severity == c["expected"])
    costs.append(resp.usage.input_tokens * IN_PRICE + resp.usage.output_tokens * OUT_PRICE)
print(f"accuracy={statistics.mean(scores):.3f} cost_per_case=${statistics.mean(costs):.4f} n={len(cases)}")
```

Report precision and recall per label rather than a single accuracy figure when the classes are unbalanced, because a change can trade one for the other invisibly. When a model grades another model's output, use structured output for the verdict, never let the model under test judge itself, and read a sample of graded cases by hand before trusting the rubric. Keep the eval set out of the prompt and out of any few-shot examples, split it into a tuning set and a held-out set if you iterate on the prompt against it, and record the model ID, prompt hash and per-case latency and cost alongside the score so a regression can be traced. Run the full set through the Batches API to halve the cost of each evaluation run.

## OpenAI-compatible endpoints

Many servers speak the OpenAI Chat Completions wire format: vLLM, Ollama, llama.cpp's server, LM Studio, and gateway products such as LiteLLM that translate to other providers. The OpenAI SDKs target them by changing `base_url`; the API key is often ignored but the SDK requires a non-empty value. Coverage varies: most implement `/v1/chat/completions` and `/v1/models`, fewer implement `/v1/responses`, tool calling depends on the model and the server's parser, and `response_format` JSON schemas may be enforced, ignored or approximated.

```python
from openai import OpenAI

local = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama")   # Ollama; vLLM defaults to :8000/v1
resp = local.chat.completions.create(
    model="llama3.3",
    messages=[{"role": "system", "content": "Answer briefly."}, {"role": "user", "content": "ping"}],
    max_tokens=64,
    temperature=0,
)
print(resp.choices[0].message.content, resp.usage)
```

```sh
# Same call from the shell; works against OpenAI, vLLM, Ollama and LiteLLM with only the URL and model changed
curl -s http://127.0.0.1:8000/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"my-model","messages":[{"role":"user","content":"ping"}],"max_tokens":64,"stream":false}' \
  | jq -r '.choices[0].message.content'

# List what the server serves
curl -s http://127.0.0.1:8000/v1/models | jq -r '.data[].id'
```

Anthropic also exposes an [OpenAI-compatible `/v1/chat/completions` layer](https://platform.claude.com/docs/en/cli-sdks-libraries/libraries/openai-sdk), documented as intended for testing and comparing models rather than production (`base_url="https://api.anthropic.com/v1/"`, model set to a Claude ID), but it maps only a subset of features; use the native Messages API for anything beyond a first comparison. The reverse (a Claude-shaped client against a local model) is what gateways are for. Whichever shim you use, `usage` accounting, error codes and rate-limit headers differ from the real provider, so do not tune retry logic against one and deploy against the other. Keep the provider behind one interface in your code (a function that takes messages and returns text plus usage) so switching is a configuration change, and run the same eval set against each candidate before switching.

## Troubleshooting

| Symptom | Likely cause | Check |
| --- | --- | --- |
| 400 mentioning prefill | Last message is from the assistant | End on a user message; use structured output |
| 400 on `temperature` or `top_p` | Sampling parameters removed on this model | Remove them |
| 400 `"thinking.type.enabled" is not supported` | `budget_tokens` on a 4.7 or later model | Use `thinking={"type": "adaptive"}` and `output_config.effort` |
| 400 on `tool_choice` `any` or `tool` | Opus 5.5 or Fable 5.1 | `tool_choice` `auto` plus `strict` |
| 400 about thinking blocks being modified | History was filtered or edited before resending | Append `resp.content` unchanged |
| Answer cut off mid-sentence | `stop_reason` is `max_tokens` | Raise `max_tokens`; thinking counts towards it |
| `content[0].text` raises `AttributeError` | First block is `thinking` or `tool_use` | Filter blocks by `type` |
| Empty or odd output with HTTP 200 | `stop_reason` is `refusal` | `stop_details.category` |
| 429 that never clears | Monthly spend cap, no `retry-after` | `error.details.error_code` is `enforced_spend_limit_reached` |
| 429 well below the RPM limit | Token limit (ITPM or OTPM) or a burst | `anthropic-ratelimit-*-remaining` headers |
| `cache_read_input_tokens` is always 0 | Prefix changes per request, or is below the minimum | Timestamps or unsorted JSON in `system`; prefix length |
| Non-streaming call rejected or times out | Large `max_tokens` without streaming | Use `messages.stream` and `get_final_message()` |
| 401 in CI but works locally | Different or empty key in that environment | `test -n "$ANTHROPIC_API_KEY"`, without printing it |
| Tool loop never terminates | Tool result omitted, or the same failing result returned each turn | Log `stop_reason` per turn; cap turns; return `is_error: true` with a useful message |
| `tool_use.input` fails your parser | Escaping differs between models, or `strict` not set | `json.loads` the raw input; set `"strict": true` and `additionalProperties: false` |
| Structured output rejected with 400 on the schema | Property missing from `required`, or `additionalProperties` not `false` | Every property required; nested objects follow the same rule |
| Streamed tool input is not valid JSON | Parsed `input_json_delta` fragments before `content_block_stop` | Buffer deltas per block index and parse once |
| Stream arrives all at once through a proxy | Response buffering | `proxy_buffering off` (nginx) or the equivalent on the gateway |
| Batch result missing for a `custom_id` | Result type is `errored`, `canceled` or `expired` | Filter on `.result.type`; resubmit server errors, fix `invalid_request` |
| Batch stuck in `in_progress` for hours | Normal within the 24-hour window | Poll every minute or more; check `request_counts` for progress |
| Local OpenAI-compatible server ignores `response_format` | Server or model does not enforce schemas | Validate the output yourself; check the server's docs for JSON mode |
| `openai.AuthenticationError` against a local server | Empty `api_key` | Pass any non-empty string; the server ignores it |
| Eval score moves between identical runs | Non-deterministic output, or a judge with no fixed reference | Run each case several times and report the mean; freeze reference outputs for pairwise judging |

## Keys and data

```sh
# Load the key from Vault at runtime (never commit it)
export ANTHROPIC_API_KEY="$(vault kv get -mount=secret -field=key llm/anthropic)"
```

Keep keys in a secret manager and inject them at runtime; see [Vault](https://www.wiki.jodisand.me/vault/). Never put a key in a repository, an image layer, a client-side bundle or a CI log. Scope keys per workload (workspaces) so one leak is one revocation, and rotate on a schedule.

Do not send credentials, customer personal data or production secrets in a prompt unless your data handling agreement covers it. Redact before the call, not after.

## Oneliners

```sh
# Smoke test a key (prints the reply text)
curl -s https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' \
  -d '{"model":"claude-haiku-4-5","max_tokens":64,"messages":[{"role":"user","content":"ping"}]}' \
  | jq -r '.content[] | select(.type == "text") | .text'
```

```sh
# Show token usage and stop reason for one call
curl -s https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' \
  -d '{"model":"claude-haiku-4-5","max_tokens":64,"messages":[{"role":"user","content":"ping"}]}' \
  | jq '{stop_reason, usage}'
```

```sh
# Stream text deltas from the shell
curl -sN https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' \
  -d '{"model":"claude-sonnet-5","max_tokens":1024,"stream":true,"messages":[{"role":"user","content":"list three checks"}]}' \
  | sed -un 's/^data: //p' | jq -rj --unbuffered 'select(.type == "content_block_delta" and .delta.type == "text_delta") | .delta.text'
```

```sh
# Show rate limit headroom and the request ID (spends a few tokens)
curl -sD - -o /dev/null https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' \
  -d '{"model":"claude-haiku-4-5","max_tokens":1,"messages":[{"role":"user","content":"x"}]}' \
  | grep -iE '^(anthropic-ratelimit|retry-after|request-id)'
```

```sh
# Count tokens in a file (free, separate rate limit, no generation)
jq -n --rawfile p prompt.txt '{model:"claude-sonnet-5", messages:[{role:"user", content:$p}]}' \
  | curl -s https://api.anthropic.com/v1/messages/count_tokens \
      -H "x-api-key: $ANTHROPIC_API_KEY" -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @- \
  | jq .input_tokens
```

```sh
# Redact AWS access key IDs and JWTs from a log before using it as context
sed -E 's/(AKIA[0-9A-Z]{16}|eyJ[A-Za-z0-9_-]{20,})/[REDACTED]/g' app.log > safe.log
```

```sh
# OpenAI smoke test
curl -s https://api.openai.com/v1/responses \
  -H "Authorization: Bearer $OPENAI_API_KEY" -H 'content-type: application/json' \
  -d '{"model":"gpt-6-luna","input":"ping","max_output_tokens":64}' \
  | jq -r '.output[] | select(.type == "message") | .content[] | select(.type == "output_text") | .text'
```