↓Skip to main content

From completions to native APIs: how an LLM call works now

·22 mins

If your code treats an LLM call as text going back and forth, with messages and tool definitions in and text and tool calls out, it’s missing a significant part of what the latest models offer. A response now contains encrypted reasoning, results of tools run by the provider and, optionally, conversation state held by the provider. Code that passes only the text along still works in some cases, but it gets less out of the model. According to OpenAI, its reasoning models perform better on its native Responses API than on Chat Completions and make better use of the prompt cache. In some setups, the model also loses its earlier reasoning between tool calls without any error.

This post shows what an LLM call looks like today, which parts only each provider’s native API supports and what to check in your own setup. I ran into these problems in my own system, which calls several providers and models through Pydantic AI in the application and Bifrost as the gateway, but these findings apply to other setups as well: parameters that worked with earlier models were dropped, and the model lost its earlier reasoning while the response looked normal. This first post stays at the provider level. My earlier article on provider fragmentation covers the economics behind the shift.

In short: for the latest models, and especially with tools, use each provider’s native API. Chat Completions still works for plain generation, but it has no standard field for the model’s reasoning between tool calls, and most hosted tools and all server-side conversation state need the native APIs.

From completions to chat completions #

OpenAI’s API started in 2020 with a completions endpoint: one prompt string in, a continuation of that string out. Any structure, such as a dialogue between a user and an assistant, lived inside the prompt as plain text, and the application had to parse the output itself.

In March 2023, three months after ChatGPT’s launch, OpenAI released Chat Completions together with gpt-3.5-turbo, the model behind ChatGPT. Instead of a single string, the model consumed “a sequence of messages together with metadata”, each message with a role such as system, user or assistant. Function calling followed in June 2023. The application sends the messages and a list of tool definitions, and the model answers with the tool calls it wants. The application runs the tools and sends the results back with the full history, and the model writes the final answer. The provider uses nothing from an earlier request, so every request contains the whole conversation. That state is what most developers and users in general still have in mind.

Anthropic and Google had their own versions of more or less the same behavior. Anthropic moved from Text Completions, a single prompt with Human: and Assistant: turns, to the Messages API at the end of 2023, and Google launched the Gemini API with generateContent in December 2023.

Which endpoint, when, and where each one is heading

own format Chat Completions format Chat Completions adapter beta legacy ✕shut down 2027+: likely continued support 1model or change 3requests that worked before now fail
20222023202420252026now2027+

OpenAI

/v1/completionssince 2020, legacy
Completions · Jun 2020
One prompt string in, a continuation out. The launch path was /v1/engines/{engine}/completions.
Completions · legacy since 2023
Still served for a few older models such as gpt-3.5-turbo-instruct.
/v1/chat/completionssupported; newest models need Responses for tools
Chat Completions · Mar 2023
Messages with roles, function calling from Jun 2023. Still supported; OpenAI recommends Responses for new projects.
1gpt-3.5-turbo · Mar 2023
First model on Chat Completions.
3GPT-5.4 · Mar 2026
Tools on Chat Completions only with reasoning off.
4GPT-6 Astra · Sep 2026
Tool calling only on Responses.
/v1/assistantshosted tools and threads, shut down Aug 2026
Assistants API · Nov 2023
Hosted tools such as Code Interpreter, and conversations stored as threads on OpenAI's side.
✕Assistants API · shut down 26 Aug 2026
Requests to /v1/assistants and /v1/threads no longer work. The migration path is the Responses API.
/v1/responsesMAIN APIsince Mar 2025, recommended
Responses API · Mar 2025
Hosted web search, file search and computer use. Responses are stored by default (store: true).
Responses API · 2027+, likely continued support
OpenAI recommends Responses for all new projects, and its newest models need it for tool calling.
2o1-pro · Mar 2025
First model only on Responses.

Anthropic

/v1/completeText Completions, legacy
Text Completions · 2023
One prompt with Human: and Assistant: turns.
Text Completions · legacy
Legacy since the Messages API. Newer models and features don't support it.
7Python SDK 1.0 · Aug 2026
Text Completions removed.
/v1/messagesMAIN APIsince Dec 2023, all features
Messages API · Dec 2023
Beta in Dec 2023, generally available in Feb 2024. Server tools, thinking and message threads all run here.
Messages API · 2027+, likely continued support
Anthropic's only native API. Every current feature, from server tools to message threads, runs here.
5Claude Opus 4.7 · Apr 2026
Accepts temperature, top_p and top_k only at their defaults.
8API version 2026-10-21 · Oct 2026
Sampling parameters rejected on every model.
/v1/chat/completionsadapter since Feb 2025, "for testing"
OpenAI-compatible layer · Feb 2025
Meant for testing. Ignores strict, doesn't return thinking and has no prompt caching.
6Claude Opus 5 · Jul 2026
Thinking on by default; the adapter doesn't return it.

Google

…/models/{model}:generateContentsince Dec 2023, legacy since Jun 2026
generateContent · Dec 2023
contents with the roles user and model.
generateContent · legacy since Jun 2026
Still supported after the Interactions API became the default.
9Gemini 1.0 Pro · Dec 2023
First model on generateContent.
10Gemini 3 Pro · Nov 2025
Function calls need thought signatures.
/v1beta/openai/chat/completionsadapter since Nov 2024
OpenAI-compatible endpoint · Nov 2024
Gemini through the OpenAI libraries, including thought signatures. Google recommends calling the Gemini API directly.
/v1beta/interactionsMAIN APIbeta Dec 2025, default since Jun 2026
Interactions API · beta Dec 2025
Launched together with the Deep Research agent.
Interactions API · Jun 2026
Generally available and Google's default API. Agents run only here.
Interactions API · 2027+, likely continued support
Google's default API since June 2026; generateContent is legacy.
11Deep Research agent · Dec 2025
Only on Interactions.

Each provider fragmented in its own way. OpenAI added endpoints, first the Assistants API with hosted tools and server-side threads, then Responses, and moved its newest capabilities there: o1-pro was the first model only on /v1/responses, and GPT-6 Astra and GPT-6.1 Sol accept tool calls only there. Anthropic kept /v1/messages and changes the rules per model and API version instead, which is why its red markers are on the same bar. Google moved from generateContent to the Interactions API, where its agents run. Responses, Messages and Interactions are the APIs each provider recommends or treats as its default, and the 2027+ zone is my forecast that providers will add new capabilities only there. The faded orange bars still reach today, so a Chat Completions request is accepted by all three providers.

Why it felt like a standard #

For a while, Chat Completions seemed to me like a standard the industry had converged on, but in late 2025 it became clear that it isn’t. The format felt universal because almost everything around the providers adopted it. OpenAI shipped first and had the largest developer base, so SDKs, tutorials and frameworks were written against its API. Newer providers such as Mistral offered compatible APIs, so switching meant changing a base URL, and gateways like Bifrost put the same format in front of everything else.

Google and Anthropic accept it too, through adapters on top of their own APIs. Google’s arrived in November 2024 and Anthropic’s in February 2025, and both treat it as a secondary option next to their own API. Chat Completions became the shared format through the ecosystem around the providers.

Translating between the formats worked because a conversation consisted of text, function calls and function results, which every format could express one to one. A gateway could turn a Chat Completions request into a Messages request and back without losing anything that mattered.

Reasoning becomes the default #

Reasoning models changed what an assistant turn contains, and they did so in three steps.

Hidden reasoning: OpenAI’s o1, released in September 2024, reasons before it answers. The reasoning tokens are billed as output but not returned. The application has nothing extra to store or send back, so Chat Completions still fits.

Optional thinking with a budget: Anthropic introduced Claude 3.7 Sonnet in February 2025 as “the first hybrid reasoning model”. Thinking is off by default and limited by a token budget. When it’s on, the response contains thinking blocks with a signature, and those blocks have to be sent back unchanged when a tool call follows. Since thinking was disabled by default, the shared format kept working after upgrading.

Reasoning enabled by default, and it has to round-trip: with the latest models, thinking is enabled by default, so the same upgrade now breaks the shared format, in a different way for each provider.

  • Anthropic: adaptive thinking is on by default on Claude 5 models, and Opus 5.5 doesn’t accept switching it off. Adaptive means the model decides per request whether and how long to reason, guided by the effort setting, so a simple question can still get no reasoning at all. Thinking blocks are “Required” within a tool-use turn.
  • Google: Gemini 3 and 4 models require the thought signatures from a function-calling response to be sent back in the current turn and return a validation error otherwise.
  • OpenAI: from GPT-5.4, Chat Completions supports tool calling only with reasoning_effort: "none", and GPT-6 Astra and GPT-6.1 Sol need the Responses API for any tool calling. Responses can return reasoning as encrypted items for applications that don’t keep state at OpenAI.

The newest models add a further condition: the conversation before the reasoning has to stay unchanged. Anthropic’s newest models check that a replayed thinking block still follows the conversation prefix that produced it, and depending on the model and account they reject the request when it doesn’t. On the Responses API, OpenAI’s newer models drop the reasoning that follows a changed prefix and list it in input_transformations. Rewriting the instructions, changing the tool definitions or shortening an earlier tool result each counts as a change.

In all three cases the assistant turn now contains an opaque, provider-signed blob that only that provider can read and that has to be sent back unchanged. While Anthropic and Google returned the raw reasoning with their first reasoning models, none of the three does anymore. Anthropic cites misuse as the reason, and OpenAI cites safety monitoring and competitive advantage. Since the model still needs that reasoning in the next request, it’s returned encrypted, and the signature lets the provider detect any change.

One tool loop, then and now #

The clearest way to see what changed is one question that needs tools, sent once to Chat Completions and once to Anthropic’s Messages API.

Two requests per tool call

SAME QUESTION, SENT TO BOTH“What's the weather in Berlin, and any news?”The weather needs your get_weather tool, and the news needs a web search.
THENGPT-4oChat Completions format/v1/chat/completions
Your appOpenAI API
1messages + tools: [get_weather]Request 1
The whole conversation so far plus the tool definitions.
no hosted toolsGPT-4o on Chat Completions can only ask your app to run tools.
2tool_calls: get_weather("Berlin")Response 1
The model asks for a tool call. The API keeps nothing after sending it.
runs get_weather
3all messages again + tool resultRequest 2
Repeats every message and adds the tool result, because the API stored nothing.
4answer: weather onlyResponse 2
There's no search tool on this side, so the answer has the weather but no news.

The API keeps nothing between 2 and 3. Your app holds the whole conversation.

On Chat Completions with a model like GPT-4o, a question that needs one local tool takes two requests. The first returns tool_calls, and the second contains the full history plus the tool result and returns the answer. The provider uses nothing from the first request to answer the second.

NOWClaude Opus 5.5own format/v1/messagesOpenAI and Google work similarly
+ the provider runs part of the loop+ signed thinking has to be sent back unchanged
Your appAnthropic API
1messages + tools: [get_weather, web_searchServer tool
You declare it, Anthropic runs it inside the turn.
]
2thinking + signatureThinking block
On Claude 5 the text is empty by default. The signature contains the reasoning, encrypted.
, tool_use, server_tool_useSearch call
The model asked for both tools at once, so the search waits for request 2.
runs get_weather
3all messages + thinking unchangedMust match exactly
Sent back as received, after an unchanged system prompt, tools and history. Otherwise the request fails, or thinking is dropped for it.
+ tool result
runs web_searchRuns at Anthropic
The search from response 1 runs now, with no extra step in your code. Results are returned as web_search_tool_result blocks.
4answer: weather and newsResponse 2
Built from your tool result and the search results Anthropic fetched.

Still two requests, and your app still holds the history, now including signed thinking it can't read, unless you opt into server-side state.

The current version still takes two requests, and the application still holds the history, but three things were added:

  1. Hosted tools: the provider can run tools such as web search or code execution inside the turn and returns their results as part of a response.
  2. Signed thinking: when the model reasons, the response contains a thinking block that has to be part of the second request exactly as it was returned. On Claude 5 models the block usually contains no readable text at all, only the signature with the encrypted reasoning. If the block is modified, Anthropic’s API rejects the request. If it’s missing, the request still succeeds, but the model continues without its earlier reasoning.
  3. Optional server-side state: the provider can keep the conversation itself, through previous_response_id on OpenAI’s Responses API, previous_interaction_id on Google’s Interactions API or message threads on Anthropic, so the second request only sends what’s new.

To see what this looks like in practice, I sent the example directly to Anthropic’s API. Here’s the first request and response, with the IDs changed and the signature and code shortened:

The first round trip, from a real run

REQUEST 1Your app → Opus 5.5Messages format, sent to Anthropic's API
{
"model": "claude-opus-5-5",
"max_tokens": 16000,
1 "output_config": { "effort": "high" },
"tools": [
2 {
"name": "get_weather",
"description": "Get the current weather for a city.",
"input_schema": {
"type": "object",
"properties": {
"location": { "type": "string", "description": "City name" }
},
"required": ["location"]
}
},
3 {
"type": "web_search_20260209",
"name": "web_search",
"max_uses": 2
}
],
4 "messages": [
{ "role": "user",
"content": "What's the weather in Berlin, and any news?" }
]
}
  1. 1No thinking parameter, so Opus 5.5 runs adaptive thinking and decides by itself whether to think. Switching it off returns a 400, as does a non-default temperature. At the default effort, medium, it didn't reason for this question in four of four runs, so this request sets high.
  2. 2A function tool: name, description and JSON Schema. Your app runs it, as on Chat Completions.
  3. 3A server tool, declared only by its type. The provider runs it inside the turn.
  4. 4The whole conversation so far, here just the question.
RESPONSE 1Opus 5.5 → your appHTTP 200, shortened
{
"id": "msg_011Dk7RwQe4MhTzL9pVnc2Xb",
"type": "message",
"role": "assistant",
"model": "claude-opus-5-5",
"content": [
{
"type": "thinking",
5 "thinking": "",
6 "signature": "CAQSrwYKEAgTGAI4AUIIdGhp…" // 1,096 characters
},
{ "type": "text",
"text": "I'll check the weather and look up recent news at the same time." },
7 {
"type": "tool_use",
"id": "toolu_01Fq3nRZxk8WcTd5LmHyV2sP",
"name": "get_weather",
"input": { "location": "Berlin" }
},
8 {
"type": "server_tool_use",
"id": "srvtoolu_01Mb7KwYt2PqXn9DzRfh4LcJ",
"name": "code_execution",
"input": { "code": "…web_search({\"query\": \"Berlin news today\"})…" } // 7 lines of Python
}
],
9 "stop_reason": "tool_use",
"usage": {
"input_tokens": 5976, "output_tokens": 278,
"output_tokens_details": { "thinking_tokens": 69 }, …
}
}
  1. 5Empty by default on Claude 5 models. With "display": "summarized" in the thinking parameter, it holds a summary of the reasoning. The model did reason here: 69 of the 278 output tokens are thinking.
  2. 6The reasoning itself, encrypted. Request 2 has to send this block back unchanged.
  3. 7Your tool, the block closest to Chat Completions' tool_calls.
  4. 8The search from 3, requested but not run yet. The model asked for your tool at the same time, so the search runs during request 2 and its results are part of response 2. This tool type searches from Python, hence code_execution.
  5. 9Your app runs get_weather and sends request 2: the history, this content unchanged and the tool result.

Most of what matters in the response can’t be read. The thinking text is empty (5), although 69 of the output tokens are reasoning, which is encrypted in the signature (6). The search (8) is only requested here. Since the model asked for the weather at the same time, the search runs during the second request, and its results are part of the second response.

In the second request, my application sent that content back together with the weather result. To see how strict the round trip is, I also sent the second request with the thinking block left out, with one character of its signature changed and with text added to its empty thinking field, each case directly and through Bifrost, my gateway, in front of the same API. The full example with the search doesn’t get through Bifrost, because it adds an empty code execution result for the pending search to the conversation, and Anthropic’s API rejects the second request with a 400. For a clean comparison, I repeated the loop with only the weather tool:

The thinking block in request 2: four cases

The same response 1 with 36 reasoning tokens, weather tool only, sent back directly and through my gateway.

What request 2 sends back Anthropic's API, directinput tokens of request 2 Through my gatewayBifrost v2.5.1, same API behind it
Thinking block unchanged used37 tokens more than without the block: the reasoning from response 1. The model continues without reasoning again.HTTP 200 · 521 input tokens reasons againBifrost deletes thinking blocks with empty text before it forwards a request, and Claude 5 returns them empty by default.HTTP 200 · 484 input tokens
Thinking block left out reasons againAccepted, because in adaptive mode no assistant turn has to start with a thinking block. The earlier reasoning is gone.HTTP 200 · 484 input tokens reasons againHTTP 200 · 484 input tokens
Signature changed in one character 400 error"Invalid signature in thinking block", as documented: "Modified thinking blocks are rejected with a 400 error."HTTP 400 reasons againDeleted by Bifrost for its empty text, like the unchanged block, so the changed signature never reached Anthropic's API.HTTP 200 · 484 input tokens
Text added to the empty "thinking" field used, text ignoredSame count as the unchanged block: the reasoning from the signature is used, the added text isn't read. The one documented exception: text in the empty field of an omitted block is ignored rather than rejected.HTTP 200 · 521 input tokens used, text ignoredWith text in it, the block isn't deleted. It reaches Anthropic's API with its signature, which uses the reasoning and ignores the text, as directly.HTTP 200 · 521 input tokens

The input token count shows whether the block reached the model: with it, the count is 37 tokens higher, the earlier reasoning. On Anthropic’s API, all four cases behave as documented, including the 400 for the changed signature. Through the gateway, even the unchanged block was dropped. Claude 5 models don’t return their reasoning as text by default: the thinking block is returned with an empty text field, and the reasoning is only in the encrypted signature. Before Bifrost forwards a request, it removes every thinking block with an empty text field, so these blocks never reached Anthropic’s API, even unchanged. The block with the changed signature was removed the same way, so the 400 never happened. On this route, the model lost its reasoning in three of four cases, each time with a normal response. I added both gateway problems, with reproductions, to Bifrost’s open issues: the dropped thinking blocks to #7064 and the added result for the pending search to #9591.

Why no shared standard emerged #

My April post on this topic describes why the divergence is rational for providers: once base models converged in quality, providers competed on capabilities and shaped their APIs around their own products (Why it’s rational). Server-side tool loops, signed reasoning and long-running agents each need fields and states that a shared schema would first have to agree on. I don’t see a reason for any provider to wait for that agreement while it competes on these features.

The closest attempt is Open Responses, a specification derived from OpenAI’s Responses API. It was published in January 2026 and handed to the Agentic AI Foundation in October 2026. It standardizes the request and response format, while hosted tools remain vendor extensions. Neither Anthropic nor Google implements it natively, and Google calls its own Interactions API its primary interface. I expect Open Responses to become the common format for gateways and open-weights model servers, while the large providers keep shipping new capabilities on their own APIs first.

What still works on Chat Completions #

Chat Completions hasn’t gone away. OpenAI’s migration guide states that it “remains supported”, and every provider and gateway still accepts it in some form. What it supports has shrunk, though: text generation still works with most models, tool calls while the model reasons and prompt caching work only with limits, and hosted tools and server-side state mostly need the native APIs.

What Chat Completions still covers

OpenAI's own endpoint and the adapters from Google and Anthropic, compared with each provider's native API.

OpenAIown endpoint
native: Responses
Googleadapter
native: Gemini, Interactions
Anthropicadapter
native: Messages
WORKS ON CHAT COMPLETIONS
Text generationText in, text out
GPT and o-series
Gemini models
all Claude models
LIMITED ON CHAT COMPLETIONS
Tool calls while the model reasonsThe agent case
only with reasoning offFrom GPT-5.4, tools on Chat Completions need reasoning_effort: "none".none on GPT-6 Astra and GPT-6.1 Sol
thought signatures requiredGemini 3 and later reject a tool loop whose thought signatures aren't sent back.
thinking not returnedThe response doesn't include the thinking, so the next request can't send it back.
Prompt caching
less effectiveOpenAI reports 40% to 80% better cache use on Responses (native API) in internal tests.
explicit only
not supported
ONLY IN THE NATIVE APIS where providers add new capabilities
Models
Pro and Codex
agents, Omni video
every model is in both
Hosted tools
file search, code interpreter, MCP
search, code execution, URL context
web search, code execution
Server-side conversation state
stored responses
stored interactions
message threads (beta)The API keeps the conversation for a short time, so each request sends only what's new.

Providers add new capabilities to the native APIs. OpenAI recommends Responses for all new projects, and xAI, one of the providers that adopted the format, now marks its Chat Completions endpoint as deprecated.

What I’d do #

What Chat Completions costs: for the latest models, it costs features and response quality. From GPT-5.4, OpenAI’s models accept tools there only with reasoning switched off, and GPT-6 Astra and GPT-6.1 Sol not at all. Anthropic’s adapter never returns the thinking, so the next request can’t send it back, and the model continues without its earlier reasoning. OpenAI’s own numbers point the same way: according to its migration guide, reasoning models on Responses show “better model intelligence” and use the cache 40% to 80% better in its internal tests. Most hosted tools and all server-side conversation state need the native APIs. Where Chat Completions keeps the reasoning, it uses extension fields, such as Google’s thought signatures or the reasoning_details field of OpenRouter and Bifrost, and only clients that know the extension send it back.

Recommended: one native API per provider. Send each request in the format of the provider whose model answers it: Responses for OpenAI, Messages for Anthropic and Interactions for Google. A framework with one model class per provider does this for you, such as Pydantic AI with OpenAIResponsesModel and AnthropicModel. Keep the reasoning parts of each response in the history exactly as they were returned, and send them back with the next request.

What I wouldn’t do:

  • Run agents through a compatibility adapter. Google and Anthropic treat theirs as a secondary option, and Anthropic’s never returns the thinking.
  • Edit or drop reasoning blocks to save tokens. A changed block fails on Anthropic’s API, and a dropped one costs the model its earlier reasoning.
  • Rewrite earlier turns, instructions or tool definitions during a tool loop, for example to shorten old tool results. Anthropic and OpenAI then reject or drop the reasoning that follows.
  • Take a normal response as proof that the model received its reasoning.

How to measure: for each model, check which endpoint your SDK, framework or gateway actually calls. To check whether the model receives its reasoning, send the second request of a tool loop once with the thinking block and once without it, and compare the input tokens. Send both through the same route as your application, gateway included. Anthropic’s token counting endpoint gives the count without running the model, but it rejects requests with server tools, and a direct count doesn’t show what a gateway changes. If both counts are equal, the reasoning doesn’t reach the model. On OpenAI’s Responses API, input_transformations in the response lists the reasoning items it dropped.

For the broader picture, six common assumptions and what to check for each:

Six assumptions, checked against current behavior

Chat Completions is the industry standard.
partly
It's OpenAI's API. Several newer providers adopted it; Anthropic and Google kept their own APIs and added adapters later.Check: which providers you reach through an adapter instead of their own API.
Every major provider supports Chat Completions.
partly
Anthropic documents its adapter as intended for testing. Google recommends calling the Gemini API directly.Check: whether a production path depends on one of these adapters.
A call is messages and tool calls in, text out.
incomplete
The loop still works that way. Providers also run hosted tools inside the turn and return reasoning that has to be sent back.Check: whether your history keeps server tool results and reasoning blocks.
Reasoning is hidden and can be dropped.
no
Gemini 3 and later reject tool loops without thought signatures. Anthropic requires thinking blocks within a tool-use turn and may strip or disable thinking without an error.Check: every store, serializer and gateway keeps provider fields unchanged.
Same endpoint, same behavior.
no
Claude 4.7 and later accept sampling parameters only at their defaults. Anthropic API version 2026-10-21 rejects them on every model, even at the defaults.Check: the current Anthropic SDK sends 2026-10-21 by default. If your requests set sampling parameters, remove them before upgrading.
Open Responses works with every provider.
no
Anthropic and Google don't implement it natively, and hosted tools are vendor extensions.Check: which of your providers you would actually reach through it.

Conclusion #

Every LLM call still runs on the loop that Chat Completions made popular. What changed is what has to travel through it. Chat Completions still handles plain generation, but with the latest models it rejects tool loops, quietly drops the model’s reasoning between tool calls, or returns it only in extension fields that every layer has to keep. For tool loops, use each provider’s native API. New capabilities will likely only arrive there.