One Gateway, Every Dialect: Why Zilla Doesn’t Make You Choose Between Governance and Fidelity

Your LLM gateway may be quietly changing your requests. Not on the easy route, where every request already looks like OpenAI’s, but as soon as an agent falls back to another provider, or a developer brings in a tool that speaks Anthropic’s API. Settings go missing and costs creep up. The usual fix is a passthrough route, but that bypasses your policy too. In this post we walk through where that happens and why, and then show how we designed Zilla’s LLM gateway so you don’t have to choose.

One agent, three routes

Most teams put an LLM gateway between their applications and their model providers for the same reasons they put an API gateway in front of their services. It holds the provider credentials, so your applications don’t have to. It routes requests between models and providers, and falls back when one is having a bad day. It applies your guardrails to prompts and responses. And it records who used which model, and what it cost. All of that only works if every request goes through the gateway, which is why it matters what the gateway does to those requests along the way.

That gets harder once your teams speak more than one model API, and most do. The typical LLM gateway grew up around the first one, OpenAI’s. Follow one agent as its needs grow, and you can see where that starts to hurt.

OpenAI to OpenAI

Your agent uses the OpenAI SDK and runs on GPT. It asks for low reasoning effort to keep routine steps fast and cheap:

client = openai.OpenAI(base_url="https://llm-gateway.internal/v1")

client.chat.completions.create(
    model="gpt-5.6",
    reasoning_effort="low",
    messages=[...],
)

Every gateway handles this well. Most gateways convert every request into OpenAI’s format internally, so when your agent speaks that format too, there is nothing to translate, apart from smoothing over the quirks of “OpenAI-compatible” servers that turn out to be not quite as compatible as advertised.

OpenAI to Claude

Your platform team adds Claude on Amazon Bedrock as a fallback. Now the gateway has to translate out of its internal OpenAI format into Bedrock’s API, and the effort setting has to become something Claude understands.

This is where translation starts to buckle. Concepts that several APIs share under different names have to be recognised and mapped, and a gateway that rebuilds the request from the fields it knows simply leaves the rest behind. Here is what your agent’s effort setting looks like when it reaches Bedrock’s Converse API:

{
  "messages": [ … ],
  "inferenceConfig": { "maxTokens": 16000 }
}

Two popular gateways build the Converse request from a fixed list of fields, and reasoning_effort isn’t on either list, so it never arrives. The call still succeeds, and nothing tells your agent that it changed.

Claude to Claude

Then your developers adopt Claude Code, which speaks Anthropic’s API rather than OpenAI’s. Now even Claude-to-Claude traffic is converted into the gateway’s internal OpenAI format and back, and it can come out the other side changed.

One gateway joins Claude’s system prompt blocks into a single string, which removes their caching markers. Another leaves the effort setting out when it converts the request for Bedrock, and a third has a documented Bedrock setup that strips it:

{
  "model": "claude-sonnet-5-5",
  "max_tokens": 16000,
  "thinking": { "type": "adaptive" },
  "output_config": { "effort": "low" },
  "messages": [ … ]
}

{
  "messages": [ … ],
  "inferenceConfig": { "maxTokens": 16000 },
  "additionalModelRequestFields": {
    "thinking": { "type": "adaptive" }
  }
}

Compare the two:

• The format changed. The request was rewritten into Bedrock’s Converse format, even though Bedrock accepts Claude’s own Messages format directly.
• Some fields moved. max_tokens became inferenceConfig.maxTokens, and thinking moved under additionalModelRequestFields. That’s harmless reshuffling.
• The effort setting is gone. Nothing in the Converse request carries output_config.effort, so Claude never sees it.

The usual fix is a passthrough route that forwards requests untouched. You keep the provider’s features, but you typically give up what you deployed the gateway for in the first place: guardrails, fallback, load balancing and consistent telemetry. You end up choosing, route by route, between governance and fidelity, and the line between them moves with every release.

Losing effort isn’t cosmetic. Thinking tokens are billed as output, and in our own test a hard prompt used about six times as many tokens at max effort as at low. And the cost doesn’t fall on the same people at each step:

Zilla’s approach: every dialect, first-class

Zilla also uses a common shape for the concepts every model API shares, such as messages, tools, tool calls and media. The difference is what Zilla does with everything else.

Here is the shape of it in zilla.yaml:

bindings:
  llm_server:
    type: llm
    kind: server
    exit: llm_proxy

  llm_proxy:
    type: llm
    kind: proxy
    options:
      guardrails:
        ...
    routes:
      - exit: llm_client_openai
        ...
      - exit: llm_client_bedrock
        ...

  llm_client_openai:
    type: llm
    kind: client
    options:
      dialect: openai
      server: https://api.openai.com
    exit: sys:http_client

  llm_client_bedrock:
    type: llm
    kind: client
    options:
      dialect: aws-bedrock
      server: https://bedrock-runtime.us-east-1.amazonaws.com
    exit: sys:http_client

The LLM gateway is three bindings in a pipeline:

• llm server speaks the client’s dialect and converts it into common sections.
• llm proxy sits in the middle, where routing, fallback and guardrails operate on those sections.
• llm client converts the sections into the provider’s dialect.

Zilla does not discard provider-specific details that its common format can’t hold. They travel alongside the common sections, in three tiers:

Check: carried exactly or translated. Partial: translated where the target supports it. X: dropped.

This gives you two properties that matter. When the client and the provider speak the same dialect, nothing is lost: an Anthropic client calling Claude gets byte-faithful requests and responses, including caching markers and thinking signatures. When they speak different dialects, Zilla maps shared concepts wherever the target can express them, and drops only what has no equivalent. As we promote more shared concepts into the x: tier, cross-dialect translation becomes more faithful without any cost to same-dialect fidelity.

The same three routes, through Zilla

OpenAI to OpenAI. Still nothing to translate, and nothing changes.

OpenAI to Claude. Zilla recognises effort as a concept that OpenAI and Claude share, and maps it across, so after a fallback Bedrock’s Converse API receives the setting your agent asked for:

{
  "messages": [ … ],
  "inferenceConfig": { "maxTokens": 16000 },
  "outputConfig": { "effort": "low" }
}

Claude to Claude. Claude Code and Claude on Bedrock speak the same dialect, so Zilla treats the route as native. Only the envelope changes, the authentication, the URL and the API version, and the request reaches Bedrock exactly as Claude Code sent it:

{
  "anthropic_version": "bedrock-2023-05-31",
  "max_tokens": 16000,
  "thinking": { "type": "adaptive" },
  "output_config": { "effort": "low" },
  "messages": [ … ]
}

All three routes go through the same proxy, under the same policy. There is no passthrough to fall back to, because you don’t need one.

Streaming, any dialect to any dialect

Streaming is where translation layers tend to show their seams. Each API streams differently, with different event names, different ways of sending tool-call arguments in fragments, and different places to report token usage. Converting a finished response is straightforward. Converting a live stream event by event, without buffering it and without breaking a tool call halfway through, is considerably harder.

If you have read How Zilla Works, the shape of our LLM gateway will look familiar. Every Zilla protocol pipeline is a routed graph of bindings, and streams of data flow between those bindings over shared memory, one engine worker per CPU core. Whatever the protocol, every stream has the same simple shape, with built-in flow control so a fast sender never overwhelms a slow receiver. The engine also avoids object allocations on the hot path, so latency stays low and consistent even under heavy load.

We built the LLM gateway on exactly the same foundation. Model traffic is streaming traffic, often long-lived and token by token, so a gateway that buffers whole responses works against the grain of the workload. In Zilla, a model response is just another stream, flowing through the gateway as fast as your client can consume it.

In Zilla, the llm server and client bindings absorb those differences. The proxy in the middle observes the same guardrail inputs whether the client is streaming or not; streaming guardrail inputs simply arrive in parts. This is the same internal fragmentation that lets large messages share an engine worker fairly with small ones, applied to model traffic.

That design gives you combinations that are hard to achieve any other way:

• Cross-dialect streaming in every direction, including tool-call argument fragments.
• Streaming in, non-streaming out, so a streaming client can use a backend that only returns whole responses.
• Non-streaming in, streaming out, so a request-response client can still benefit from a streaming backend.

For the people using your application, this is the difference between text appearing as the model writes it and a spinner waiting for the whole answer. For an agent, it means tool calls can start sooner, and no buffer in the middle holds long responses hostage.

Every cloud, in both directions (Zilla Plus)

The open source Zilla Community Edition supports OpenAI (Chat Completions and Responses), Anthropic and Gemini as both the inbound API your application speaks and the outbound API Zilla calls. Zilla Plus adds AWS Bedrock, Google Vertex and Azure Foundry, in both directions.

Two aspects of Zilla Plus cloud support deserve a closer look.

The clouds count as native. When Claude runs on Bedrock, Vertex or Foundry, the request body is still essentially Anthropic Messages, wrapped in cloud-specific authentication, URLs and framing. Zilla Plus handles those platform differences behind the scenes and treats the route as same-dialect, so an Anthropic client reaching Claude on any of the three clouds keeps full fidelity. The same is true for OpenAI models on Azure and Gemini on Vertex.

Cloud SDKs work as clients. An agent written against the Bedrock, Vertex or Azure SDK can send its requests to Zilla Plus unchanged, and Zilla Plus can route them to an OpenAI, Anthropic or Gemini backend. Moving between models or providers no longer means rewriting the agent.

You get flexible credentials in both editions. Each route can forward the caller’s own API key, use a provider key held by the gateway, or derive access from the caller’s identity. Zilla Plus adds provider-specific authentication for the clouds, such as AWS STS.

Guardrails that apply everywhere

Because every request reaches the proxy as the same kind of section data, Zilla applies guardrails the same way however the request arrived. Your policy applies whether the route is same-dialect or cross-dialect, streaming or non-streaming or any mix of the two, and whichever client SDK sent it.

That consistency matters most where gateways are typically weakest. Guardrails on a streamed response can often only block it, rather than mask a secret or redact personal data as it flows past. Many gateways skip guardrails on passthrough routes entirely. Zilla gives you no ungoverned path to fall back to, so your policy never quietly stops applying.

For your platform and security teams, that turns a list of exceptions into a single rule: if traffic goes through Zilla, your policy applies.

What it means for the people using it

Architecture is only interesting because of what it changes for people, so let’s be concrete.

For end users, answers arrive sooner and are better. Streaming works end to end in every combination, so text and tool calls appear as the model produces them, and the gateway itself adds very little to that path. Features such as extended thinking survive the gateway instead of being stripped by it, and fallback to another model or provider keeps working mid-conversation, even across APIs.

For application and agent developers, the SDK you already know keeps working: OpenAI, Anthropic or Gemini, and with Zilla Plus the Bedrock, Vertex or Azure SDKs as well. Changing provider becomes a routing change rather than a code change, and the provider features you depend on still work through the gateway.

For platform and finance teams, prompt-caching markers survive, so your cached tokens stay cached and your provider bills them at the lower rate. You can buy capacity wherever the terms are best, and move workloads when those terms change. Zilla reports usage per caller and per model that actually answered, including after a fallback.

For security and compliance teams, one policy covers every path, with no passthrough exceptions and no streaming blind spots. Zilla is self-hosted, so your prompts and responses never pass through someone else’s service. And because telemetry reports the model that actually answered, your audits show every fallback to another model or provider.

How Zilla compares

Cross-dialect streaming is becoming common, but gateways still differ in which dialects they accept, what survives the trip, and whether your policy still applies along the way. The comparison below reflects each vendor’s public documentation as of October 2026. These products move quickly, so check current documentation before you rely on any particular row.

Check: supported. Partial: partial support. X: not supported. P: Zilla Plus.

To be fair to the alternatives, where they lead today, they lead on breadth. LiteLLM and Portkey route to far more providers and endpoint types, and Kong surrounds its AI features with a mature API management platform. Much of that breadth comes from OpenAI-compatible endpoints, which Zilla reaches through its OpenAI dialect as well. agentgateway is the closest to Zilla in approach: open source, translating between dialects, and applying guardrails to streams.

Governance and fidelity, together

Every LLM gateway converts between APIs through some common shape. The more interesting question is what it does with everything that common shape cannot hold.

Zilla keeps it. Same-dialect traffic stays exact, Zilla translates cross-dialect traffic as faithfully as the target allows, and the same guardrails govern every route, streaming or not. Underneath, it is the same streaming engine we have spent years tuning for low-latency, high-throughput protocol traffic, now carrying model traffic too.

We think you should not have to choose between a gateway that governs your models and a gateway that understands them. See it for yourself: the LLM Gateway get started guide runs the routes from this post, with the OpenAI and Anthropic SDKs, fallback and guardrails, on your own machine with Docker Compose. Then join our community and tell us which provider, SDK or feature you’d like us to add next!