Zilla for LLM
Secure and Govern Access to AI Models.
Give applications and agents governed access across LLMs. Keep your SDKs while Zilla authenticates callers, controls model access, applies guardrails, routes requests and tracks token usage.
The Challenge
Every new model adds another place to enforce security.
Provider-specific APIs, credentials, access rules and monitoring spread across applications as model adoption grows. Zilla gives you one enforcement point without adding policy logic to every SDK client.
Keep your SDKs
Point supported OpenAI and Anthropic clients at a new base URL.
Enforce in one place
Authenticate callers and apply role-aware model routes and guardrails.
See what is used
Track tokens, interactions and latency without per-app metrics code.
Governance
Enforce model governance at the gateway
Control who can call which models and what content gets through. Zilla checks identity, role-aware routes and configured prompt or response guardrails before it forwards traffic to a provider.
Authenticate every caller
Validate the inbound token once at the gateway, regardless of how many models and providers sit behind it.
Scope access by role
Guarded model routes reserve specific deployments for authorized callers.
Handle provider keys
Forward caller keys, or supply provider credentials at the gateway.
Inspect prompts and replies
Per-model checks for secret patterns, personal data and jailbreak attempts.
Try a model access policy
Request permitted by configured policy
The Standard role has a matching route to gpt-4o-mini, and the request passes the configured checks.
Model not permitted for this role
The Standard role has no authorized route to gpt-4o, so Zilla does not forward the request to the provider.
Prompt rejected before provider access
The configured user-text guardrail finds a pattern that resembles a secret or personal data and rejects the request at the gateway.
Request permitted by configured policy
The Premium role has a matching route to gpt-4o-mini, and the request passes the configured checks.
Request permitted by configured policy
The Premium role matches the guarded route to gpt-4o, and the request passes the configured checks.
Prompt rejected before provider access
The configured user-text guardrail finds a pattern that resembles a secret or personal data and rejects the request at the gateway.
Model Routing
Keep your SDK. Choose your model.
Point an existing SDK at Zilla and route by requested model, even when the target speaks a different supported API. An OpenAI client can reach Claude and get an OpenAI-format response back.
Route by model: Match model IDs or patterns, and optionally restrict routes by role.
Translate when needed: Convert supported messages, tool calls, errors and streaming events between API dialects.
Fall back when unreachable: Try a configured fallback when a deployment cannot be reached. Not a general retry policy for 429 or 5xx responses.
One SDK, several model targets
Returned in OpenAI-compatible Chat Completions format, including supported tool calls and streaming events.
Returned in Anthropic-compatible Messages format, with supported content, tool calls and streaming events.
Returned in OpenAI-compatible format. Zilla Plus signs Bedrock requests with its AWS credentials.
Observability
Token usage and latency, visible at the gateway
Measure LLM interactions, input and output tokens, request latency and time to first token per provider binding. Export supported metrics to Prometheus or OpenTelemetry without instrumenting each application.
From the Track Token Usage guide
How it Works
Configure once. Enforce consistently.
Zilla combines LLM server, proxy and provider client bindings in one declarative, streaming-native gateway. Supported client traffic needs no custom proxy service and no agent-side policy SDK.
Keep the SDK integration: Supported OpenAI or Anthropic SDKs connect to Zilla through its LLM server binding.
Define policies and routes: Configure model routing, caller authorization and guardrails in YAML.
Forward to the provider: Zilla translates supported requests and streams responses through the selected client binding.
Streaming-native
LLM requests and streaming replies run in Zilla's event-driven runtime.
Stateless data plane
Policy enforcement needs no gateway database on the request path.
Light to deploy
Run on Docker or Kubernetes and configure behavior declaratively.
Quickstart
See governed LLM access for yourself
The official LLM quickstart runs both SDKs, cross-provider translation, guardrail rejection, fallback and provider-level Grafana metrics.
Route
OpenAI SDK to Claude, and Anthropic SDK to GPT.
Protect
Reject a prompt that contains a secret or an email address.
Recover
See fallback when a deployment is unreachable.
Observe
Inspect tokens and latency in Grafana.
Govern every model call.
Run the LLM quickstart to see cross-provider routing, guardrails, and token metrics, or discuss how Zilla fits your security and integration requirements.
Join the Community
Trade notes with the engineers running Zilla in production, and get help from the team in Slack.
