The State of AI Gateways in 2026

Executive Summary

The AI gateway is changing. What began as a shared proxy for authentication, model routing, rate limits, cost control, and observability is becoming a broader enforcement point for agentic systems.

Agents do more than exchange prompts with models. They discover tools, call APIs, query enterprise data, delegate work, respond to events, and take actions with business consequences. Adoption is advancing faster than governance: McKinsey's 2025 survey found that 62% of organizations were at least experimenting with agents and 23% were scaling one somewhere in the enterprise, while Deloitte reported that only 21% had a mature agentic-AI governance model.

The market is responding. Microsoft, Google, Kong, Envoy AI Gateway, agentgateway, and others are extending control beyond model requests to MCP tools, APIs, agents, and data. The shift is from governing which model receives a prompt to governing everything an agent can touch.

For enterprise architects, this creates three linked decisions. They must define the gateway's scope, choose whether to build, buy, adopt open source, or combine those approaches, and decide whether request-level controls are sufficient. A contract renewal, refund, procurement task, or incident response may span multiple agents, tools, systems, approvals, and days. No single request represents the whole activity; the meaningful governance object is increasingly the business transaction.

That suggests a longer-term progression:

The gateway remains essential, but as the enforcement foundation beneath a runtime for identity, delegated authority, policy, approvals, durable state, reliability, observability, audit, and business outcomes.

From LLM Proxies to Agentic AI Gateways

The category's expansion becomes clearer when the original proxy architecture is read alongside the four generations that followed. Each stage widens both the resource boundary and the governance responsibility.

Same Name, but Different Architecture

Early AI applications followed a simple pattern: an application called a model and returned the response. As providers multiplied, the AI gateway centralized credentials, routing, limits, telemetry, cost allocation, and fallback:

Those functions remain important, but they are now widely available in products such as LiteLLM, Cloudflare AI Gateway, and Kong. The harder problem is the architecture around the model. An agent may discover an MCP tool, invoke an internal API, delegate to another agent, query data, wait for approval, and then update a system of record.

The interaction therefore looks less like Application → Model and more like:

The highest-risk resource may be a payment API, regulated dataset, or downstream agent rather than the model itself. Products from different architectural lineages are converging on this broader control problem.

Four Generations of the AI Gateway

The category has evolved through four broad stages, each widening the resource boundary under governance.

Generation One: The LLM Proxy

The first gateway centralized model connectivity: API keys, proxying, basic logs, simple limits, and one application endpoint. Its boundary was narrow because the application called a model and the gateway governed that call.

Generation Two: The Multi-Model Gateway

As provider portfolios expanded, gateways added normalization, routing, retries, fallback, load balancing, token accounting, budgets, quotas, caching, and model telemetry. Model access became shared infrastructure rather than repeated application integration.

Generation Three: The Enterprise AI Gateway

Production use brought identity, authorization, data-loss controls, content safety, policy, audit, networking, self-service access, and organizational cost controls. The gateway became part of enterprise security and governance, not just a developer convenience.

Generation Four: The Agentic AI Gateway

Agentic systems extend the boundary to MCP tools, APIs, agents, and data. Microsoft now describes governance across models, MCP servers, and A2A APIs; Google exposes existing APIs as MCP tools through Apigee; and Envoy AI Gateway combines multi-provider LLM access with tool routing, filtering, authorization, and observability.

The center of gravity is moving from governing the model request to governing the agent interaction.

The 2026 AI Gateway Landscape: Taxonomy and Capabilities

The 2026 market is defined by two overlapping questions: where each product lineage began and which capabilities are now baseline, advanced, or emerging.

What Does “AI Gateway” Mean in 2026?

There is no single definition. The market is easier to understand by looking at where each architectural lineage began.

LLM Gateways

LLM gateways center on provider abstraction, routing, resilience, token and spend controls, caching, guardrails, and model telemetry. Their natural unit of control is the model request. They remain valuable for heterogeneous model estates, but their design assumes that the primary governed resource is a model.

API Gateways Extended for AI

API-management platforms begin with identity, authorization, traffic policy, lifecycle management, networking, portals, and observability. Microsoft, Google, and Kong are extending those foundations to models, MCP, and agent traffic. The open question is whether request-oriented API abstractions can fully represent tool discovery, delegated authority, long-running activity, and agent-to-agent interaction.

Cloud and Edge AI Gateways

Cloud and edge platforms emphasize routing, availability, caching, resilience, latency, telemetry, and cost control. They are strong operational control points for inference, though their eventual role in governing broader agent actions remains unsettled.

MCP and Agent Gateways

This newer lineage begins with tool and agent connectivity. Arcade federates MCP tools while connecting access to end-user identity and downstream OAuth. Agentgateway proposes one data plane for HTTP, gRPC, LLM, MCP, and A2A traffic. Its premise is simple: the agent is the new client.

Unified AI Gateways and AI Data Planes

A fifth category is emerging from convergence. It treats models, tools, APIs, agents, and data as resources within one AI execution surface. Redpanda's Agentic Data Plane and agentgateway differ in implementation, but both point toward a common enforcement model above individual protocols.

The defining question is becoming less about which protocol a gateway carries and more about where the enterprise enforces control over AI-driven interactions.

The 2026 Capability Baseline

As baseline functions spread across commercial and open-source products, differentiation is moving outward from model traffic.

Model-Plane Capabilities

Multi-provider access, standardized APIs, routing, retries, fallback, budgets, limits, caching, and telemetry are increasingly expected. Semantic routing, prompt transformation, content filtering, and policy-driven provider selection remain advanced, but are unlikely to define the category alone.

Tool and Agent Capabilities

MCP is accelerating support for server federation, discovery, filtering, tool-level authorization, OAuth mediation, sessions, streaming, and agent connectivity. The gateway may now decide not only which model receives a prompt, but which tools an agent may discover and execute.

Enterprise-Control Capabilities

Production platforms must connect human, workload, and agent identity to credentials, authorization, DLP, audit, compliance, catalogs, configuration, cost allocation, and organizational quotas. The challenge is consistent enforcement across every reachable resource.

Data and Integration Capabilities

Agents depend on APIs, databases, SaaS systems, events, and real-time data. As AI moves from answering questions to participating in workflows, integration and data context become part of the governance boundary.

Architectural Camps and the Fragmentation Problem

The major architectural choices differ in where they place control. The fragmentation problem reveals what happens when those choices accumulate without a shared governance model.

Three Architectural Camps Are Emerging

The vendor landscape can be reduced to three philosophies distinguished primarily by where they draw the control boundary.

Camp One: LLM-Centric

This specialized model layer is simple and appropriate for model-oriented workloads. Its limitation is visibility: it may govern reasoning infrastructure while seeing little of the resulting actions.

Camp Two: API-Management-Centric

This path extends mature identity, policy, networking, lifecycle, and operations capabilities. It is operationally natural for many enterprises, though agent-native semantics may strain infrastructure designed around discrete requests.

Camp Three: Agent-Native or Unified

A unified design offers consistent identity, policy, telemetry, and traffic management across the interaction surface, but expands the gateway's operational responsibility. Enterprises must decide whether control should evolve from the LLM layer, from API management, or from a new agent-native plane.

The Three-Gateway Problem

Rapid adoption can produce a fragmented topology:

Each decision can be sound in isolation while the overall architecture loses context.

Identity Fragmentation

Consider:

SAP may see Alice, one of the agents, or a shared service identity. Authentication alone does not show which authority was delegated, how it was narrowed, or when it expires. Agentic systems need to preserve both identity and effective authority.

Policy Fragmentation

Model, tool, API, identity, data, and workflow policies may live in different systems. Each layer can be well controlled while the enterprise still cannot answer:

That is a policy-composition problem.

Observability Fragmentation

A model call, tool invocation, API write, data query, and approval may be recorded separately. Incident response should reconstruct one trace:

Operational Fragmentation

Every additional gateway can introduce another deployment, policy language, credential store, dashboard, upgrade cycle, logging pipeline, certificate lifecycle, and on-call surface. Separation may be justified by security or ownership boundaries; the risk is unintentional fragmentation. The objective is one coherent governance model, even when enforcement remains distributed.

Build vs. Buy vs. Open Source

Acquisition strategy and economics are inseparable. The relevant decision is not only which product or codebase to adopt, but which operational responsibilities the enterprise is prepared to own over time.

Build vs. Buy vs. Open Source

The acquisition decision is an architectural commitment because each option relocates responsibility rather than eliminating it.

Build

A custom gateway often begins as a useful proxy with authentication, routing, and logs. Production needs quickly add providers, fallback, streaming, budgets, DLP, MCP, OAuth, tool policy, audit, high availability, dashboards, patching, and protocol upgrades. The organization discovers that it has built a product, not merely a proxy.

Forrester's 2025 predictions warned that three out of four firms attempting ambitious agentic architectures themselves would fail. That forecast is not measured failure-rate data, but it captures the central risk: complexity rises sharply between a useful prototype and an enterprise platform.

Building is rational when the infrastructure creates strategic differentiation, requirements are unusually specialized, or a strong platform team intentionally wants to own the product. Otherwise, prototype speed can conceal long-term product and operations work.

Open Source

Open source offers transparency, extensibility, self-hosting, and rapid access to emerging protocols. It avoids creating the platform from first principles, but the enterprise may still own deployment, IAM integration, upgrades, compatibility testing, vulnerability response, support, and project risk. Supported open source can provide a middle ground.

Buy

Commercial products can accelerate deployment and transfer parts of support, security response, upgrades, compliance, and high-availability operations. The tradeoff is dependency on the vendor's architecture and definition of the category. A strong LLM proxy may be too narrow as governance expands; an MCP-first platform may be too specialized in another direction. The buying question is whether the architecture can evolve, not only whether today's feature list is complete.

The Hybrid Reality

Many deployments combine approaches: an open-source core with support, a commercial control plane with a self-hosted data plane, a commercial gateway with plugins, or an internal platform built on open source. Two questions organize the choice: how much differentiation is required, and how much infrastructure the enterprise wants to own.

Low differentiation and low ownership tend to favor buying. Low differentiation with high ownership favors open source. High differentiation with limited ownership favors an extensible commercial platform; high differentiation with high ownership may justify building or deeply extending open source. Acquisition strategy should follow architecture, not substitute for it.

The Economics Are Bigger Than License Cost

License fees versus engineering salaries is the wrong comparison. A custom gateway carries development, integrations, provider and protocol maintenance, security reviews, on-call work, upgrades, infrastructure, and documentation. Open source still requires deployment, patching, compatibility testing, security ownership, and operational expertise. Commercial products add subscriptions, integration, premium features, services, and switching costs.

These economics matter as organizations demand clearer AI returns. Gartner predicts that more than 40% of agentic-AI projects will be canceled by the end of 2027 because of rising costs, unclear value, or inadequate risk controls. Forrester's 2026 predictions reported that only 15% of surveyed AI decision-makers had seen an EBITDA lift in the preceding year and forecast that enterprises would defer 25% of planned AI spending into 2027.

The useful cost model is:

Evaluating Gateways for Action Governance

Feature lists reveal less than architecture. Evaluation should test protocol adaptability, identity and authority, policy scope, observability, and the platform's ability to govern consequential actions.

How to Evaluate an AI Gateway in 2026

Feature counts can obscure the assumptions that matter. Evaluation should begin with architecture.

Protocol Architecture

Determine which protocols are genuinely first-class, whether model, MCP, A2A, HTTP, gRPC, and streaming traffic can coexist, whether APIs can become governed tools, and whether new protocols can be added without replacing the data plane.

Identity

The platform should distinguish human, application, workload, agent, and downstream identities. More importantly, it should preserve attribution to the original principal through autonomous delegations.

Authorization

Policy should reach models, individual tools, arguments, APIs, datasets, agents, destinations, and side effects. The meaningful test is not merely whether Alice may access finance, but whether an agent acting for Alice may create a purchase order below a threshold without approving or paying it.

Governance

Look for one consistent model across policy, configuration, audit, catalogs, developer access, and security operations, rather than unrelated subsystems under one product name.

Observability

Telemetry should answer not only which model was called, but what the agent did afterward:

Extensibility

Protocols, models, frameworks, and controls will change. Plugins, policy hooks, authentication extensions, transformations, custom providers, and telemetry integrations should therefore be treated as first-class architecture.

The Competitive Frontier Is Moving From Model Governance to Action Governance

The original question was, “Which model should handle this request?” The emerging question is, “What may this agent do on behalf of this user?” The second adds authority, context, data sensitivity, tool risk, approvals, audit, rollback, and ownership.

Governance must increasingly control what AI is permitted to cause. A hallucinated recommendation and an agent placing an order are different risk classes. Reads may execute automatically; writes may require stronger validation, audit, and idempotency; irreversible or high-consequence actions may require elevated authority, approval, checkpoints, rollback, or circuit breakers.

Gartner's May 2026 guidance reflects this shift by recommending governance proportional to autonomy, from read-only agents through approval-bound systems to fully autonomous agents. At the highest level, continuous monitoring, enforced guardrails, rapid rollback, circuit breakers, and clear ownership become essential because action can outpace human oversight.

This is where gatewaying begins to overlap with runtime governance for autonomous software.

The Business Transaction as the Unit of Governance

Request-level enforcement remains necessary, but autonomous business activity spans many interactions and often persists across hours or days. That makes the transaction (not the individual call) the more meaningful governance object.

The AI Gateway Is Necessary, but Not Sufficient

Consider: “Renew the Acme contract for another year at no more than a 7% discount and send it for signature.” Completing it may involve Salesforce, models, pricing and legal tools, another agent, executive approval, DocuSign, and two days of elapsed time.

No single HTTP request, MCP call, model trace, or A2A exchange represents the activity. The whole sequence is one enterprise transaction.

A gateway can authorize each step yet miss the larger context: Alice initiated a renewal, the discount is capped, VP approval is pending, AI spend is limited, and the task has a deadline. Those facts change policy, audit, recovery, and the meaning of success.

From Request Governance to Transaction Governance

A transaction-oriented architecture makes the business activity a durable object. It can carry an ID, initiator, intent, state, deadline, delegation chain, models and tools used, data classifications, budget, policy decisions, approvals, cost, and final outcome.

Unlike a request, the transaction survives model failures, process restarts, expired credentials, deployment changes, human delays, and periods with no active compute. The gateway governs each interaction; the transaction layer preserves why it exists, how it relates to prior steps, and what outcome it is meant to produce. The gateway becomes an insertion point for a broader governance runtime.

Authority, Identity, and Policy

Once transactions become first-class, the control plane must separate reasoning from permission, preserve delegated authority, and apply policy across models, tools, agents, data, and side effects.

Reasoning Is Not Authority

An agent deciding that something should happen does not mean it may make it happen. “Transfer $100,000” is a proposal, not an authorization.

This boundary separates intelligence from trust. The model, framework, and individual MCP server should not be the final security boundary for business action. Agents can reason wherever they are effective, while consequential execution passes through an independent trusted control point.

Identity Must Become an Authority Chain

Identity asks who an actor is. Agentic governance must also ask whose authority it carries, for what purpose, and within what limits:

Passing only user = Alice grants too little context and may imply too much power. The purchasing agent might be allowed to request quotes and create a purchase order up to $25,000, but not approve it or release payment. Its authority should expire with the transaction.

Each delegation may narrow authority but should never expand it:

Policy Must Span Models, Tools, Agents, and Data

Governance is often split among AI gateways, MCP gateways, IAM, API gateways, data platforms, and application code. That division becomes difficult when one autonomous transaction crosses them all.

Policy context may need to combine user + agent + data + intent + action + risk + transaction. It can then express controls such as approved models for restricted data, finance approval for high-value payments, DLP before external sends, transaction-level AI budgets, and limits on external writes.

The key is semantic context. A call named createPayment should also be understood as financial, irreversible, high risk, and valued at a specific amount. Policy can then govern classes of action rather than every tool implementation separately.

Durable Execution, Reliability, and Accountability

A transaction-aware runtime must remain safe and recoverable while work is paused, resumed, retried, event-driven, or partially complete. Approval, reliability, observability, and accountability therefore become execution semantics.

Human Approval Becomes Part of the Runtime

For consequential systems, approval should be an execution state rather than ad hoc application logic:

Revalidation matters because identity, price, contracts, credentials, classifications, or budgets may change while the system waits. Approval should not replay an old request; it should resume only after authority and assumptions are checked again. Human oversight therefore becomes part of execution semantics.

Reliability Moves From Requests to Outcomes

A gateway can recover from a model timeout:

A transaction runtime must recover from partial business completion:

The correct response may be a retry, alternate processor, finance pause, compensation, timeout, escalation, or operator intervention. That requires knowledge of the whole activity. Reliability therefore expands from request fallback to step retries, alternate paths, compensation, idempotency, and outcome-level recovery.

Observability Becomes Accountability

Model telemetry explains provider, latency, tokens, prompts, and cost. Agent governance must also show who initiated the activity, which agents received authority, which models influenced decisions, which tools and data were used, what policies applied, who approved actions, what the full transaction cost, and what outcome resulted.

That record can serve engineering, security, compliance, finance, operations, and business owners. Over time it becomes an agent flight recorder containing the user, agent and prompt versions, model, tool schema, policy version, arguments, authorization, approval state, and executed action.

The goal is not to replay irreversible effects, but to reconstruct enough of the decision environment to identify whether failure arose from the model, prompt, policy, authority, agent logic, tool definition, or upstream service.

Events and Real-Time Data Become First-Class Inputs

Agents will increasingly start or resume work from events such as failed payments, closed opportunities, vulnerabilities, incidents, schedules, and human instructions.

This replaces the simpler Prompt → Model → Response pattern. Real-time context can change a transaction and its applicable policy, expanding governance across models, tools, APIs, agents, identity, events, and data.

The AI Governance Stack and Coherent Unification

The resulting architecture is a layered governance stack: enterprise resources at the base, a gateway enforcing access, a transaction runtime carrying durable context, and business outcomes at the top. Physical unification is optional; coherent governance is not.

A Broader AI Governance Stack

The architectural progression can be represented as a stack, with each layer answering a different governance question.

Business Outcomes

The top layer represents what the enterprise values: renewals, refunds, procurement, onboarding, and incident resolution.

Transaction Governance

This layer preserves intent, state, delegated authority, approvals, budgets, reliability, audit, and replay across the complete activity.

AI Gateway

The gateway authenticates and authorizes actors, routes and mediates traffic, handles credentials, enforces policy, and records interactions.

Enterprise Resources

Models, MCP tools, APIs, agents, databases, events, applications, and real-time data form the execution base.

The stack preserves the gateway as the enforcement foundation while adding durable context above it. Forrester's emerging agent-control-plane category supports a similar separation between systems that build agents, systems that orchestrate work, and an independent plane that supervises heterogeneous agents. Deloitte's finding that only 21% of surveyed organizations have mature agentic-AI governance helps explain why this layer is emerging.

The architecture follows the risk: as autonomous systems act across resources and time, governance must extend from access to outcomes.

When Unification Is (and Is Not) the Right Answer

Unification does not require replacing every gateway with one product. Separate enforcement points may remain appropriate across teams, security domains, established API programs, narrow workloads, or different performance profiles.

The better objective is one coherent governance model across however many enforcement points the architecture requires. Unification becomes more compelling when agents cross models, tools, APIs, and data; identity and policy must persist end to end; multiple agents participate; or enterprise-wide audit, incident reconstruction, and operational efficiency matter.

Enterprise Recommendations and the Road to an AI Governance Runtime

The practical task is to choose principles that survive rapid market change while preparing for a broader runtime that governs complete agent-driven activity.

Recommendations for Enterprise Architects

The market will continue to change, so evaluation principles should outlast individual products.

1. 01

   Define the Boundary Before Selecting the Product

   Decide whether the gateway governs models, models and tools, the complete agent interaction surface, or eventually full transactions. Otherwise, products built for different problems will be compared as peers.

2. 02

   Follow Identity End to End

   Every consequential action should remain attributable to the human or workload that caused it, with authority narrowed and recorded at each delegation.

3. 03

   Treat MCP as Architectural, Not Merely as Another Connector

   MCP changes discovery, identity, credentials, sessions, authorization, policy, and audit. It is not simply another backend integration.

4. 04

   Evaluate Action Governance, Not Just Model Governance

   Test reads, writes, financial operations, external communications, production changes, and irreversible effects. The highest risk may appear several calls after the model response.

5. 05

   Decide What You Actually Want to Own

   Choose build, open source, commercial software, or a hybrid based on differentiation, platform capability, operational ownership, extensibility, protocol risk, and total economics.

6. 06

   Optimize for Change

   Models, protocols, frameworks, and standards will evolve. Favor an enforcement plane that can absorb new resources without being rebuilt.

7. 07

   Plan for Durable Activity

   Tasks lasting hours or days need persistent state, approvals, timers, events, retries, and recovery.

8. 08

   Design Observability Around Outcomes

   Trace complete business activity, not model calls alone, so technical and business stakeholders can investigate the same transaction.

9. 09

   Keep Governance Independent of the Agent Framework

   Governance should survive changes in model, runtime, framework, cloud, and tool implementation.

From AI Gateway to AI Governance Runtime

The category is an architectural boundary in motion. The LLM Gateway governs model access and economics. The AI Gateway adds enterprise controls. The Agent Gateway extends to tools, MCP, and agent connectivity. The Unified AI Gateway applies a common enforcement model across models, tools, APIs, agents, and data.

The next step may be Transaction Governance, where intent, authority, durable state, approvals, reliability, and outcomes become first-class concerns, followed by an AI Governance Runtime for policy, execution, security, observability, cost, audit, and forensic reconstruction.

This is a hypothesis rather than an inevitability. Gartner predicts that by 2028, 33% of enterprise software applications will include agentic-AI capabilities, up from less than 1% in 2024, and that at least 15% of day-to-day work decisions will be made autonomously. Its 2026 research also anticipates greater demand for platforms organized around workflow outcomes, especially in approval-heavy and timing-sensitive processes.

If agents make more consequential decisions over longer periods, request governance alone will not be enough.

Conclusion: The Gateway Is the Foundation, Not the Destination

The first AI gateways answered how applications could connect safely to multiple models. The 2026 question is how autonomous software can connect safely to the enterprise.

That requires governance across models, tools, APIs, agents, events, and data; identity that carries delegated authority; controls over actions rather than prompts alone; and visibility across systems that were historically separate. Enterprises must define the gateway's scope, choose a specialized or unified architecture, and decide which responsibilities to build, operate, or transfer.

A fourth question now sits behind those decisions: what happens when the governed object is a business transaction rather than a request? A gateway can authorize a model call or tool invocation, but the business ultimately asks whether the agent safely accomplished what it was authorized to accomplish.

Answering that requires a durable transaction graph and policy history connecting the human, agent, model, tool, data, decision, approval, action, failure, and outcome. From that record come stronger risk scoring, anomaly detection, cost control, incident reconstruction, and forensic analysis.

The AI gateway will remain a critical control point. But its lasting role may be as the enforcement layer beneath a broader runtime for identity, authority, policy, approvals, reliability, observability, audit, and business outcomes. The first generation governed model traffic. The next governs agent actions. The infrastructure that follows may govern complete enterprise transactions.