AI Gateways Are Now Critical Infrastructure. Treat Them Like It.
Your AI gateway may look like plumbing. In practice, it is becoming one of the most privileged systems in the enterprise.
The gateway routes prompts to models, stores or retrieves provider credentials, applies usage policy, connects applications to retrieval systems, and often holds the configuration that decides which model can reach which data. That makes it a control plane, not a convenience layer.
A recent Microsoft Threat Intelligence report makes the point uncomfortably concrete. In investigations of compromised LiteLLM, RAGFlow, and Kestra environments, attackers pursued the same prize: credentials, persistence, and access to the systems behind the AI workload. In the LiteLLM case, Microsoft observed credential harvesting from the gateway runtime, collection from a backing database, and persistence activity. This is not a story about a model producing a bad answer. It is a story about an AI service becoming an enterprise beachhead.
For security leaders, the implication is simple: the organisation cannot govern AI agents only at the prompt, model, or user layer. It also has to govern the infrastructure that connects those layers.
Why the gateway has become the soft underbelly
Traditional application architecture spreads trust across services. AI architecture often compresses it.
A gateway or orchestration layer can have model-provider keys, tenant routing rules, virtual keys, database connection strings, tool credentials, and observability data in one runtime. A retrieval platform may see internal documents and embed access to data stores. A workflow system may trigger code, containers, or SaaS actions. Each component may be defensible in isolation. Together, they create a high-value operational choke point.
That is why the useful security question is no longer, "Is our model approved?" It is, "What can a compromise of the AI control plane reach?"
If the answer includes production data, privileged SaaS actions, reusable provider keys, or other systems' secrets, the gateway deserves the same engineering discipline as an identity provider, API management layer, or production deployment platform.
The mistake: treating AI infrastructure as a developer tool
Many organisations still deploy AI infrastructure through the developer-experience path: fast evaluation, a shared API key, a public management endpoint for convenience, then a production workload built around it. The controls mature later, usually after the system becomes useful.
That sequence is understandable. It is also backwards.
An exposed admin surface on a conventional internal tool is a problem. An exposed admin surface on an AI gateway can be a multiplier: it may expose access to several models, several applications, several databases, and the policy records that join them together. A stolen model-provider key is not merely a billing issue when it can be used to impersonate an approved service or reach a provider endpoint outside the expected path.
This is where Shadow AI reappears in a more technical form. The unsanctioned tool is not always a browser tab. It may be a small gateway, retrieval server, or orchestration workflow assembled by a capable team to ship an internal feature. Security may know the model vendor, but not the control points that make the model operational.
Three controls that should be non-negotiable
1. Inventory the control plane, not just the models
Your AI inventory should include every gateway, retrieval platform, agent runtime, orchestration service, plugin server, and workflow engine. For each, record the owner, environment, internet exposure, data sources, model providers, credentials held, and actions it can initiate.
Do not settle for a spreadsheet of approved model names. That list cannot answer whether a forgotten test gateway has a production key or whether a retrieval service can read a sensitive document store.
Start with evidence: cloud assets, container registries, Kubernetes namespaces, DNS records, source repositories, identity logs, and network telemetry. The aim is a living map of the systems that route or execute AI work.
2. Make credentials narrow, short-lived, and separable
A gateway should not need a single permanent credential that can do everything. Separate provider access from database access, separate read paths from administrative paths, and use scoped identities wherever the platform allows it. Rotate credentials and ensure a compromised gateway cannot silently mint broader access.
This is especially important for agentic workflows. An agent that can call tools needs its own constrained identity and an explicit action boundary. Do not let a shared service principal become a universal permission slip for every agent a team experiments with.
The practical test is blunt: if a gateway container is compromised today, which secrets can it read, and what can those secrets do tomorrow? If the answer is unclear, the access design is not ready.
3. Monitor AI infrastructure as an execution environment
Logging prompts is useful, but it is not enough. Security teams need telemetry that can show unusual process launches, secret access, outbound traffic, changes to model routing, creation of virtual keys, and abnormal tool calls from the gateway and orchestration layers.
Microsoft's observed activity is a helpful model for detections: gateway-originated shell or Python execution, reads of process environment variables, database access outside normal application behavior, service-account key changes, and unexpected egress should all be high-signal events.
The owner of the AI platform should be able to answer three questions quickly: what changed, who authorised it, and what did it reach? If an incident response team cannot reconstruct those answers, governance is theoretical.
Governance is the operating model that keeps controls alive
A policy that says "use approved AI" will not protect an unowned gateway. The policy has to be paired with a path that makes safe deployment easier than unsanctioned deployment.
That means a lightweight intake for new AI services, reusable secure deployment patterns, a named owner for each workload, and reviews triggered by changes in data sensitivity or agent permissions. It also means giving security a way to discover AI usage without turning every experiment into a months-long approval queue.
The goal is not to block teams from building. It is to stop critical trust from accumulating in systems nobody can see or defend.
The enterprises that handle this well will treat AI infrastructure as first-class infrastructure. They will know where the control planes are, minimise the authority each one holds, and watch them with the same seriousness they apply to identity and production APIs.
That is the operational definition of AI governance: not a set of intentions about responsible AI, but the ability to prove which systems can access data, take action, and change the rules.
Aona helps security teams discover the AI systems operating across the enterprise, understand their data and access exposure, and put governance around the tools employees and teams are already using. If your AI inventory stops at model names, it is time to map the control plane.
