Key Takeaways More enterprises are deploying enterprise-grade AI agents in private environments with controlled data, stable performance, and predictable costs. As AI moves from model calls to continuously running enterprise workloads, agent readiness introduces new requirements for enterprise IT infrastructure; as the pipeline involves model inference and agent runtime, each poses unique requirements on infrastructure capabilities.
|
Trend #1: Agents Turn AI from “Model Calls” into Continuously Running Enterprise Workloads
Enterprise AI is moving from “model calls” to “agent runtime.” An agent is essentially a model-driven software execution system that can complete tasks autonomously within defined constraints. As agents enter use cases such as knowledge management, R&D, customer service, data analytics, and operations, AI is shifting from one-off requests to continuously running enterprise workloads.
In its Enterprise AI Factory design guide, NVIDIA notes that agentic AI is moving from static pipelines to autonomous, stateful workflows. These agents coordinate multi-step work across tools, memory, and policies, and rely on file systems to persist state, skills to package reusable actions, and sandboxes to ensure execution safety.[1]
This shift is reshaping enterprise AI infrastructure requirements. As the number of enterprise agents grows, call chains become longer, and token usage rises, infrastructure pressure expands beyond model inference and GPUs to CPU, storage, networking, security, permission governance, and end-to-end observability.
AMD points out in Why is Everyone Talking About CPUs and Agentic AI? that orchestration, tool calls, inference requests, and instruction execution in agentic AI primarily happen on the CPU side. The research it cites shows that CPU-side tool processing and orchestration can account for 50% to 90% of end-to-end runtime in an agentic AI pipeline.[2]
Intel also states in Agentic AI Requires More CPUs that agentic AI turns inference from a single model call into a multi-step workflow similar to a distributed system. Orchestration, planning, branching, retries, multi-agent collaboration, and tool calls are mainly executed by CPUs. Its tests for financial document anomaly detection and code generation both show that CPU-side tasks can significantly affect overall pipeline efficiency. Intel also notes that real production systems typically use containers or VMs to run sandbox environments, which further adds CPU overhead.[3]
Trend #2: Agent Runtime Characteristics Make Private Deployment the Mainstream Approach
The value of enterprise agents comes from their continuous connections to enterprise knowledge bases, databases, business systems, MCP servers, tools, and model services. Accordingly, the context, working directories, files, intermediate results, and call records that agents generate make the security governance and data sovereignty of these agents top priorities for enterprise users. The massive token consumption required by model services for agent runtime is also accelerating the private deployment os AI.
VMware by Broadcom notes in Private Cloud Outlook 2026: The AI Tipping Point that when AI inference is embedded into applications, workflows, and business processes, enterprises place greater emphasis on production requirements such as cost predictability, data control, security, and performance. They also need more stable performance expectations and tighter governance than a public-cloud-only model can typically provide.[4]
In the HPE Private Cloud AI solution jointly released with NVIDIA, HPE states that Private Cloud AI can help enterprises deploy trusted, enterprise-grade agentic AI with stronger control, observability, and efficiency. Its capabilities include local agent registration, model/skill/tool approval, a unified model gateway, centralized governance and security policies, and optimization for token costs and GPU utilization.[5]
Together, these trends point to a practical infrastructure demand: enterprises need to run agents stably, securely, and at scale on private infrastructure.
The infrastructure must keep data within controlled boundaries and make agent access, operations, and call chains governable. It must support operational management of inference costs and GPU utilization, provide sufficient CPU resources for agent orchestration, tool calls, and sandbox execution, and centrally manage CPUs, GPUs, and AI accelerators across different architectures to fit heterogeneous compute environments. Enterprises also need to reuse existing VMs, Kubernetes, storage, networking, security, and operations systems to reduce duplicate construction and long-term management costs.
Challenges of Building Production-Grade Agent Infrastructure: Distinct Demands for Model Inference and Agent Runtime
Production-grade agent workflows can be divided into two typical stages: model inference and agent runtime. Model inference provides model services for agents and other AI applications, and depends more on GPUs, heterogeneous compute scheduling, model management, and service governance. Agent runtime hosts workloads such as the agent runtime itself, orchestration, MCP servers, tool services, business system connections, and context processing, and depends more on CPU, storage, networking, security isolation, and access governance.
The two scenarios are closely related and together determine whether agents can reliably enter production.
Infrastructure Challenges for Model Inference
When building model inference infrastructure, the challenge is not only whether GPU resources are sufficient, but also whether heterogeneous compute, model files, inference services, and call governance can be brought into a unified system.
As the number of models and AI applications increases, inference infrastructure can easily move from single-point validation to fragmented construction, creating pressure on resource utilization, model iteration, and service quality management.
Challenge #1: Fragmented Compute Resources Are Difficult to Schedule and Reuse
Different teams or vendors often introduce GPU servers, AI appliances, workstations, or standalone model services for specific applications. This can accelerate application rollout in the short term, but over time, it creates isolated resource pools. Compute resources cannot be centrally managed, models cannot be reused across applications, and operations and monitoring lack a global view, leading to duplicate construction and resource waste.
Challenge #2: Coexisting Heterogeneous Compute Resources Lack Fine-Grained Allocation
Production servers often use GPUs from different brands, generations, and architectures, and may also combine domestic compute resources with existing GPUs.
Compute requirements also vary significantly by model. Large models may require full GPUs, multiple GPUs, or even cross-node scheduling, while lightweight workloads such as small models, embedding models, and reranking models are better suited to GPU partitioning or sharing.
Without unified scheduling, quota management, and resource isolation, some compute resources may remain idle for long periods while others stay under constant pressure.
Challenge #3: Fragmented Model Files and Versions Affect Release Efficiency and Continuous Iteration
Model files are often large, come in many versions, and use different formats. Fragmented deployment can lead to repeated downloads, duplicate storage, and repeated release work. As model iteration accelerates, enterprises also need to flexibly access, switch, and update models on a stable foundation, avoiding tight coupling between upper-layer applications and a specific model, vendor, or appliance.
Challenge #4: Lack of Unified Inference Service Governance Can Lower Service Quality
When model inference enters production, enterprises need more than the ability to “run” models. They need to publish models as standardized services that are governable, observable, and operable. A unified entry point, authentication, routing, rate limiting, quotas, call logs, and usage statistics determine whether enterprises can clearly observe call volume, resource consumption, and chain latency across different applications and models.
Infrastructure Challenges for Agent Runtime
When building agent runtime infrastructure, enterprises face challenges caused by a fundamental change in workload patterns. Agents are no longer typical stateless applications. They continuously reason around tasks, call tools, access data, generate files, and accumulate context, execution traces, and intermediate results during multi-agent collaboration.
This means agent runtime infrastructure must provide not only compute resources, but also state persistence, security isolation, network access governance, and end-to-end observability.
Challenge #1: Stateful Runtime Creates Storage and Persistence Pressure
Agents are stateful, continuously executing workloads that maintain working directories, context, memory, files, and intermediate results during runtime. Long context, multi-round reasoning, and multi-agent collaboration further increase demand for KV cache, vector indexes, high-frequency reads and writes, and shared files. Rapid sandbox startup also depends on fast storage provisioning and configuration.
As a result, the agent runtime infrastructure requires diverse storage capabilities such as block storage, file storage, cache offloading, shared access, and permission control.
Challenge #2: Untrusted Execution Necessitates A Secure Sandbox
Agent execution is inherently uncertain. An agent may run model-generated code, shell commands, or scripts, and may also process input from web pages, documents, emails, and external systems. Different tasks have different risk levels, so the infrastructure must provide isolated environments of different strengths, from containers to MicroVMs and VMs.
This allows low-risk tasks to start quickly while giving high-risk code execution, external file handling, and sensitive data access stronger boundaries, limiting the blast radius and preserving audit evidence.
Challenge #3: Dynamic Access Paths Increase Network Security Governance Complexity
Agent accesses are highly dynamic. An agent may connect to model services, MCP servers, databases, code repositories, enterprise APIs, SaaS tools, and external web services at the same time. Large volumes of east-west traffic are also generated between agents, between agents and models, and between agents and toolchains.
The infrastructure needs to use workspaces as isolation domains, centrally manage boundaries across agent runtimes, controlled file systems, and isolated networks, and govern tool calls, external access, credential issuance, and policy decisions through gateways and policy components.
Challenge #4: Fragmented Chains Lead to Insufficient End-to-End Observability
An agent execution chain spans the agent runtime, sub-agents, model services, MCP servers, tool calls, databases, file systems, and external APIs. Logs from a single component are not enough to reconstruct what context an agent actually sent, which tools it called, what data it accessed, and how many resources it consumed.
Production-grade agents require end-to-end observability covering context, call chains, permission usage, file generation, model calls, token consumption, and chain latency, supporting troubleshooting, behavior auditing, and continuous optimization.
SmartX ECP: Evolving from Traditional Applications to the Agent Era with One Platform
Initially built on distributed storage and hyperconverged infrastructure, SmartX ECP helps enterprises create stable, high-performance, elastically scalable private infrastructure for traditional services, databases, and other critical applications.
With the introduction of SMTX Kubernetes Service (SKS), it integrated virtualization, containers, storage, networking, and security in one platform, enabling enterprises to manage virtualized and cloud-native workloads on the same infrastructure.
On this foundation, the platform further expanded with components for networking and security, disaster recovery, unified management, self-service, file storage, and AI, adding capabilities for heterogeneous compute management, model services, security governance, observability, and diverse storage for model inference and agent runtime.
The core value of this evolution path is that AI capabilities are not built as a separate new platform. Instead, they become an extension of existing cloud infrastructure capabilities, upgrading enterprise cloud from a simple resource foundation into a unified infrastructure that supports traditional applications, containerized applications, model inference, and agent runtime.
With SmartX ECP, users can modularly combine compute, storage, networking, security, containers, disaster recovery, and AI capabilities according to their business stage. This supports different workload types, reuses existing infrastructure investment, and reduces resource fragmentation and management complexity caused by running multiple platforms in parallel.
Flexible Product Modules Support Parallel Evolution of Model Inference and Agent Runtime
SmartX ECP does not rely on a single product to meet every requirement in AI deployment. Instead, it provides different capability combinations around model inference and agent runtime. The two scenarios can be hosted together on the same cloud infrastructure, or combined on demand according to the user’s existing resources and construction stage.
For model inference service, users can primarily build on SMTX AI Platform together with SKS to quickly create production-grade model infrastructure for different scenarios, ensuring efficient rollout and stable operation of model services:
- Model services: Centrally access private models and internal/external public models, supporting model management for DeepSeek, Qwen, GLM, and other models.
- High-performance inference: Integrate inference engines such as vLLM and SGLang to provide high-throughput, low-latency inference services for long-context and multi-turn conversation use cases.
- Model governance: Provide a built-in model gateway that unifies the entry point for internal and external model invocations. Usage statistics, quota management, access control, and access logs enable fine-grained management of every model call, making troubleshooting, security auditing, and resource governance easy to check.
- Resource management: Centrally manage AI servers across multiple regions and types, with built-in GPU virtualization to improve heterogeneous chip utilization.
- End-to-end observability: Cover key areas such as GPU hardware, inference instance health, model resource consumption, and call logs to help quickly locate issues, support security auditing, evaluate performance, and optimize resource usage.
For agent runtime, users can use SmartX ECP as the foundation to integrate VMs, containers, storage, network security, and agent-native file sharing capabilities into one architecture:
- Containers: Provide agents with a unified runtime environment and secure sandboxing capabilities to ensure reliable and secure operation.
- High-performance storage: Provide high-performance block storage and file storage to meet requirements for concurrent agent reads and writes, private context data storage, and cross-agent data sharing.
- Security: Use network traffic visualization to clearly present the call topology between agents and MCP services. Unified security policies across VMs and containers help secure communications between agents and between agents and MCP services.
- Agent-friendly: Provide native file sharing components for agent collaboration scenarios, better supporting multi-agent coordination and data flow.
Customer Stories: Extending from Enterprise Cloud Platform to Agent-Ready Infrastructure
SmartX ECP has long been adopted by enterprise users, making it a reusable foundation for building agent-ready infrastructure. For users already using SmartX ECP, agent and model inference capabilities do not need to be built from scratch. They can be extended gradually on top of existing compute, storage, networking, security, and operations systems.
The following practices correspond to key challenges such as agent runtime, model service governance, heterogeneous compute management, and hosting multiple types of model workloads.
User Practice #1: Intelligent Operations Based on Lightweight Cloud Infrastructure
Highlight: Existing infrastructure operates data, operations knowledge, and management workflows, and further introduces capabilities that allow agents to call, execute, and audit.
Details: In a large securities institution, SmartX has long supported the operation of its large-scale enterprise cloud infrastructure. As the infrastructure expanded, disk fault diagnosis and handling became a frequent and complex operation scenario. SmartX connected product operations skills to the agent-based intelligent analysis process for disk alert handling, log analysis, and fault diagnosis.
User Practice #2: Building a Complete Infrastructure for Agents and Model Services
Highlight: Agent runtime and model inference are hosted together, while a unified gateway and observability capabilities provide the foundation for production-grade AI governance.
Details: A manufacturer built on SmartX ECP and SMTX AI Platform to centrally support agent applications such as an internal policy query assistant, translation, and operations. The AI platform also enables unified management and governance of existing internal models and self-built models. Through unified compute management, model management, and observability, the user can clearly gain information about resource usage, model instance performance, and token consumption.
User Practice #3: Building a Unified Model-as-a-Service Platform
Highlight: Dispersed models, compute resources, and call chains are brought under unified governance, improving resource utilization and model service observability.
Details: A Grade-A tertiary general hospital built a unified Model-as-a-Service platform based on SMTX AI Platform, providing model services for clinical diagnosis and treatment, medical record writing, diagnostic quality control, medical Q&A, and other scenarios. With GPU virtualization and unified observability, the user can improve utilization of critical compute resources and reduce the complexity of troubleshooting and operations management for model services.
User Practice #4: Unified Delivery of Compute and Model Resources to Host Multiple Types of Model Workloads
Highlight: Compute resources and model resources are delivered together via a unified foundation, adapting to the different requirements of diverse model workloads for runtime environments, resource granularity, and service governance.
Details: A Grade-A tertiary general hospital uses SmartX ECP and SMTX AI Platform to host different types of model workloads, including small image recognition models and large language models. Compute resources are provided through VMs, containers, and GPU virtualization, while SMTX AI Platform enables unified management and rapid delivery of large language models, supporting medical AI use cases such as internal diagnosis, radiology reports, and AI-based quality control.
Making Agent Readiness the Default Standard for Enterprise Cloud Infrastructure
As AI agents enter enterprise production scenarios, the value of cloud infrastructure will become even more significant. The coordination of compute, storage, networking, security, containers, disaster recovery, and AI capabilities will determine whether enterprises can turn agents from pilot tools into reliable productivity systems.
The evolution of SmartX ECP is designed to help enterprises continuously add these capabilities on top of existing infrastructure, using one platform to achieve a smooth evolution from traditional applications to the agent era.
References
[1] NVIDIA, Agentic AI in the Factory, NVIDIA Enterprise AI Factory Design Guide White Paper. https://docs.nvidia.com/ai-enterprise/planning-resource/ai-factory-white-paper/latest/agentic-ai-in-the-factory.html
[2] AMD, Why is Everyone Talking About CPUs and Agentic AI?, 2025. https://www.amd.com/content/dam/amd/en/documents/epyc-business-docs/other/why-is-everyone-talking-about-cpus-and-agentic-ai.pdf
[3] Intel, Agentic AI Requires More CPUs: Avoiding GPU Idle Time in Modern Inference Pipelines, 2026.
[4] VMware by Broadcom, Private Cloud Outlook 2026: The AI Tipping Point, 2026. https://www.vmware.com/docs/private-cloud-outlook-2026
[5] HPE, HPE brings agentic AI into production with NVIDIA, delivering security, governance, scale, and sovereignty, 2026. https://www.hpe.com/us/en/newsroom/press-release/2026/06/hpe-brings-agentic-ai-into-production-with-nvidia-delivering-security-governance-scale-and-sovereignty.html
You may also be interested in:
AI Models from PoC to Production: Overcoming 5 Key Challenges to Double Your Efficiency
