Agents present a bursty workload, leaving CPUs frequently idle, which is placing new demands on CPU infrastructure.

The requirements of agentic AI are placing new demands on data center CPUs and infrastructure as a whole, Mark Lohmeyer, vice president and general manager for AI and computing infrastructure at Google Cloud, told EE Times.

“Chat is relatively straightforward—there’s a question and an answer, and it relies heavily on inference across GPUs or TPUs,” he said. “With agents, the nature of those interactions is changing quite a bit.”

With agents, the user’s high-level intent can be broken down into subtasks and assigned to a team of agents that can access other APIs, visit websites, write and run code—CPU tasks. Sessions can be much longer and can involve multiple human users, which means context and state need to be preserved over potentially days or weeks.

“Agents are often bursty, and often idle,” he said. “They don’t do anything for [say] a day or two until you come back to them, or maybe until some API responds… It’s unique from older CPU workloads, in that it needs to be stateful over a long period of time, but is often idle during that period of time, and then when it’s not idle, it’s highly bursty.”

Handling this in a performant and secure way is a big challenge the industry is facing today, Lohmeyer said.

“Demand is already very high and is growing rapidly,” he said. “So, how do we provide the infrastructure that can support that growth and do it cost-effectively?”

New workloads

Lohmeyer distinguished between the compute needed to run the agent itself, which is a new workload requiring new infrastructure, and other tasks such as accessing websites and APIs, which already existed but were previously interfacing with humans.

“Agentic workloads will have a meaningful impact on both of those elements,” he said.

Existing parts of the workload will grow dramatically since agents are much more demanding than humans and hundreds or thousands of agents can be brought online very quickly while the number of human users can’t grow as fast.

“Ideally, you’d like to have those agents and their context, the applications and data, relatively close to each other from a latency perspective, for performance reasons,” Lohmeyer said.

Locality is important, but enterprises will continue to be a distributed world, with most continuing to use multiple clouds and private clouds for the time being, Lohmeyer said. This means high-performance networks will also continue to be incredibly important both within the data center and between data centers.

“Dynamic, stateful agents will need to have a new type of computing infrastructure that supports them,” he said. “They’re accessing all the existing apps and data, so the demand there is growing, and all this needs to be connected together through the network. Google is investing in all of these elements.”

For the new infrastructure, the optimum ratio of CPU-to-GPU compute for agentic AI is still evolving with the workload, Lohmeyer said, but the proportion of CPU compute required is definitely increasing as we switch from training foundation models to agentic inference.

“We expect both [CPU and TPU/GPU] workloads to continue to grow rapidly, but the proportion of CPUs is going from a relatively small portion to a meaningful portion, and we think that will continue to evolve as agents expand into new use cases and new environments,” he said.

Hardware-software-workload co-design

Solving the agentic inference problem will continue to require co-design across the infrastructure stack, Lohmeyer said, including CPUs, storage, memory, networking, and how it all works together.

“Google Cloud has always had a philosophy of workload-optimized infrastructure,” he said. “This is more important than ever in the agentic era.”

Since agentic workloads are bursty and often leave the CPU idle, agents can effectively be “hibernated”—their state can be sent to storage so the CPU can be used by other agents, but reloaded quickly when required again.

“This requires a new type of orchestration in software, and in this environment, having really high-performance storage that’s attached to that server, so I can restart it quickly, is important,” he said.

Google Cloud uses microVMs, like a “slimmed down” version of a virtual machine, to provide a secure boundary between agents packed onto the same server. Rather than 100 traditional VMs, the same hardware might run 1,000 microVMs.

“Security and governance for agents is absolutely critical, and Google is investing a lot in every layer of the stack,” he said.

At the lowest layer, virtualization is key, Lohmeyer said. Google plans to support a variety of ways to sandbox agents, but the microVM is particularly important. A microVM’s streamlined feature set means performance is improved, and a smaller attack surface helps security, versus traditional virtual machines. The company is also investing in agent gateways that can provide egress controls for agents at the network level, along with other higher-level controls.

Looking forward, the requirement to run thousands of microVMs for agents will have implications for core density, storage, how storage is attached, and for orchestration software to manage it all, Lohmeyer said. The benefits will be higher utilization of hardware, which should drive down the cost per agent, he said.

“This will allow our customers to scale their business, but do so with a good business model, good profitability, but without having to sacrifice performance as they are scaling,” he said. “These capabilities will be very important going forward.”

CPU choice

There won’t be a single optimum CPU choice for every type of agent, Lohmeyer said. Google Cloud uses a variety of x86 CPUs from Intel and AMD as well as its in-house designed Axion CPU, which is Arm-based.

“If you’ve got a data-heavy agentic use case, maybe you need super high core performance and a large amount of local SSD, but maybe if it’s more web-serving, you just want a lot of cores,” he said. “We think that ultimately, customer choice and being able to map the specific agentic workload to the specific CPU option in Google Cloud will be important.”

There is “a lot of customer interest” in Axion at the moment, Lohmeyer said, across performance- and efficiency-optimized instances as well as interest for bare-metal instances from larger customers.

https://www.eetimes.com/google-agentic-ai-will-drive-new-demands-on-data-center-cpus/?shem=aimgspe,