Over the past year, the field of artificial intelligence has fundamentally reshaped enterprise architecture. The rise of agentic AI—autonomous systems that can pursue objectives rather than just responding to queries—is driving a comprehensive reassessment of infrastructure, data management, and cost optimization strategies.

The Myth of Thousands of GPUs

One common misconception is that enterprises need vast numbers of graphics processing units (GPUs) to run AI workloads. In reality, enterprise AI is dominated by inference, not training. For agentic AI, inference is the primary computational requirement, and it often requires only a handful of GPUs—sometimes as few as 16 for thousands of users.

This marks a significant departure from the early days of the AI boom, when organizations rushed to secure massive GPU clusters for model training. According to John Roese, global CTO of Dell Technologies, the enterprise's biggest workload runs on just 16 GPUs yet supports 40,000 people. The key insight is that agents do not require training; they rely on pre-trained models that can be fine-tuned or adapted through inference.

Balancing CPUs and GPUs

The architecture for agentic AI is also different from chatbots. Chatbot workloads are GPU-heavy with light CPU utilization. However, agents interact with external tools, communication protocols, knowledge graphs, and other components that do not naturally reside in the GPU. This creates a more balanced compute profile, with approximately one CPU for every two GPUs.

Enterprises must therefore build AI infrastructure that combines both GPU and traditional CPU compute, rather than simply piling on GPUs. This architectural shift requires careful planning to avoid bottlenecks and ensure efficient resource utilization.

Cloud, On-Premise, and the Edge

A year ago, the most powerful frontier models were only accessible via cloud APIs. Now, hyperscalers like Google offer distributed cloud services that allow top-tier models to run on-premise, in virtual private clouds, or even air-gapped from external networks. This flexibility enables enterprises to deploy AI in environments that meet their security, performance, and data sovereignty requirements.

At the same time, AI is moving to the edge. New agentic frameworks such as OpenClaw run natively on devices and AI PCs, providing structured environments for autonomous agents. This is not a passing trend but a powerful capability that allows AI to operate in decentralized, low-latency settings.

Rethinking the Data Layer

Data strategies are also evolving. Simply bolting standard storage systems onto AI compute clusters is no longer sufficient. Agentic AI requires knowledge and context layers built from vector databases, graph databases, and data annotation tools. These components must be deeply integrated with compute to avoid performance bottlenecks.

One significant challenge is getting data to GPUs fast enough. If the data pipeline is slow, expensive GPUs sit idle. To address this, Dell's AI data platform is directly connected to Nvidia's Cuda-X interfaces, allowing data layer services to run at GPU speed. This reduces latency and maximizes hardware utilization.

Mastering Tokenomics and Model Routing

As AI consumption grows, managing costs becomes critical. Although the cost per token is expected to decline over time, total AI spending will not necessarily decrease. Enterprises must treat AI workloads as an arbitrage game, using model routing to send different tasks to the most cost-effective models.

For example, a specification-driven development workflow might spawn dozens of coding tasks. Complex planning tasks should be routed to expensive frontier models, while routine coding can be handled by smaller, on-premise open-source models where energy is the only operational cost. Mastering this routing will be a competitive differentiator, reducing overall product development costs.

The Human Element

The hardest part of operationalizing agentic AI is not technology but people. Traditional jobs are like containers filled with a mix of hygiene, productivity, coordination, and expert tasks. AI agents excel at specific types of work but cannot yet replace an entire job. Dell audited 6,400 jobs and found that every single role will change.

If AI removes half the tasks from a job container, the organization can either reduce headcount or expand the remaining work to focus on higher-value activities. This requires careful change management, which has become a key remit for IT leadership. Roese noted that he now spends 50% of his time on human dynamics, as AI adoption is no longer just a technology or ROI discussion—it is fundamentally about adapting the workforce.

Enterprises that fail to address the human dimension risk losing the benefits of agentic AI, as employees may resist or misuse the technology. Effective change management involves transparent communication, reskilling programs, and rethinking organizational structures to leverage augmented capabilities.

As agentic AI continues to mature, enterprises must adapt their architectures, cost models, and people strategies to thrive in this new era. The next year will likely bring even more profound shifts as autonomous agents become embedded in every aspect of business operations, from software development to customer service to supply chain management.

Source:ComputerWeekly.com News