On Premise Gen AI

In 2025, enterprises transitioning generative AI (GenAI) into production after years of experimentation are increasingly considering on-premises deployment as a cost-effective alternative to the cloud.

Since OpenAI ignited the AI revolution in late 2022, organizations have tested large language models powering GenAI services on platforms like AWS, Microsoft Azure, and Google Cloud. These experiments demonstrated GenAI’s potential to enhance business operations while exposing the substantial costs of cloud usage.

To avoid difficult conversations with CFOs about escalating cloud expenses, CIOs are exploring on-premises AI as a financially viable solution. Advances in software from startups and packaged infrastructure from vendors such as HPE and Dell are making private data centers an attractive option for managing costs.

A survey conducted by Menlo Ventures in late 2024 found that 47% of U.S. enterprises with at least 50 employees were developing GenAI solutions in-house. Similarly, Informa TechTarget’s Enterprise Strategy Group reported a rise in enterprises considering on-premises and public cloud equally for new applications—from 37% in 2024 to 45% in 2025.

This shift is reflected in hardware sales. HPE reported a 16% revenue increase in AI systems, reaching $1.5 billion in Q4 2024. During the same period, Dell recorded a record $3.6 billion in AI server orders, with its sales pipeline expanding by over 50% across various customer segments.

“Customers are seeking diverse AI-capable server solutions,” noted David Schmidt, senior director of Dell’s PowerEdge server line.

While heavily regulated industries have traditionally relied on on-premises systems to ensure data privacy and security, broader adoption is now driven by the need for cost control. Fortune 2000 companies are leading this trend, opting for private infrastructure over the cloud due to more predictable expenses.

“It’s not unusual to see cloud bills exceeding $750,000 or even $1 million per month,” said John Annand, an analyst at Info-Tech Research Group.

Global manufacturing giant Jabil primarily uses AWS for GenAI development but emphasizes ongoing cost management. “Does moving to the cloud provide a cost advantage? Sometimes it doesn’t,” said CIO May Yap. Jabil employs a continuous cloud financial optimization process to maximize efficiency.

On-Premises AI: Technology and Trends

Enterprises now have alternatives to cloud infrastructure, including as-a-service solutions like Dell APEX and HPE GreenLake, which offer flexible pay-per-use pricing for AI servers, storage, and networking tailored for private data centers or colocation facilities.

“The high cost of cloud drives organizations to seek more predictable expenses,” said Tiffany Osias, vice president of global colocation services at Equinix.

Walmart exemplifies in-house AI development, creating tools like a document summarization app for its benefits help desk and an AI assistant for corporate employees. Startups are also enabling enterprises to build AI applications with turnkey solutions.

“About 80% of GenAI requirements can now be addressed with push-button solutions from startups,” said Tim Tully, partner at Menlo Ventures. Companies like Ragie (RAG-as-a-service) and Lamatic.ai (GenAI platform-as-a-service) are driving this innovation. Others, like Squid AI, integrate custom AI agents with existing enterprise infrastructure.

Open-source frameworks like LangChain further empower on-premises development, offering tools for creating chatbots, virtual assistants, and intelligent search systems. Its extension, LangGraph, adds functionality for building multi-agent workflows.

As enterprises develop AI applications internally, consulting services will play a pivotal role. “Companies offering guidance on effective AI tool usage and aligning them with business outcomes will thrive,” Annand said.

This evolution in AI deployment highlights the growing importance of balancing technological innovation with financial sustainability.