In a meeting held in May, a manager at Amazon Web Services (AWS) conveyed a severe directive to engineers: to ensure sufficient computing capacity for all customers of its flagship Elastic Cloud Compute (EC2) service, engineers must take every measure to conserve resources, according to a person familiar with the meeting.
The compute resources needing reduction include both general-purpose central processing unit (CPU) server capacity and the persistently scarce AI-specific chip capacity, the source said. While the internet industry has relied on CPU chips for decades, many AWS engineers now report that wait times for requesting CPU server capacity for research and development have lengthened significantly.
An engineer noted that server capacity that used to be approved within hours now takes days, often causing project delays. This engineer, with many years at AWS, said they have never experienced such a long queue for resource allocation. Another person familiar with internal planning disclosed that AWS has set a deadline for teams to complete compute reduction by the second half of this year.
Engineers are shutting down idle EC2 virtual servers (cloud instances) after software development is complete, freeing up those resources for external customers. The compute shortage that has swept the tech industry this year has now spread to the traditional general-purpose computing sector.
It is well known that the surge in demand for AI chips, such as Nvidia graphics processing units (GPUs), has directly caused a shortage of AI-specific compute power. Meanwhile, driven by the AI industry, CPU servers are also facing supply constraints. Shortages of memory chips used with CPUs and limited space in data center physical racks have further exacerbated the compute gap.
Jing Xie (a phonetic translation), co-founder and director of financial AI service provider Erendil Labs, analyzed that as more employees use AI agents to develop software, enterprise CPU consumption has surged. Among her client companies, per-employee IT spending has doubled directly, as a large amount of work is now handled by AI agents, requiring more cloud compute capacity, with such tasks generally relying on CPUs.
Even in the AI model development process, CPUs play a critical role: for example, when preprocessing data before training a model, AI companies rely on CPUs to read raw materials like documents, images, and videos. Xie stated, "Currently, the vast majority of model development, generation, and inference tasks require far more CPU capacity than in the past."
Intel CEO Pat Gelsinger revealed on an earnings call in April that the ratio of CPU to GPU compute power for AI inference (model operation) was 1:4 at the time. By July, Intel CFO David Zinsner said the ratio had approached 1:1, with similar assessments from executives at AMD and Arm. AWS denied any change in its operational strategy, stating in an official statement, "Even with sustained high demand, we can still meet the compute needs of the vast majority of our internal and external customers. We have always worked with internal teams to ensure their compute supply while urging them to use EC2 resources efficiently, and this management approach has never changed."
Amazon said the guidelines for internal engineers to optimize resources are identical to the advice given to external customers: for example, shut down idle cloud instances and switch to instance types that better match their workload. If a customer's current instance is underutilized, they can move to a smaller configuration. AWS has set a clear optimization standard: if an EC2 server's CPU and memory average utilization is below 40% for four consecutive weeks, it recommends downgrading to a smaller instance. AWS provides automated scanning tools to identify underutilized servers.
Amazon emphasized that the resource management standards for internal employees have not been adjusted, even amid the current memory chip shortage. Balancing internal R&D compute with external customer compute supply has become a common challenge for major tech companies over the past two years.
Last year, Google set up a special executive committee to allocate compute resources across three segments: Google Cloud, DeepMind AI lab, and consumer businesses, but conflicts over resource allocation remain prominent. Reports indicate that this summer, top Google AI researcher Noam Shazeer left the company due to limited compute resources hindering his research. Companies that can precisely manage compute resources can convert idle capacity into revenue faster.
Amy Hood, CFO of Microsoft, said on the latest earnings call last month that growth in Azure cloud services was partly due to the team's efficient scheduling and optimization of CPU and GPU computing clusters. Hood stated, "Our engineering team has achieved significant results in activating compute capacity this quarter. Under the influence of the supply-demand imbalance, as long as we improve compute utilization efficiency, the newly available resources can quickly generate revenue." An AWS industry consultant said that reserved CPU capacity locked in by customers through contracts has not yet shown a supply gap, but in recent months, it has become significantly more difficult to apply for AWS spot instances (idle surplus capacity sold at a low price, which can be reclaimed with two minutes' notice) in large quantities.
Comments