Why Meta Reserves Thousands of GPUs for Its Biggest Training Runs

In an engineering example published Monday, Meta provisions 37,656 graphics processors for a training job using 32,768. The 4,888-chip difference helps keep training moving when parts fail.

Meta's Prometheus supercluster in New Albany, Ohio, connects hundreds of thousands of graphics processing units, or GPUs. Its completed first phase trained Muse models. Expansion toward roughly one million GPUs remains planned. Meta's engineering account

Meta reports more than 85% goodput for Muse Wonder pretraining and more than 90% for reinforcement learning. Goodput measures useful training progress.

Those percentages direct attention to a practical question for companies spending heavily on computing equipment: how much of the installed machinery can reliably contribute to a finished model?

Reserving chips to finish more work

Meta groups 72 GPUs together, using 64 and reserving eight, with additional spare racks. Its example uses 523 racks. The 4,888-chip difference equals 13% of installed equipment, or 14.9% of the active count. Those percentages use different denominators.

An earlier presentation by Meta researchers Kevin Quirk, Matthew Lennie, Ehsan Ardestani and colleagues explains why reserving equipment can improve total output. More spare chips reduce time spent waiting for a replacement, while removing chips from productive use. Their analysis compares those competing costs instead of assuming every purchased chip should participate in every run. MLSys presentation

The researchers also distinguish availability from reliability. Availability describes how much equipment is ready to begin a job. Reliability describes how long it can keep working before a failure. A system can look healthy when assigned a task and still interrupt it repeatedly.

This changes how a chip count should be read. A purchase announcement describes equipment. A training plan also needs to account for reserves and recovery. Comparing two clusters by installed GPUs alone leaves those operational differences unresolved.

The earlier presentation's 72-chip comparison also keeps eight chips in reserve, 11.1% of that grouping. Expanding its active count requires adding reserves alongside it. The presentation evaluates alternatives with different group sizes and reserve arrangements.

A failure every 18 minutes

A separate paper by Omkar Salpekar, Rohan Varma and colleagues describes consequences at roughly 100,000 GPUs. They estimated a failure every 18 minutes. A conventional recovery taking 10 minutes would leave eight minutes of useful training in that cycle, roughly 44% of elapsed time. Fault-tolerant training paper

Conventional synchronous training requires GPUs to be “healthy simultaneously,” the authors wrote. Their alternative organized training into groups that could recover separately. When one group failed, healthy groups continued. Recovering machines obtained a recent saved state and caught up.

In their reported evaluation, recovery stalls fell from 10 minutes to three and effective training time rose from 44% to 80%. That experiment describes a particular system and method. It should not be treated as a direct comparison with Monday's Muse Wonder goodput figures, which concern different runs.

The new recovery method also raises a research question: whether letting some groups advance while another catches up changes the model being trained. The authors tested model-quality effects at a smaller, 256-GPU scale, a useful limit when interpreting their result.

Electricity has to follow the software

Keeping chips available also requires coordinating their electricity supply. A Meta power-management paper, revised in October, examines a 150-megawatt facility with 83,000 GB200 GPUs. Its authors describe decisions that begin six to 12 months before new accelerators become available. Engineers must choose equipment and supporting infrastructure before they can measure every relevant workload. Power-management research

Buying enough electrical capacity for every device's maximum rating can leave usable power stranded. Installing more accelerators with carefully chosen operating limits can increase the amount of computing that fits within a site's supply. The balance depends on measurements of actual workloads and equipment.

The paper also shows why reducing power on individual chips can disappoint. In synchronized training, slower workers hold up other machines. The authors' Dimmer controller considers running jobs when reducing power, so that a local electrical constraint does not unnecessarily slow a much larger task.

The researchers also found that rack power-supply readings overstated consumption. They calibrated those measurements against sensors higher in the electrical distribution system. Accurate readings helped engineers decide how much equipment could safely run and when it needed lower power limits.

Earlier designs show how much changed

Meta's March 2024 infrastructure account described two clusters with 24,576 H100 GPUs each. One used an Ethernet-based network and the other InfiniBand. The company used both to study how connections between machines affected large training jobs. Meta's 2024 account

That earlier account also described storage designed to let thousands of GPUs save and reload model checkpoints together. A checkpoint preserves a training job's state so engineers can resume after an interruption. Creating one consumes resources, but losing hours of unsaved training can cost more.

Meta's 2024 Llama 3 paper provides an unusually detailed record of those interruptions. During a 54-day snapshot of pretraining, it recorded 466 interruptions: 47 planned and 419 unexpected. Confirmed or suspected hardware issues accounted for roughly 78% of the unexpected events. The researchers reported effective training time above 90%, despite that frequency of interruptions. Llama 3 paper, reliability section

Most incidents were handled automatically. Significant manual intervention was needed on three occasions during the snapshot. The team described recording communication events and collecting more detailed traces when needed to diagnose stalls. This historical account shows both a failure count and a recovery result, allowing readers to examine what the system did after equipment stopped working. It concerns the older Llama 3 run, rather than a second measurement of Muse Wonder.

Cooling introduces another constraint. In an August explanation, Meta described water and glycol circulating through sealed loops and heat exchangers. Moving heat directly from chips allows denser equipment arrangements than relying solely on air. The company said it expects some coolant to remain in use for up to a decade. That describes coolant replacement, not a universal measure of a facility's total water use. Meta's cooling explanation

These engineering choices have wider significance as computing demand grows. The International Energy Agency's 2025 outlook projected global data-center electricity demand reaching around 945 terawatt-hours in 2030. That is a forecast, covering all data centers, rather than a measurement of Prometheus. IEA outlook

Future Prometheus disclosures can be compared on active chips, reserves and productive training time as expansion proceeds.