Meta’s AI jobs are getting so large that now a single cluster can take up more capacity than an entire data center has to offer. 

These clusters require thousands of GPUs working together, say to pre-train a large language model. So they have to be in complete synchronization.   

On Monday, the Meta engineering team released a post describing how they are using a new architectural approach they devised called backend aggregation (BAG), a network “super-spine” to tie together thousands of GPUs across adjacent data centers on different network fabrics.

BAG offers immense bandwidth across regions, reaching 16-48 Pbps per pair.

BAG is a distributed Ethernet-based L3 network layer that interconnects multiple L2 fabrics, strictly adhering to distance, buffer, and latency constraints

The company is deploying BAG to build a single AI cluster, Prometheus, which will span multiple data centers, and deliver 1-gigawatt of capacity.

Inter-BAG connectivity utilizes either a planar (direct match) or spread connection topology, depending on site size and fiber availability. Planar is a one-to-one connection, while spread connection has multiple connectivity points. 

Why Does Meta Need Such Large AI Clusters?

“The advent of AI has changed all of our assumptions on how to scale our infrastructure,” wrote Facebook engineering staff Yee Jiun Song and Kaushik Veeraraghavan, in another blog post in September.  “Building infrastructure for AI requires innovation at every layer of the stack, from hardware and software, to our networks, to our data centers themselves.”

Meta’s first brush with AI came in the late 2010s with personalized recommendations for short-form video. Until then, Facebook’s recommendations were centered on groups, not individuals. So personalization was a step up in computing complexity. 

Gone were the days when Facebook could just rank content based on what your friends liked. Now you need to rank all the content uploaded for every single user. 

Meta’s first AI clusters lashed together 4,000 GPUs each, to train multiple models for ranking and recommendation. These jobs typically required 128 GPUs.  

But then LLMs came along. LLMs required more, MORE, GPUs.The more compute you could give to a pre-training job, the better model you’d get. Single Llama3 jobs quickly consumed 2,000, even 4,000, GPUs. 

All these GPUs had to operate in synchronization. One bad GPU could ruin the entire job. And any sort of network jitter was No Bueno.

The AI clusters increased in size until they could take up an entire data center. Meta engineers started referring to clusters in terms of how much energy was being consumed by the data center they occupied, typically in the low 10s of megawatts.

In 2023, they built two clusters of 24,000 NVidia H100s, perhaps the largest of its kind at the time, the Meta engineers noted. 

But one data center was not enough! Fortunately the data center running these clusters was on a campus with additional data centers. In a matter of months, they built out a single AI cluster with 129,000 H100 GPUs across five production data centers, thanks to BAG.

 

And now Meta engineers want to take it to the next level again. They are currently building Prometheus, a 1 gigawatt system spanning “multiple” data centers. 

“As our AI clusters continue to grow, we expect BAG to play an important role in meeting future demands and driving innovation across Meta’s global network,” the engineers wrote.

A New Kind of Switch

Key to the success of BAG are two types of L2-layer fabrics Meta designed that are customized for low-latency traffic across thousands of end-points.

One is Disaggregated Schedule Fabric (DSF), a cell-based architecture that jettisons traditional monolithic chassis-switch architecture, by separating line cards and fabric cards. Line cards are those connecting to the GPUs, while fabric cards deal with the internal network. 

In effect, DSF makes the entire network one big switch, reducing hot spots.

Also crucial to BAG is a new kind of switching silicon co-designed by Meta, the Jericho3-AI ASIC, which, when used in modular switches, provide a collective “deep buffer” to prevent packet loss during sudden bursts of traffic.  

 For those older, less reliable, deployments, the engineering team also created Non-Scheduled Fabrics, which, like DSF, also offers adaptive routing for effective load-balancing.

“An important advantage of BAG’s distributed architecture is it keeps the distance from the L2 edge small, which is important for shallow buffer [non-scheduled fabric] switches,” the Meta engineers wrote.   

For NSF deployments, Meta uses two 51T Ethernet switches. One is the Minipack3, based on Broadcom Tomahawk5, and the other is Cisco 8501, based on Cisco Silicon One G200. Both run Meta’s large-scale network operating system, FBOSS.

Meta has no plans to stop increasing capacity. After Prometheus will come Hyperion, due in 2028, which will offer 5-Gigawatts of AI processing power.  Clearly all the work on customizing every layer of the network stack is paying off.