Meta's AI Storage Blueprint at Scale

Over the past several years, model capabilities and training dataset sizes have experienced exponential growth. During the past year or so, the time between new-frontier-model releases has gone down from months to weeks. Reliable and fast access to storage is important to both the speed and computational cost of this AI innovation. If AI is the brain, storage is the memory: Capability and speed are highly dependent on the size of memory and speed of retrieval.

过去几年里,模型能力与训练数据集规模都经历了指数级增长。仅在过去一年左右,新一代前沿模型的发布间隔已从数月缩短到数周。可靠而快速的存储访问,对这场 AI 创新的速度和算力成本都至关重要。如果说 AI 是大脑,那么存储就是记忆:能力和速度高度取决于记忆的容量和检索的速度。

Yet while AI compute performance has roughly tripled every two years, storage and interconnect performance growth have been more modest. As a result, storage bottlenecks continue to be one of the primary contributors to GPU stalls for AI workloads, directly impacting expenditures and time to market. Aside from GPU utilization, storage architecture also directly impacts the speed of iteration in AI research; with GPUs increasingly becoming geo-distributed and dataset sizes increasingly becoming massive, researchers spend a significant amount of time ingesting and moving data across regions, thus impacting research velocity. In this blog post, we discuss how Meta’s BLOB-storage architecture evolved to address two primary challenges: maximizing GPU utilization and maximizing research velocity.

然而,尽管 AI 算力性能大约每两年翻三倍,存储与互连的性能增长却要温和得多。结果就是,存储瓶颈始终是 AI 工作负载中导致 GPU 停顿的主要原因之一,直接影响开支和上市时间。除了 GPU 利用率之外,存储架构还直接影响 AI 研究的迭代速度;随着 GPU 越来越呈地理分布式部署、数据集规模越来越庞大,研究人员要花大量时间在跨地域摄取和搬运数据上,从而拖慢研究进度。在这篇博文中,我们将讨论 Meta 的 BLOB 存储架构如何演进,以应对两大主要挑战:最大化 GPU 利用率和最大化研究速度。

Meta operates hundreds of exabyte-scale storage clusters that serve all of Meta’s external and internal products, including Facebook, Instagram, Reality Labs, Meta AI, Ads, Data Warehouse, and internal Databases. Our storage service exposes object storage, file systems, and block-device APIs, and these API abstractions are built on top of a horizontally scalable foundational block layer called Tectonic. The Tectonic layer is a regional, multi-tenant storage fabric that provides high durability and availability leveraging erasure-coding techniques, supports tiering across media types (e.g., HDD and flash), and manages smart placement of hot, cold, and warm data for efficient utilization of I/O across tenants. The BLOB-storage layers that operate on top of Tectonic expose a global, infinitely scalable storage fabric, and expose policies that let users make tradeoffs between durability and availability.

Meta 运营着数百个 EB 级存储集群,服务于 Meta 全部对外和对内产品,包括 Facebook、Instagram、Reality Labs、Meta AI、Ads、数据仓库和内部数据库。我们的存储服务对外暴露对象存储、文件系统和块设备 API,而这些 API 抽象都构建在一个可水平扩展的基础块层之上,该层称为 Tectonic。Tectonic 层是一个区域级、多租户的存储底座,借助 erasure coding 技术提供高持久性和高可用性,支持跨介质类型(如 HDD 和闪存)的分层,并对热、冷、温数据进行智能放置,以便在各租户间高效利用 I/O。运行在 Tectonic 之上的 BLOB 存储层对外暴露一个全局的、可无限扩展的存储底座,并暴露相应策略,让用户在持久性和可用性之间做权衡。

In a previous @Scale talk titled, “Training Llama: A Storage Perspective,” we discussed how Meta trained Llama directly over the Tectonic block layer by exposing an NFS-like FileSystem interface on top of it. While this architecture continues to be used widely within Meta, our modern training stack has been migrating slowly on top of the BLOB-storage interface, as is the case across the industry. This transition is motivated by the need for unified storage access to massive data lakes in the BLOB-storage layer as well as the need for high performance.

在此前一场题为《训练 Llama:存储视角》的 @Scale 演讲中,我们讨论了 Meta 如何在 Tectonic 块层之上暴露一个类 NFS 的文件系统接口,从而直接基于该层训练 Llama。虽然这一架构在 Meta 内部仍被广泛使用,但正如整个行业的趋势一样,我们现代的训练栈正在缓慢地迁移到 BLOB 存储接口之上。推动这一转变的,既是对在 BLOB 存储层统一访问海量数据湖的需求,也是对高性能的需求。

Modern AI workloads are “data hungry” and have very different workload characteristics than traditional web applications: bursty and sustained high throughput, predictable and bounded pMax latencies, and variable I/O patterns. The focus for BLOB storage, in recent years, has largely shifted to maximizing GPU utilization.

现代 AI 工作负载"饥渴于数据",其负载特征与传统 Web 应用大不相同:突发且持续的高吞吐、可预测且有上界的 pMax 延迟,以及多变的 I/O 模式。近年来,BLOB 存储的重心已在很大程度上转向最大化 GPU 利用率。

To see why bounded and low-pMax latencies are important, let’s consider model training. During that training, hundreds of thousands of GPUs iterate over vast amounts of data in storage multiple times (i.e., over multiple epochs), and the GPUs train datasets in batches. Periodically, after every certain number of steps or batches, the GPUs synchronize their state among themselves. If one GPU is slow, this step will slow down all GPUs as well as the entire training.

要理解为什么有界且低的 pMax 延迟如此重要,我们来看看模型训练。在训练过程中,数十万块 GPU 会对存储中的海量数据反复迭代多遍(即经历多个 epoch),并以 batch 为单位在数据集上训练。每隔一定步数或若干个 batch,GPU 之间会周期性地同步各自的状态。如果有一块 GPU 慢了,这一步就会拖慢所有 GPU,乃至整个训练。

Figure 1 shows a data-loading pipeline across two GPUs. The dataloader in every GPU host prefetches the next dataset batch, while the GPU is processing the current batch for maximum compute or I/O overlap. In the case of GPU1, the storage-fetch latency is well within bounds, so the GPU is never stalled waiting on I/O. In the case of GPU2, there are two instances where storage fetch exhibits high latency, stalling GPU. As a result of these stalls, the overall step-completion time is delayed.

图 1 展示了跨两块 GPU 的数据加载流水线。每台 GPU 主机中的 dataloader 会预取下一个数据集 batch,与此同时 GPU 正在处理当前 batch,以实现计算与 I/O 的最大重叠。对于 GPU1,存储取数的延迟始终在界限之内,因此 GPU 从不会因等待 I/O 而停顿。对于 GPU2,则有两次存储取数出现高延迟,导致 GPU 停顿。这些停顿的结果是,整体的单步完成时间被延后。

Figure 1: Dataloading across two GPUs.

图 1:跨两块 GPU 的数据加载。

Over the years, BLOB storage evolved organically, adding layers on top of layers in a true service-oriented fashion. Many of these layers were stateful and maintained their own metadata stores. While these metadata-access latencies typically weren’t the bottleneck for the traditional use cases served by global HDDs, they were showstoppers for AI workloads with millisecond access to data in flash. Figure 2 shows the request flow for a typical getObject(“/bucket/path”) API. After the request arrives at the API server, the server does many metadata lookups across the namelayer, volumeslayer, and containerlayer before resolving the path to a set of (blockId, offset, size) tuples. Some of these lookups can cross regions, and it’s not uncommon for latencies to add up to hundreds of milliseconds; one slow response from any of the lookups was sufficient. After the lookups, the API server proxies the data from the Tectonic layer to the client.

多年来,BLOB 存储是有机演进而来的,以典型的面向服务方式层层叠加。其中许多层都是有状态的,维护着各自的 metadata 存储。对于全局 HDD 服务的传统用例,这些 metadata 访问延迟通常并不构成瓶颈,但对于要以毫秒级访问闪存中数据的 AI 工作负载,它们却是致命障碍。图 2 展示了一次典型 getObject("/bucket/path") API 的请求流。请求到达 API 服务器后,服务器会在 namelayer、volumeslayer 和 containerlayer 之间做多次 metadata 查找,才能把路径解析为一组 (blockId, offset, size) 元组。其中有些查找可能跨区域,累计延迟达到数百毫秒并不罕见;任何一次查找响应慢了都足以拖垮整体。查找完成后,API 服务器再把数据从 Tectonic 层代理给客户端。

Figure 2: Old request flow for getObject API.

图 2:getObject API 的旧请求流。

While this architecture served conventional workloads well, the foundational assumptions that dictated design tradeoffs have since shifted. Some of these are:

尽管这一架构很好地服务了传统工作负载,但当初决定设计权衡的那些基础假设,如今已经发生了变化。其中一些是:

In short, the tradeoff space has shifted enough for us to rethink the entire architecture.

简而言之,权衡空间已经发生了足够大的变化,足以让我们重新思考整个架构。

As we set out to build the new foundation, we made the following major design choices:

在着手构建新的底层基础时,我们做出了以下几项重大设计决策:

Figure 3: New request flow for getObject API.

图 3:getObject API 的新请求流程。

Figure 3 shows the new request flow for getObject(“/bucket/path”). When the SDK on the client receives this API call, it now issues a getReadPlan(“/bucket/path”) request to the API server. The API server does O(1) lookup per chunk to the new metadata store to map the path to (blockId, offset, size) tuples. It then returns the ReadPlanResult to the SDK. The SDK has Tectonic BlockClient embedded within it, and so is now able to stream data from these blocks directly from Tectonic. With these changes, we have rebuilt the foundations and met the goal of adding zero overhead on top of Tectonic. By eliminating the data proxy, we also stay within budget for the power footprint.

图 3 展示了 getObject(“/bucket/path”) 的新请求流程。当客户端上的 SDK 收到这个 API 调用时,它现在会向 API 服务器发起一个 getReadPlan(“/bucket/path”) 请求。API 服务器针对每个 chunk 向新的 metadata 存储做 O(1) 查找,把路径映射为 (blockId, offset, size) 三元组,然后把 ReadPlanResult 返回给 SDK。SDK 内嵌了 Tectonic BlockClient,因此现在能够直接从 Tectonic 上的这些 block 流式读取数据。通过这些改动,我们重建了底层基础,达成了在 Tectonic 之上零额外开销的目标。通过消除数据代理,我们也把功耗控制在预算之内。

During data and checkpoint loading, AI workloads are known to access data concurrently across hundreds of GPUs. Subsets of data such as model weights are often “hot,” and events such as GPU restarts trigger sharp traffic spikes. With the foundations now fixed, our next problem was dealing with those spikes and hot spots. Luckily, the BLOB-storage layer has had experience dealing with hot spots over the years, so we adapted existing solutions to AI workloads here. Specifically, we employed two approaches:

在加载数据和 checkpoint 时,AI 工作负载会跨数百块 GPU 并发访问数据,这是众所周知的。诸如模型权重这类数据子集往往是“热”的,而 GPU 重启之类的事件会触发陡峭的流量尖峰。既然底层基础已经修好,我们的下一个问题就是应对这些尖峰和热点。幸运的是,BLOB 存储层多年来在处理热点上积累了经验,于是我们把既有方案适配到这里的 AI 工作负载上。具体来说,我们采用了两种做法:

In practice we observe an average cache hit rate of 80% on the distributed data cache, and the read-plan cache provides 1-2 ms access to metadata. In essence, these simple mechanisms do three things:

在实践中,我们观察到分布式数据 cache 的平均命中率为 80%,而 read-plan cache 提供 1-2 ms 的 metadata 访问延迟。本质上,这些简单机制做了三件事:

What we’ve discussed so far got us 80% of the way. We achieved the remaining 20% by identifying and fixing bottlenecks across the stack. Below are some noteworthy problems, though not an exhaustive list by any means:

到目前为止讨论的内容,让我们完成了 80% 的路程。剩下的 20% 是靠在整个栈中识别并修复瓶颈来实现的。以下是一些值得一提的问题,但绝非详尽无遗:

With all of the above, the new BLOB-storage stack is now capable of serving AI workloads without causing GPU stalls, adding negligible overhead on top of the Tectonic layer. Our next focus shifted to research.

有了上述所有改进,新的 BLOB 存储栈如今已能服务 AI 工作负载而不会导致 GPU 停顿,在 Tectonic 层之上只增加可忽略的开销。我们的下一个关注点转向了研究。

GPUs are scarce and increasingly becoming geo-distributed; at the same time, training workloads need data colocated with GPUs for performance reasons. This creates an interesting challenge for researchers: They are now on the hook for ingesting and moving datasets across regions.

GPU 资源稀缺,且正日益走向地理分布式部署;与此同时,出于性能考虑,训练负载又要求数据与 GPU 同地部署。这给研究人员带来了一个有意思的挑战:他们如今得自己负责把数据集摄入并跨区域搬运。

At Meta, a typical training-job submission involves the following:

在 Meta,一次典型的训练作业提交包含以下步骤:

Steps 2 through 4 can take hours and directly impact the speed of iteration for researchers. Ideally, we like our researchers’ time to be spent on tuning models, not waiting for storage. Currently, researchers copy snapshots before starting their jobs to colocate data with GPUs, which results in the most optimal performance. While this optimization for performance makes sense for large-scale training jobs that span weeks or months, the vast majority of jobs are much smaller; the researchers owning these jobs are more than willing to trade off occasional performance degradation for iteration speed.

第 2 步到第 4 步可能耗时数小时,直接影响研究人员的迭代速度。理想情况下,我们希望研究人员的时间花在调优模型上,而不是等待存储。目前研究人员会在启动作业前先拷贝快照,让数据与 GPU 同地部署,从而获得最优性能。对于跨越数周乃至数月的大规模训练作业,这种性能优化是合理的;但绝大多数作业规模要小得多,拥有这些作业的研究人员非常愿意用偶尔的性能下降来换取迭代速度。

And so, we needed a system where researchers are able to ingest data once and access data anywhere without thinking about regional boundaries. We needed a workflow that allows researchers to iterate in minutes and not hours. As we went back to the drawing board, the write-once, read-many characteristic of these datasets rang a bell. What if we think of storage as a disk in a planet-scale computer and borrow ideas from the operating-system world? When a Linux process running on a CPU core attempts to read a file from disk, the operating system transparently hydrates data on demand across the various layers of the cache—page cache in memory and L2 and L1 CPU caches. This intuition led to the architectural evolution in Figure 4:

因此,我们需要一套系统:让研究人员只需摄入数据一次,就能在任何地方访问,而无需操心区域边界。我们需要一套让研究人员以分钟而非小时为单位迭代的工作流。当我们重新回到设计白板前时,这些数据集一次写入、多次读取的特性让我们灵光一闪。如果把存储看作一台行星级计算机里的磁盘,并借鉴操作系统世界的思路呢?当运行在某个 CPU 核心上的 Linux 进程试图从磁盘读取文件时,操作系统会透明地按需在各级 cache 层——内存中的 page cache 以及 L2、L1 CPU cache——之间填充(hydrate)数据。这一直觉引出了图 4 所示的架构演进:

Figure 4: Dataloading architecture evolution.

图 4:数据加载架构的演进。

The core idea is to leverage the various on-host and off-host storage resources as a tiered cache with global BLOB-storage fabric backed by HDDs as the ultimate source of truth. Specifically, we leverage the memory and flash on the GPU host as L1 and L2 caches. And we leverage the regional BLOB-storage fabric backed by flash as the L3 cache dataloader continues to access storage through the familiar BLOB-storage SDK. To effectively hide latencies and to simplify the data life cycle, we rely on the following:

核心思路是把各类主机内与主机外的存储资源当作一个分层 cache 来利用,并以 HDD 支撑的全局 BLOB 存储网格作为最终的事实来源(source of truth)。具体而言,我们把 GPU 主机上的内存和闪存分别用作 L1 和 L2 cache;并把由闪存支撑的区域级 BLOB 存储网格用作 L3 cache,dataloader 则继续通过熟悉的 BLOB 存储 SDK 访问存储。为了有效隐藏延迟并简化数据生命周期,我们依赖以下几点:

We saw rapid adoption of this new data-loading paradigm as soon as production rollout started, and we continue to support both of the data-loading paradigms in production today. To illustrate the impact in numbers, Figure 5 shows roughly the ingestion times before and after the rollout across all workloads:

生产环境一开始铺开,我们就看到这套新数据加载范式被迅速采用;如今我们在生产环境中仍同时支持两种数据加载范式。为了用数字直观展示其影响,图 5 展示了铺开前后跨所有负载的大致摄入时间:

Figure 5: Ingestion times before and after the rollout.

图 5:铺开前后的摄入时间。

In a world where new frontier models get released in weeks, this shift in the data-loading paradigm is a much-needed change to move even faster.

在一个前沿模型以周为单位不断发布的世界里,数据加载范式的这一转变正是我们进一步提速所急需的变革。

Modern AI workloads are data hungry, and storage plays an important role in both the computational cost and speed of innovation. Storage bottlenecks directly impact GPU utilization and computational cost, and in a world with geo-distributed GPUs, time spent on cross-region data ingestion directly impacts the speed of iteration in research. The BLOB-storage architecture at Meta was built to serve Meta’s family of apps, and we needed a step-function improvement in performance to serve AI workloads. This led to rethinking the entire architecture. By rebuilding the metadata subsystem and by adopting a tiered caching architecture with prefetching/on-demand hydration, we are able to meet the needs of today’s workloads effectively.

现代 AI 负载对数据极度饥渴,而存储在计算成本和创新速度两方面都扮演着重要角色。存储瓶颈直接影响 GPU 利用率和计算成本;在一个 GPU 地理分布式的世界里,花在跨区域数据摄入上的时间直接影响研究的迭代速度。Meta 的 BLOB 存储架构最初是为服务 Meta 的系列应用而构建的,而要服务 AI 负载,我们需要性能上的阶跃式(step-function)提升。这促使我们重新审视整个架构。通过重建 metadata 子系统,并采用带预取/按需填充(hydration)的分层 cache 架构,我们得以高效满足当今负载的需求。

We are continuously evolving storage at Meta to keep up with hardware evolution and workload demands. Some future work in this area will include:

我们在 Meta 持续演进存储,以跟上硬件演进和负载需求。该领域未来的一些工作将包括: