RL Infra · Engineering notes
When is delta all-gather (sharded diff) necessary?
Two weight-sync paths: NCCL and Disk. When PP-level and TP-level sharded diff are needed.
English
In this article
Background
Two months ago, Changyi Yang and I added delta weight sync support to verl.
Besides reducing the trainer → rollout payload, our implementation also performs sharded diff: compute the diff on the shards first, then gather only the delta, instead of gathering full weights before computing the diff.
When is this necessary? We consider two paths: NCCL, where weights are AllGathered to one node before broadcast to rollout, and Disk, where PP nodes write to storage in parallel. The discussion assumes a large model distributed across PP nodes, with TP communication staying within each node.
The decision depends on two things: whether one node's CPU RAM can hold the gathered weights, and whether full AllGather becomes the bottleneck after delta compression reduces the output payload.
TL;DR
- Both NCCL and Disk require PP-level sharded diff.
- For NCCL, TP-level sharded diff is not required, but recommended.
- For Disk, TP-level sharded diff is not required.
Two levels of sharded diff
PP-level: do not gather full weights across PP nodes. Each PP node computes the diff for its own portion, then sends or writes its delta. Full AllGather can still happen inside each node.
TP-level: each TP rank computes the diff on its local shard, so even node-local AllGather gathers only the delta.
In other words, PP-level sharded diff avoids cross-node full-weight gathers; TP-level sharded diff also reduces node-local gather traffic.
1. NCCL: AllGather to one node, then broadcast
PP nodes → gather to one source node → broadcast to rollout
PP-level sharded diff is required. Gathering the full model onto one node can exceed its CPU RAM (computing the diff requires keeping a pinned copy of the previous step's weights): a 1T-parameter BF16 model alone is about 2 TB. The cross-node gather also uses the network, just like trainer → rollout transfer. If we compress only the latter, the full gather becomes the new bottleneck. Compute the diff within each PP node and gather the deltas instead.
For example, assume the following effective bandwidths and payload sizes:
cross-node gather bandwidth = 30 GB/s
trainer → rollout bandwidth = 30 GB/s
full weights = 1 TB
delta density = 10%
full cross-node gather:
1000 GB / 30 GB/s ≈ 33 s
delta trainer → rollout:
100 GB / 30 GB/s ≈ 3.3 s
If we optimize only trainer → rollout, the trainer-side full gather immediately becomes the new bottleneck. Both stages use a network of similar bandwidth, but the full gather still moves ten times as much data.
TP-level sharded diff is not required, but recommended. Once the cross-PP gather is delta-aware, consider the full TP AllGather that still happens inside each node. NVLink is faster than cross-node RDMA, but delta compression also makes the network payload smaller.
For example, assume:
weights for this comparison = 300 GB
NVLink AllGather bandwidth = 300 GB/s
RDMA bandwidth = 30 GB/s
delta density = 10%
NVLink is 10× faster than RDMA, while the delta reduces the network payload by 10×:
full local AllGather:
300 GB / 300 GB/s = 1 s
delta RDMA transfer:
(300 GB × 10%) / 30 GB/s = 1 s
The two stages are roughly balanced, so TP-level sharded diff is not strictly required.
But if delta density falls to 1%:
full local AllGather:
300 GB / 300 GB/s = 1 s
delta RDMA transfer:
(300 GB × 1%) / 30 GB/s = 0.1 s
The full local gather now takes ten times as long as the delta transfer. This is why TP-level sharded diff is still recommended for NCCL: node-local NVLink can become the bottleneck when the payload reduction outweighs its bandwidth advantage.
2. Disk: PP nodes write to storage in parallel
Each PP node: local TP AllGather → diff → write delta to storage
PP-level sharded diff is required. Each PP node handles its own weights and writes its own delta, avoiding a full-model snapshot on a single node.
TP-level sharded diff is not required. Full node-local NVLink AllGather is usually much faster than writing to disk or object storage. Using the example bandwidths:
PP stage weights = 200 GB
NVLink AllGather bandwidth = 300 GB/s
storage write bandwidth = 5 GB/s
full local AllGather:
200 GB / 300 GB/s ≈ 0.67 s
full-weight storage write (baseline):
200 GB / 5 GB/s = 40 s
storage write with a 10% delta:
20 GB / 5 GB/s = 4 s
Even after delta compression, the 4 s storage write is much slower than the 0.67 s local gather. Making the gather 10× faster would reduce it from about 0.67 s to 0.07 s, saving about 0.6 s; it would not remove the storage bottleneck. Full local AllGather followed by diff is therefore usually sufficient.
This is the key bandwidth difference between NCCL and Disk. In these illustrative examples:
NVLink: hundreds of GB/s (300 GB/s)
RDMA: tens of GB/s (30 GB/s)
Disk / storage: a few GB/s/node (5 GB/s)
NVLink / RDMA bandwidth ratio = 10×
NVLink / storage bandwidth ratio = 60×
A 10× payload reduction can already offset NVLink's advantage over RDMA. The same reduction still leaves storage substantially slower than the full local gather. That is why TP-level sharded diff is recommended for NCCL, while Disk can usually skip it.
For a concrete disk-based implementation, see Slime's Delta Weight Sync documentation. It describes computing a diff for each gathered HF tensor, encoding and compressing the result, and publishing it through disk transport. The PP/TP recommendations here are our analysis of the topology and bottlenecks.
Conclusion
| Path | PP-level sharded diff | TP-level sharded diff |
|---|---|---|
| NCCL: AllGather to one node → broadcast | Required | Not required, but recommended |
| Disk: PP nodes write in parallel | Required | Not required |
中文
文章目录
背景
两个月前，我和Changyi Yang一起给verl支持了delta weight sync。
除了减少 trainer → rollout 的传输 payload，我们的实现还做了 sharded diff：先在 shard 上算 diff，再只 gather delta，而不是先 gather 完整权重再算 diff。
什么时候需要这样做？我们只讨论两种情况：NCCL（AllGather 到一个 node，再 broadcast 给 rollout）和 Disk（各 PP node 并行写 storage）。这里考虑的是大模型跨 PP node 分布、TP 通信留在单机内部的场景。
判断主要取决于两点：单个 node 的 CPU RAM 能否装下 gather 后的权重，以及 delta 压缩输出 payload 后，全量 AllGather 会不会成为新的瓶颈。
TL;DR
- NCCL 和 Disk 都必须做 PP-level sharded diff。
- NCCL：TP-level 非必须，建议做。
- Disk：TP-level 非必须，可以不做。
两种粒度的 sharded diff
PP-level： 不跨 PP node gather 完整权重。每个 PP node 对自己的部分算 diff，再发送或写出自己的 delta；机器内部仍然可以做全量 AllGather。
TP-level： 每个 TP rank 直接对自己的 local shard 算 diff，即使是单机内部的 AllGather，也只 gather delta。
也就是说，PP-level 避免跨机 gather 完整权重；TP-level 则进一步减少单机内部的 gather 数据量。
1. NCCL：AllGather 到一个 node，再 broadcast
各 PP node → gather 到一个 source node → broadcast 给 rollout
PP-level sharded diff 必须做。 完整模型聚合到一台机器，CPU RAM 可能放不下（由于要算 diff，所以必须 pin 一版上个 step 的权重）：1T 参数的 BF16 模型，仅权重就约 2 TB。同时，跨 PP node 的 gather 和 trainer → rollout 都走跨机网络。如果只压缩后者，前面的全量 gather 就会成为新瓶颈。因此，应该先在各 PP node 内算 diff，再聚合 delta。
例如，假设有效带宽和 payload 大小如下：
跨机 gather bandwidth = 30 GB/s
trainer → rollout = 30 GB/s
full weights = 1 TB
delta density = 10%
全量跨机 gather：
1000 GB / 30 GB/s ≈ 33 s
delta trainer → rollout：
100 GB / 30 GB/s ≈ 3.3 s
如果只优化 trainer → rollout，trainer-side full gather 就会直接变成新瓶颈。 两边使用的跨机网络带宽相近，但全量 gather 仍然要搬十倍的数据。
TP-level sharded diff 非必须，建议做。 跨 PP node 已经只 gather delta 后，再看机器内部仍然保留的全量 TP AllGather。NVLink 比跨机 RDMA 快，但 delta 压缩也会减少网络传输的 payload。
例如，假设：
本次对比的权重大小 = 300 GB
NVLink AllGather = 300 GB/s
RDMA = 30 GB/s
delta density = 10%
NVLink 比 RDMA 快 10 倍，而 delta 又把网络 payload 缩小了 10 倍：
单机全量 AllGather：
300 GB / 300 GB/s = 1 s
delta RDMA 传输：
(300 GB × 10%) / 30 GB/s = 1 s
两边基本平衡，所以 TP-level sharded diff 不是绝对必须的。
但如果 delta density 进一步降到 1%：
单机全量 AllGather：
300 GB / 300 GB/s = 1 s
delta RDMA 传输：
(300 GB × 1%) / 30 GB/s = 0.1 s
这时全量 local gather 的时间反而是 delta 传输的十倍。因此，NCCL 下仍然建议做 TP-level sharded diff：当 payload 的压缩倍数超过 NVLink 的带宽优势时，单机内部也可能成为瓶颈。
2. Disk：各 PP node 并行写 storage
每个 PP node：单机 TP AllGather → diff → 写 delta 到 storage
PP-level sharded diff 必须做。 各 PP node 只处理自己的权重、写出自己的 delta，避免在一个 node 上保存完整模型 snapshot。
TP-level sharded diff 非必须，可以不做。 单机 NVLink 全量 AllGather 通常远快于写 disk / object storage。沿用示例带宽：
PP stage weight = 200 GB
NVLink AllGather = 300 GB/s
storage write = 5 GB/s
单机全量 AllGather：
200 GB / 300 GB/s ≈ 0.67 s
全量权重写 storage（作为基线）：
200 GB / 5 GB/s = 40 s
10% delta 写 storage：
20 GB / 5 GB/s = 4 s
即使经过 delta 压缩，4 s 的 storage 写入仍然明显慢于 0.67 s 的 local gather。假设把 gather 加速十倍，从约 0.67 s 降到 0.07 s，也只省约 0.6 s，并不能消除 storage 瓶颈。因此，单机全量 AllGather 后再算 diff，通常就够了。
这就是 NCCL 和 Disk 的关键带宽差异。在这些示例中：
NVLink： 数百 GB/s （300 GB/s）
RDMA： 数十 GB/s （30 GB/s）
Disk / storage： 每个 node 几 GB/s （5 GB/s）
NVLink / RDMA 带宽比 = 10 倍
NVLink / storage 带宽比 = 60 倍
payload 缩小十倍，就可能抵消 NVLink 相对 RDMA 的带宽优势；但同样的压缩比例下，storage 仍然明显慢于单机全量 gather。所以 NCCL 建议做 TP-level sharded diff，而 Disk 通常可以不做。
具体的 disk 实现可以参考 Slime 的 Delta Weight Sync 文档：它介绍了对 gather 后的 HF tensor 计算 diff、编码压缩，再通过 disk transport 发布的流程。本文的 PP/TP 建议则是基于上述拓扑和瓶颈的分析。
结论
| 场景 | PP-level sharded diff | TP-level sharded diff |
|---|---|---|
| NCCL：AllGather 到一个 node → broadcast | 必须 | 非必须，建议做 |
| Disk：各 PP node 并行写 storage | 必须 | 非必须 |