Unified sequence parallelism (USP)

Use Ulysses, Ring, or both. Their product is the sequence-parallel degree: ulysses_degree × ring_degree. Set an unused degree to 1.

Ulysses exchanges token shards for head shards, then restores token order

Option Communication and constraints
--ulysses_degree All-to-all exchanges redistribute attention tokens and heads. The degree must divide query and KV head counts unless the model supports head padding.
--ring_degree Ring ranks exchange KV blocks while accumulating attention. Availability depends on the model and selected attention backend.
--attention_backend Select the attention kernel. Model capabilities, device architecture, head shape, and installed packages restrict the choices.
--use_fp8_comms Quantize supported Ulysses exchanges; it does not quantize model weights or every collective. Requires a supported model/backend combination and can change output.

With both methods, the sequence-parallel degree can exceed the head count, provided the Ulysses degree remains valid. Attention size and interconnect determine whether communication costs outweigh the saved compute time. More ranks do not guarantee lower latency.

Sequence parallelism divides attention work. For weight memory limits, combine it with supported FSDP or PipeFusion layouts.

xdit --model FLUX.1-dev \
    --prompt "A cat running in a garden" \
    --ulysses_degree 2 --ring_degree 2

Before running this four-rank example, check that your attention backend supports Ring. xDiT's USP implementation integrates attention backends, Ulysses exchanges, Ring communication, and the patch-aware KV cache needed by PipeFusion. Causal attention and joint text/image layouts depend on the model. For supported FP8 exchanges, see communication options. See the runner launch guide and CLI reference for current flags and backend choices.

The current implementation is in-tree USP. For the method, see the USP paper.