Parallel VAE
The VAE converts between images or video and model latents. Its activations can be a memory bottleneck at high resolution. xDiT uses DistVAE to distribute supported VAEs.
Start with row sharding
With tiling disabled and a supported VAE, --use_parallel_vae enables row
sharding. Supported decoder layers process local rows. Convolutions exchange
boundary rows. GroupNorm shares statistics where required.
Some work remains replicated. The 2D adapter starts sharding after the input convolution and mid-block. Row sharding aims to reduce activation memory while matching unsharded decoding within numerical tolerance.
xdit --model FLUX.1-dev \
--prompt "A spectacular sunset over the ocean" \
--ulysses_degree 2 --use_parallel_vae
This uses two ranks for both transformer sequence parallelism and VAE decoding. The runner reuses tensor, sequence, pipeline, and CFG ranks within each data-parallel group. The flag does not add ranks. For models with parallel encoding support, the same flag also distributes encoding. There is no separate encoder CLI flag.
Enable tiling when needed
Tiling is a separate memory option, useful on one GPU as well as with parallel VAE. It decodes overlapping spatial windows instead of the full image or frame.
| Options | Decode path |
|---|---|
--use_parallel_vae |
Row sharding when tiling is disabled and the VAE supports row adapters. |
--enable_tiling |
Local tiled decoding, subject to the VAE's tiling threshold. |
| Both | Whole tiles are assigned to ranks when the VAE has a supported tile adapter; otherwise supported decoders shard rows within each tile. |
--enable_slicing |
Slice the batch where supported. This is independent of spatial tiling. |
Whole-tile distribution avoids per-layer row exchange. Overlapping tiles repeat some decoder work. Tile-local normalization and blending can change the output. Memory use depends mainly on the tile window, rather than the rank count. If there are fewer tiles than ranks, DistVAE falls back to decoding the full tile grid locally on every rank. Smaller tiles do not always reduce peak memory, especially for video VAEs with temporal state outside the spatial window.
MiniMax-H3 uses native tile dispatch instead of DistVAE row adapters. LTX-2's
diffusion decoder enables native tiling with parallel VAE. Some pipelines enable
VAE tiling during loading. These pipelines use tiling without an explicit
--enable_tiling flag.
Set an exact tile window
| Options | Units and requirements |
|---|---|
--vae_tile_size_height, --vae_tile_size_width |
Positive output-pixel dimensions; supply both with --enable_tiling. |
--vae_tile_overlap_height, --vae_tile_overlap_width |
Non-negative output-pixel overlap; supply both with --enable_tiling. Use zero on an axis that spans the output without tiling. |
The window and overlap must map exactly to the VAE's latent scale and strides. xDiT rejects unsupported geometry. Clipped edge tiles must also be large enough for decoder padding. Row sharding needs a valid split with at least one latent row per rank. Whole-tile dispatch does not have this row-split requirement.
For example, this two-device FLUX.2 Klein configuration uses full-width strips:
xdit --model black-forest-labs/FLUX.2-klein-4B \
--prompt "A spectacular sunset over the ocean" \
--height 2048 --width 2048 \
--num_inference_steps 4 --guidance_scale 1.0 \
--ulysses_degree 2 \
--use_parallel_vae --enable_tiling \
--vae_tile_size_height 352 --vae_tile_size_width 2048 \
--vae_tile_overlap_height 88 --vae_tile_overlap_width 0
The geometry comes from AMD's RDNA 4 recipe. Its additional precision, attention, and loading settings require the matching hardware and software environment.
Compare row sharding, tiling, and unsharded decoding on your workload with fixed
seeds. Measure end-to-end peak memory and latency. Inspect output quality.
VAE weights and final output assembly still consume memory on each rank.
DistVAE's decode strategy guide
and tile-window guide
explain the family-specific tradeoffs. See the runner guide
for model support. The xdit CLI uses existing model ranks for parallel VAE.