Implement CFG parallelism for a model

CFG parallelism assigns ranks to the positive and negative prediction branches. Use Z-Image as a current example: its runner loads Diffusers ZImagePipeline with xFuserZImageTransformer2DWrapper, and declares use_cfg_parallel=True. FLUX.1-dev's guidance embedding does not create two prediction branches.

Split branch inputs

Diffusers' Z-Image pipeline prepares positive embeddings before negative embeddings. It duplicates latents and timesteps. The transformer wrapper splits x, cap_feats, and t by CFG rank at the start of forward(): rank 0 receives the positive branch, and rank 1 receives the negative branch.

Preserve the pipeline's branch order. Each tensor for a branch must select that same branch. Do not split scalar or broadcast inputs as batch tensors.

Gather before guidance

The wrapper reconstructs sequence-parallel outputs, unpatchifies them, and uses get_cfg_group().all_gather(..., dim=0) to restore the original batch order. The unchanged Diffusers pipeline then computes pos + current_guidance_scale * (pos - neg), applies any configured normalization, and negates the prediction before the scheduler step. Keep this model-specific arithmetic in the pipeline. The wrapper gathers predictions.

Validate the contract

Declare CFG support only after both branch paths work. Compare with the original pipeline using the same batch, seed, guidance, and precision. Check prompt batches, branch order, output shapes, and matching collectives. With Z-Image's wrapper, CFG must remain active throughout the call. Custom pipeline settings such as cfg_truncation can disable a branch during generation. Supporting those settings requires additional branch handling. Z-Image-Turbo does not declare CFG parallel support.

See CFG parallelism for user commands and CFG + USP for simultaneous sequence partitioning.