Implement PipeFusion for a model

中文

Use FLUX.1-dev's stage-based path as the reference. PipeFusion divides transformer blocks between ranks. After synchronous denoising warmup, image patches pass through these stages. This path uses a dedicated pipeline and a composition transformer wrapper. See the PipeFusion guide for user flags and combination restrictions.

Components to adapt

Component Current reference Work for a new model
Runner adapter flux.py Select stage-aware loading and declare only supported options
Pipeline pipeline_flux.py Preserve input preparation; define sync/patch execution, condition transport, scheduler calls, and output conversion
Transformer transformer_flux.py Assign block lists, first/last-stage work, attention processors, and stage payloads
Patch runtime pipefusion/ Reuse layouts, payloads, transport, and schedules where their contracts fit
Scheduler FlowMatch Euler wrapper Update patches without advancing timestep state once per patch

Pipeline execution

Before partitioning inputs, FLUX's __call__() sets runtime input metadata. It prepares prompts, packed image latents, position IDs, and guidance embeddings. These inputs follow the original pipeline's behavior. Guidance embeddings do not create two CFG prediction branches.

_pipefusion_async_enabled() checks whether asynchronous execution is available. The pipeline calls _sync_pipeline() for warmup_steps. It then calls _async_pipeline() for any remaining timesteps. If no steps remain, execution stays synchronous. Pipeline warmup steps belong to one generation. Runner warmup_calls repeats complete generations.

_async_pipeline() constructs a PipeFusionPatchLayout, PipeFusionTransport, and PipeFusionImagePatchSchedule. PipeFusionPatchForward binds shared conditions and per-patch image IDs. Callbacks update patches and assemble the final output. The schedule advances runtime patch state in order. Base pipeline helpers support this work; each model still implements its execution loop.

Model-specific requirements include:

  • Keep image tokens, position IDs, and patch sizes aligned. FLUX uses token-dimension patches; other pipelines may use spatial layouts.
  • Define the condition tensors that travel between stages. Specify whether later patches reuse the first patch's condition. FLUX sets condition_reuse=True.
  • Preserve supported callback and interruption behavior. The asynchronous runtime rejects callback_on_step_end. Document this restriction.
  • Reassemble latents and apply model-specific scaling and shifting. Run VAE decoding on the ranks required by the selected path. A fixed output rank alone cannot support parallel VAE.

Transformer stages and attention

xFuserFluxTransformer2DWrapper registers for FluxTransformer2DModel. It divides transformer_blocks and single_transformer_blocks between stages. The transformer base slices these lists using attn_layer_num_for_pp, or divides blocks evenly by default. It retains checkpoint metadata for stage-aware loading.

The first stage applies image and context input projections. The last stage applies output normalization and projection. Intermediate stages return image and condition state. Match these payloads to the pipeline's forward and transport interfaces. Register new attention processors or layer wrappers where needed.

Patch attention uses current K/V for the active patch and cached K/V for other patches. The FLUX processor, USP, and cache manager implement this behavior. Preserve cache registration, patch indices, and attention-module handles. Update each cache in one place, rather than in both the processor and USP.

Scheduler and loading

The FlowMatch Euler scheduler advances its step index after the last patch or a full synchronous update. A new scheduler can also require per-patch history. Preserve its initialization and reset behavior. Without registration, xDiT logs a warning and retains the original scheduler. This fallback does not establish PipeFusion compatibility.

Stage-aware loading is part of runner integration. FLUX uses plan_pipefusion_components() and records block ownership when available. Declare LoadSupport for routes the model implements. Composition wrappers and transformer subclasses have different construction and loading interfaces.

Validation

Start with synchronous stage execution. Then enable multiple patches after warmup. Check stage splits, patch order, condition reuse, KV-cache reset, scheduler state, and final latent assembly. Test each supported combination with USP, VAE, caching, loading, or compilation.

The PipeFusion unit tests cover runtime interfaces. Complete inference and output-quality comparisons require suitable accelerators. During patch execution, PipeFusion can reuse KV from the previous step. Results can differ from full-image attention. Performance depends on the workload.