Most audio-video generators give you one soundtrack, not the individual sounds inside it. SepGen changes that: one model can generate a scene and hand back a separate waveform for every captioned source. A four-dimensional audio-visual scene combines video, dynamic geometry, and sound sources, each with its own position and trajectory.

Rendering that scene from a new viewpoint requires every source as an individual waveform, so each source can be localized and propagated before the signals are mixed. Joint audio-video generators can synthesize video and sound, but they emit one audio-mix in which the sources are not individually accessible.

Recent joint audio-video generators synthesize video with synchronized sound, but output the entire scene as one waveform whose sources cannot be moved, removed, or rerendered individually. Language-queried separation can estimate a described source afterward, but it requires an additional model run and handles each caption in isolation.

That isolation can fail when two sources share a category: asking separately for two men speaking might return the same speaker twice. Per-source waveforms matter most when the scene itself is rerendered. Monocular video methods lift video to a four-dimensional scene viewed from a new camera pose, while spatial audio renderers propagate each source to the observer.

An audio-mix recorded at the original pose exposes no source to place, so the sound stays fixed. SepGen takes a scene caption and one caption per source, then produces the video, the audio-mix, and one waveform per source, called a stem, in a single sampling run.

Predicting both stems jointly from the shared audio-mix lets each follow its own caption while sharing the scene's timing. This also yields the per-source waveforms that a spatial renderer requires. Adding stems to a pretrained backbone must not disturb what the backbone already produces.

SepGen appends the two stems to the backbone's audio sequence as additional channels, trained through a low-rank audio adapter. The audio-mix remains a protected channel that attends only to itself and bypasses the adapter, so the backbone's soundtrack and video are unchanged.

The stems attend to the audio-mix and to each other at aligned rotary positions, while block-diagonal caption routing gives each stem only its own caption. Cross-Stem Attention Guidance steers each stem away from its sibling's.

The same model can either generate a scene together with its stems or separate the stems from an observed scene. The audio-mix noise level selects between the modes: separation holds the observed audio-mix at zero noise, while generation denoises it together with the stems.

Training starts with separation and briefly continues on noisy audio-mixes, yielding one checkpoint that does both. SepGen is presented as a framework that turns a pretrained joint audio-video generator into a multi-stem generator, emitting video, its soundtrack, and one waveform per captioned source in a single sampling run.

Its mechanisms include a protected audio-mix channel, aligned rotary positions, block-diagonal caption routing, and a schedule in which stems attend to the estimated clean audio-mix. A single set of weights supports both modes through the audio-mix noise level, while a brief continuation reduces leakage between generated stems, improves speech intelligibility, and stabilizes results across seeds while preserving separation.

The evaluation covers generated, cross-generator, and real scenes, using automatic metrics and a human A/B study. SepGen extends a joint audio-video generator so that, alongside the video and its soundtrack, it outputs one waveform per sound source. That enables a four-dimensional audio-visual scene renderer to place each source.

A source can be placed in space over time, and SepGen can also act as a source separator given an existing video. In generation, the model synthesizes video, the audio-mix, and stems from conditioning text; in separation, video and audio-mix are inputs and SepGen extracts the captioned stems.

In the top row, the single mixed waveform leads to a four-dimensional scene with “no source positions,” and the rerendered result shows the same mix at both ears. In the bottom row, the orange jeep and blue taxi tracks remain separately visible through localization, three-dimensional source trajectories, and distinct left–right spatial audio at the new viewpoint.

The base system is a flow matching diffusion audio-video generator with two transformer streams: one for video tokens and one for audio tokens. Each stream self-attends over its own tokens and cross-attends to the conditioning scene caption, while the streams exchange information through bidirectional audio-video cross-attention.

Jointly denoising and decoding both latents yields a video and its audio-mix. SepGen is formulated for scenes with N sound sources described by per-source captions, and training and evaluation use two sources. The model keeps both base streams and represents each source as additional tokens in the audio stream; the audio VAE maps each stem and the audio-mix to latents and maps each latent back to a waveform.

The three latents form one audio token sequence, and each channel keeps the pretrained temporal positions so tokens at the same index refer to the same time instant. Each stem attends to itself, the other stems, the audio-mix, the video, and only its own caption, while the audio-mix and video do not attend to the stems.

A low-rank adapter updates audio-stream projections on stem tokens only, while the weights of the base generator stay frozen. In generation, SepGen samples the video, audio-mix, and one stem per source from the captions alone. In separation, it receives an observed video and audio-mix and returns only the stems.

Separation keeps the video and audio-mix clean through sampling, while generation denoises them together with the stems. In the left panel, the scene caption feeds the video and audio-mix, while each source caption points to its corresponding stem; the arrows converge through one shared prior.

In the right panel, the observed video and mixture enter alongside two target captions, and the output region contains only the two separated waveforms, with the mixture held fixed. Each training example contains a video, its audio-mix, both source waveforms, a scene caption, and a caption per source.

The dataset contains four thousand eight hundred forty-three examples from CelebV-HQ, URMP, generated two-source scenes, and synthetic audio-mixes combining VGGSound videos and MUSIC-21 solo performances. The two sources may sound at the same time or alternate, as in a dialogue where speakers take turns.

Stage one trains SepGen for separation: given an observed video, its audio-mix, and two source captions, it predicts the two stems. The low-rank adapter starts at zero, so training begins from the behavior of the base generator. The audio-mix is always clean in this stage, and the stems learn to decompose it into the sources described by the two captions.

Although this stage trains only separation, its checkpoint can already generate stems by attending to the estimated clean audio-mix, and stage two improves that ability. In generation, no clean audio-mix is available: the audio-mix is denoised together with the stems, so the stems attend to an audio-mix that is still noisy.

In both modes, the stems decompose the audio-mix they attend to into the captioned sources. The second stage starts from the separation checkpoint and continues with the same data and loss, changing only how the audio-mix noise level is drawn.

In both operation modes, Cross-Stem Attention Guidance applies Normalized Attention Guidance inside each stem's caption cross-attention, using the sibling caption as the negative branch at scale two. The evaluation compares both SepGen modes with cascade baselines that apply a language-queried separator to the same audio-mix.

It asks whether generation produces caption-faithful stems without cross-stem leakage, and whether the same model separates real and externally generated scenes after generation training. The study also reports a user study and ablations. For generation, the benchmark contains one hundred ninety-four two-source scene prompts: one hundred thirty-four sound scenes and sixty two-speaker dialogues.

For separation, the rendered-scene benchmarks contain one hundred forty-nine Veo three point one Lite scenes and sixty-four LTX two point three scenes, plus one hundred seventy-two real recordings. The comparisons include FlowSep, AudioSep, and two released SAM Audio checkpoints, SAM large and SAM large-tv.

Table one compares SepGen's joint-run stems with self-separation, which runs the same weights again over the generated video and audio-mix using the same captions. The joint-run stems lead every cascade baseline on every metric, except for the reported CLAP comparisons against FlowSep and AudioSep.

The widest lead is in J-swap: jointly generated stems stay bound to their captions, while an isolated cascade query can return the same source twice. Self-separation raises every sounds column but costs a second sampling run, while joint-run stems keep a lower own-line word error within seed variation.

The summed stems also match the audio-mix more closely than the summed stems from any cascade. The visual’s clearest separation is in the J-swap column: the two SepGen rows sit at two point zero seven and two point two nine, while the cascade rows range only from zero point four one to one point two seven.

In the speech block, the bold and underlined formatting draws attention to the small difference between zero point zero seven and zero point one zero in WER. The evaluation uses one set of weights for both generation and separation, selected by the audio-mix noise level, and reports automatic metrics together with a human A/B study.

Figure three compares three inference schedules in sigma-space. In separation, only the stems are denoised while the audio-mix remains clean; shared-noise generation denoises the stems and mix together at one noise level. SepGen instead has stems attend to the noisy mix above sigma g, then switch below sigma g to the estimated clean mix, with sigma g set to zero point nine seven.

SepGen turns a single audio-video generator into a joint multi-stem system: it can generate or separate captioned sources while preserving the shared scene timing, and its joint-run stems beat cascade baselines across the reported generation metrics.