Imagine changing the camera angle in a video and having the sound change with it. That only works if every voice and sound exists on its own track—and this system creates those tracks as the scene is made. A scene has moving pictures, changing geometry, and sound sources, each with its own position and path.

To hear the scene from a new viewpoint, every source has to remain available as an individual waveform before the sounds are mixed. Here is the everyday problem: imagine filming a conversation and saving only one finished audio track. You can replay it, but you cannot move one speaker, remove another, or place either voice somewhere new.

Earlier joint audio-video generators could make the video and soundtrack, but that soundtrack was one audio mix, with no individual sources available. SepGen extends an audio-video generator so that, from a scene description and one description for each source, it produces the video, the mixed soundtrack, and one waveform for every source in one run.

The source waveforms are predicted together from the shared mix. That lets each one follow its own description while sharing the scene’s timing, producing the separate waveforms needed to place sound in space. It makes the missing step visible: instead of one unplaceable soundtrack, separate source tracks follow the jeep and taxi into the new view and reach the listener from their positions.

The system is designed to preserve the original generator’s video and soundtrack while keeping each added source faithful to its own description. The same set of learned settings supports both making separate tracks and pulling described sounds out of an existing mix.

A short continuation reduces leakage between generated tracks, improves speech intelligibility, and keeps results steadier while preserving separation. Each separate track can listen to the shared mix, but information from the separate tracks is blocked from flowing back into that mix.

This protects the original soundtrack while allowing the sources to use shared timing and sound information. The separate tracks are kept as distinct channels alongside the mix, rather than replacing it. The evaluation compares both uses of SepGen with systems that first make a mix and then ask another separator to find each described source.

The tests ask whether generated tracks follow their descriptions without leaking into one another, and whether the same model can separate real and externally generated scenes after learning to generate them. When the tracks are produced during the same run as the video and mix, they lead the comparison systems across the reported measures.

The largest advantage appears when two tracks must stay bound to their own descriptions, because separate queries can return the same source twice. Running the model again in separation mode improves some sound measures but costs a second run.

The tracks from the joint run keep lower word error for the intended spoken line, and together they match the mixed soundtrack more closely than the comparison systems’ tracks. In listening studies, people watched the reference video, heard two unnamed candidate tracks for each described source, and chose the better track or a tie.

For separation, listeners chose SepGen in fifty-eight percent of judgments, compared with twenty-five percent for the comparison system. For generation, they chose SepGen in sixty-three percent, compared with twenty-eight percent.

There are important limits. The current layout is fixed at two sources, and the decoder rebuilds part of each waveform’s timing detail, so direct waveform comparisons also measure that rebuilding step. The separate tracks retain the original scene’s room sound and distance cues.

Before moving each source to a new viewpoint, a spatial renderer must account for those cues. SepGen gives generated scenes separate, caption-guided sound tracks instead of one inseparable mix, and listeners preferred those tracks in both creating and unpacking scenes.

That could make viewpoint changes in immersive video sound natural.