Neural generative audio

1.1 Open song generators

Models that write whole songs. Suno and Udio, etc. I am somewhat less excited about this category, but it is very hyped, so it needs mentioning.

Anyway, if there are to be models that write whole songs, I guess it’s best that there are open ones, from which we might learn.

YuE (乐, Chinese for “music” and “happiness”), out of HKUST and M‑A‑P, is a token‑LM model. A 7B model generates xcodec tokens from a genre‑tag‑plus‑lyrics prompt, a 1B stage refines, and out come several‑minute-long songs with separate vocal and accompaniment stems across English, Mandarin, Cantonese, Japanese, and Korean (Yuan et al. 2025). It is Apache‑2.0, which here means open in the sense that we can use the output commercially, not the “non‑commercial output usage” sense Stable Audio uses for “open” weights. It is slow — 30 s of audio runs about 150 s on an H800.

ACE‑Step (code, also Apache‑2.0) is a neural diffusion model. The authors assert that LLM song models get the lyrics aligned but run slowly and lose long‑range structure, so ACE‑Step pairs a deep‑compression autoencoder with a linear transformer and turns out roughly 4 minutes of music in about 20 s on an A100, some 15× faster. The stated aim is “the Stable Diffusion moment for music.” Since the model exposes readily accessible latent spaces, a number of cool features follow (as with image diffusions): voice cloning, repainting, lyric editing, lyric‑to‑vocal, accompaniment‑from‑vocal.

1.2 Unified audio‑text models

Everything above is audio‑out only: we prompt with text, a waveform comes back. But we can be more generally multi-modal.

Nemotron‑Labs‑Audex‑30B‑A3B is NVIDIA’s audio-text-pansexual design. [TODO clarify] Take a text LLM — Nemotron‑Cascade‑2‑30B‑A3B. Splice an audio encoder onto the input side which projects into the text embedding space. Extend the output vocabulary with the tokens of a neural codec. Then train the single decoder on all of it, so that text tokens and audio tokens are the same sort of object as far as the generation loop is concerned. This all happens, to be clear, in a unified architecture (almost — see below): what we are running is still a decoder emitting tokens. Out of that falls audio understanding, speech recognition, speech translation, text‑to‑speech, text‑to‑audio and speech‑to‑speech, plus the backbone’s thinking modes and 1M‑token context.

The paper is titled Unified Audio Intelligence Without Regressing on Text Intelligence (Kong et al. 2026). It’s surprising that they pulled this off without losing all that text facility.

Cons: As usual it has NVIDIA’s shitty Oneway Noncommercial license.

Also there was some sleight-of-hand in that “unified model” pitch up there. In fact the audio output path is not self‑contained, because the model emits codec tokens and something downstream has to turn those into sound: text‑to‑audio decodes through XCodec1 plus an optional 48 kHz enhancement VAE, text‑to‑speech through either a standalone causal speech decoder (streaming, and the default) or the original XCodec2 (better, not streaming).

1.3 In the DAW

OBSIDIAN Neural is a free AGPL‑3.0 VST3 / AU plugin (Windows / macOS / Linux) which wraps Stable Audio Open with MIDI triggering and tempo sync, plus several specialized fine‑tunes. The repository is still at innermost47/ai-dj under its old name; the homepage and README reflect the rebrand. It requires “30 seconds of patience per loop.”

A different approach is found in RAVE (code), Caillon and Esling’s real‑time variational autoencoder from IRCAM’s ACIDS group (Caillon and Esling 2021). Instead of prompting a big pretrained model and waiting, we train RAVE on our own corpus — a voice, a drum kit, a field recording, all of the above — and it learns a compact latent space to traverse live. Exported in streaming mode, it runs inside Max/MSP or PureData via the nn~ external at low enough latency that a Raspberry Pi 4 keeps up in real time (so I am told), and there is now a RAVE VST for grown-up DAWs. Because the latent is explicit, this makes it amenable to timbre transfer and haptic manipulation — the model as a playable instrument.

2.1 2016–18 — Raw waveform, sample by sample

WaveNet (DeepMind) and SampleRNN (Mehri et al. 2017) modelled audio one sample at a time with autoregressive networks. NSynth / WaveNet autoencoders (Engel et al. 2017) applied the idea to musical timbre. Dadabots’ SampleRNN metal albums are the entertaining proof‑of‑concept.

Sander Dieleman’s 2020 essay on waveform‑domain synthesis is still the best orientation piece for this era.

2.2 2018–20 — Alternatives to autoregression

Magenta’s DDSP (code) embedded classical synthesis modules — oscillators, filters, reverb — as differentiable layers, so we get parameter inference instead of waveform regression. The timbre transfer demo still demos well, even though Magenta itself has stagnated.

GANSynth used GANs over spectrogram representations; WaveGAN did the same for raw waveforms. MelNet went big on conditional spectrogram modelling. OpenAI’s Jukebox was the most ambitious of the era — a hierarchical VQ‑VAE plus autoregressive transformer trained on raw music — and is mostly historical now, superseded by latent diffusion.

2.3 2020–22 — Diffusion arrives

WaveGrad (N. Chen et al. 2020) and DiffWave (Kong et al. 2021) demonstrated denoising diffusion on raw audio. SaShiMi (Goel et al. 2022) showed that structured state‑space models could model raw audio as well as anything autoregressive — see the examples and code. NU‑Wave (Lee and Han 2021) and Pascual et al (Pascual et al. 2022) applied diffusion to upsampling and full‑band synthesis respectively.

2.4 2022–23 — Text conditioning

CLAP (Wu et al. 2023) gave us a contrastive joint embedding for text and audio, which is what most of the text‑to‑audio models use under the hood. Meta’s AudioGen (Kreuk, Synnaeve, et al. 2022) and Google’s MusicLM got the text‑to‑audio and text‑to‑music ball rolling. MusicGen (Copet et al. 2023) (the AudioCraft one) collapsed the cascade into a single transformer LM over EnCodec tokens.

2.5 2023–24 — Latent diffusion at scale

AudioLDM (H. Liu, Tian, et al. 2023) and AudioLDM 2 (H. Liu, Chen, et al. 2023) moved to latent diffusion, with AudioLDM 2’s “language of audio” abstraction unifying speech, music, and SFX. MusicLDM (K. Chen et al. 2023) added music‑specific tricks like tempo‑aware conditioning. Mousai (Schneider et al. 2023) is a similar latent‑diffusion approach. Stable Audio (Evans et al. 2024) went stereo and longer (47 s).

I’m not massively into spectral‑domain synthesis because I think the stationarity assumption is a stretch (heh). Or rather, my contrarian instinct says that working in the Fourier domain leaves audio quality on the table — the transient attack on percussion, the way a struck string rings out, that kind of thing — even though latent‑diffusion systems, for example, work on spectral‑adjacent latents and apparently get away with it.

2.6 2024–26 — Open weights and DAW integration

Stable Audio Open released the weights publicly. YuE and then ACE‑Step did the same for full songs with vocals, both under Apache‑2.0 — the first open weights to seriously rival Suno. OBSIDIAN Neural and friends started bringing model‑in‑plugin workflows into actual DAWs, with all the live‑performance implications that entails.

3.1 Latent diffusion

A VAE compresses waveforms into a much lower‑rate latent space; diffusion runs in the latent space; conditioning is by text embedding (CLAP, T5) cross‑attended into the diffusion transformer or U‑Net. Stable Audio, AudioLDM, MusicLDM, and Mousai are all variations on this theme. See neural diffusion for the underlying machinery.

3.2 Token‑based language models

Encode audio as discrete tokens with a neural codec (EnCodec, SoundStream, xcodec); train a transformer LM to predict the tokens autoregressively; decode the predicted tokens back to audio. MusicGen and AudioGen are archetypal examples.

Choice of codec turns out to be important here. Codecs tend to specialize on general waveforms, or on speech specifically. The X‑Codec team Ye et al. (2024) argues that the former case is leaving speech semantic information unexploited.

3.3 Differentiable DSP

DDSP threads a different needle: keep the classical synthesis topology (oscillator, filter, reverb), make the parameters differentiable, learn parameter trajectories from audio. The output is then by construction a real synthesis chain, not a regressed waveform.

3.4 State‑space models

SaShiMi (Goel et al. 2022) and successors, as seen in the state‑space models notebook, are a different way of modelling long sequences. Not currently the state of the art for music generation (which surprises me, TBH — so many connections to traditional synthesis methods) but a tidy alternative to attention for long sequences.