Neural codecs and compression algorithms

1 Audio codecs

Neural networks are in audio codecs now. Originally they looked attractive for their compact bitrates, as in the Descript Audio Codec (code) (Kumar et al. 2023) which claimed a 90× compression factor for CD-quality audio.

We might also want the encodings to be a convenient space for generative audio models to predict over. In which case, we want the tokens to be “semantically meaningful” rather than just “spectrally accurate”.

X-Codec (Ye et al. 2024) attempts to achieve this. Alongside the usual acoustic encoder it runs a frozen self-supervised speech model — HuBERT or WavLM — and folds its features in before quantization, so the residual codebooks carry semantic content rather than only spectral detail. Fancy new generative models like YuE generate over X-Codec tokens, for example.

transformers now ships it as a first-class XcodecModel, so trying it is easy:

from transformers import XcodecModel, AutoFeatureExtractor

model = XcodecModel.from_pretrained("hf-audio/xcodec-hubert-general-balanced")
extractor = AutoFeatureExtractor.from_pretrained("hf-audio/xcodec-hubert-general-balanced")
codes = model.encode(audio, bandwidth=4)   # 8 codebooks; 0.5 kbps gives 1

The checkpoints differ in which semantic model they wrap and what they saw:

NB: XCodec2 is a separate, later model, not a checkpoint in the table above. Audex uses the two for different jobs — XCodec1 for text-to-audio, XCodec2 for speech — which is easy to conflate when reading its setup instructions.

2 Images via vector embeddings

Thanks to the success of vector embeddings we can frequently represent images in a weird vector space which looks suggestively like classic encodings. In NLP, though, the embeddings are usually larger than the original text, so can we actually compress this way?

Maybe. There have been a bunch of works in that domain recently (Duggal et al. 2024; Miwa et al. 2025; Yan et al. 2025). The most viral one is (Bachmann et al. 2025), which has an elegant demonstration.

• FlexTok: Resampling Images into 1D Token Sequences of Flexible Length

3 References