LM-SPT: LM-Aligned Semantic Distillation
for Speech Tokenization

Existing speech tokenizers distill SSL (e.g., HuBERT) features into a semantic quantizer to suppress acoustics and capture content structure, but their high frame rates produce token sequences far longer than text, hindering LM integration. Attempts to reduce the token rate via uniform average pooling tend to over-smooth content-bearing regions and dilute structural information, limiting LM alignment. To address this, we propose LM-SPT, an LM-aligned speech tokenization method based on semantic speech-resynthesis distillation. Instead of directly matching teacher and student features via pooling, LM-SPT resynthesizes speech from semantic tokens only and minimizes the discrepancy between representations extracted from the original and resynthesized waveforms using a frozen, LM-aligned speech encoder. This indirect supervision avoids rigid temporal alignment and encourages dedicated semantic units that are more semantically aligned with LMs under reduced frame rates.