HeartMuLa: A Family of Open Sourced Music Foundation Models

Summary

• Introduction
• Parameters
• Usage for developers
• Reference
• License
• Citation

Introduction

HeartMuLa is a family of open sourced music foundation models including:

1. HeartMuLa: a music language model that generates music conditioned on lyrics and tags with multilingual support covering almost all languages.
2. HeartCodec: a 12.5 Hz music codec with high reconstruction fidelity;
3. HeartTranscriptor: a whisper-based model specifically tuned for lyrics transcription; Check this page for its usage.
4. HeartCLAP: an audio–text alignment model that establishes a unified embedding space for music descriptions and cross-modal retrieval.

Below shows the experiment result of our oss-3B version compared with other baselines.

Parameters

Inputs

• lyrics - (text): The textual content used as input for audio generation. This can be a full song, short phrases, or descriptive text guiding the generated audio.
• tags - (text): A comma-separated list of keywords describing the desired style, mood, or instrumentation of the audio. Example: piano, happy, romantic, synthesizer
• max_duration_seconds - (integer): The maximum length of the generated audio in seconds. Range: 30 to 240, with a step of 10.
• temperature - (float): Controls the randomness of the generation. Range: 0.1 to 2.0, step 0.1
• topk - (integer): Limits the number of highest-probability tokens considered during sampling. Range: 1 to 100, step 1
• cfg_scale - (float): Classifier-Free Guidance scale. Controls how strongly the model follows the input conditions (lyrics and tags). Range: 1.0 to 3.0, step 0.1

Output

• output_file - (audio -.mp3): The generated audio file based on the provided inputs. The output reflects the given lyrics, style tags, and generation parameters, with a duration up to the specified maximum length.

Usage for developers

Please find below the details to track the information and access the code for processing the model on our platform.

Requirements

pip install -r requirements.txt 

Code based on AIOZ structure

import os
import tempfile
import torch
from heartlib import HeartMuLaGenPipeline

...

def do_ai_task(
        lyrics: str,
        tags: str,
        max_duration_seconds: int,
        temperature: float,
        topk: int,
        cfg_scale: float,
        model_storage_directory: Union[str, Path],
        device: Literal["cpu", "cuda", "gpu"] = "cpu",
        *args, **kwargs) -> Any:
    """Define AI task: load model, pre-process, post-process, etc ..."""
    # Define AI task workflow. Below is an example
    model_dir = model_storage_directory + "/heartmula_models"
    # Determine device and dtype
    if torch.cuda.is_available():
        device = torch.device("cuda")
        dtype = torch.bfloat16
    else:
        device = torch.device("cpu")
        dtype = torch.float32

    pipe = HeartMuLaGenPipeline.from_pretrained(
        model_dir,
        device=device,
        dtype=dtype,
        version="3B",
    )
    ...

    return audio_bytes

Reference

This repository is based on and inspired by HeartMuLa's work. We sincerely appreciate their generosity in sharing the code.

License

We respect and comply with the terms of the author's license cited in the Reference section.

Citation

@misc{yang2026heartmulafamilyopensourced,
      title={HeartMuLa: A Family of Open Sourced Music Foundation Models}, 
      author={Dongchao Yang and Yuxin Xie and Yuguo Yin and Zheyu Wang and Xiaoyu Yi and Gongxi Zhu and Xiaolong Weng and Zihan Xiong and Yingzhe Ma and Dading Cong and Jingliang Liu and Zihang Huang and Jinghan Ru and Rongjie Huang and Haoran Wan and Peixu Wang and Kuoxi Yu and Helin Wang and Liming Liang and Xianwei Zhuang and Yuanyuan Wang and Haohan Guo and Junjie Cao and Zeqian Ju and Songxiang Liu and Yuewen Cao and Heming Weng and Yuexian Zou},
      year={2026},
      eprint={2601.10547},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2601.10547}, 
}