MiMo-7B-Base

Summary

• Introduction
• Highlights
• Model Details
• Parameters
• Usage for developers
• Reference
• License
• Citation

Introduction

Currently, most successful RL works, including open-source research, rely on relatively large base models, e.g., 32B models, particularly for enhancing code reasoning capabilities. Moreover, it was widely considered that achieving uniform and simultaneous improvements in both mathematical and code capabilities within a small model is challenging. Nonetheless, we believe that the effectiveness of the RL trained reasoning model relies on the inherent reasoning potential of the base model. To fully unlock the reasoning potential of language models, efforts must focus not only on post-training but also on pre-training strategies tailored to reasoning.

In this work, the author present MiMo-7B, a series of models trained from scratch and born for reasoning tasks. Our RL experiments from MiMo-7B-Base show that our model possesses extraordinary reasoning potential, even surpassing much larger 32B models. Additionally, the author perform RL training on a cold-started SFT model, resulting in MiMo-7B-RL, which demonstrates superior performance on both mathematics and code reasoning tasks, matching the performance of OpenAI o1-mini.

The author open-source MiMo-7B series, including checkpoints of the base model, SFT model, RL model trained from base model, and RL model trained from the SFT model. We believe this report along with the models will provide valuable insights to develop powerful reasoning LLMs that benefit the larger community.

Highlights

• Pre-Training: Base Model Born for Reasoning

  • The author optimize the data preprocessing pipeline, enhancing text extraction toolkits and applying multi-dimensional data filtering to increase reasoning pattern density in pre-training data. The author also employ multiple strategies to generate massive diverse synthetic reasoning data.
  • The author adopt a three-stage data mixture strategy for pre-training. Overall, MiMo-7B-Base is pre-trained on approximately 25 trillion tokens.
  • The author incorporate Multiple-Token Prediction as an additional training objective, which enhances model performance and accelerates inference.

• Post-Training Recipe: Pioneering Reasoning Model

  • The author curate 130K mathematics and code problems as RL training data, which can be verified by rule-based verifiers. Each problem undergoes careful cleaning and difficulty assessment to ensure quality. The author employ only rule-based accuracy rewards to avoid potential reward hacking.
  • To mitigate the sparse reward issue for challenging code problems, The author introduce a test difficulty driven code reward. By assigning fine-grained scores for test cases with varying difficulty levels, the policy can be more effectively optimized via dense reward signal.
  • The author implement a data re-sampling strategy for easy problems to enhance rollout sampling efficiency and stabilize policy updates, particularly in the later phases of RL training.

• RL Infrastructure

  • The author develop a Seamless Rollout Engine to accelerate RL training and validation. Our design integrates continuous rollout, asynchronous reward computation, and early termination to minimize GPU idle time, achieving $2.29\times$ faster training and $1.96\times$ faster validation.
  • The author support MTP in vLLM and enhance the robustness of the inference engine in the RL system.

Model Details

The MTP layers of MiMo-7B is tuned during pretraining and SFT and freezed during RL. With one MTP layer for speculative decoding, the acceptance rate is about 90%.

Model	Description	Download (ModelScope)
MiMo-7B-Base	Base model with extraordinary reasoning potential	🤖️ XiaomiMiMo/MiMo-7B-Base
MiMo-7B-RL-Zero	RL model trained from base model	🤖️ XiaomiMiMo/MiMo-7B-RL-Zero
MiMo-7B-SFT	SFT model trained from base model	🤖️ XiaomiMiMo/MiMo-7B-SFT
MiMo-7B-RL	RL model trained from SFT model, superior performance matching OpenAI o1-mini	🤖️ XiaomiMiMo/MiMo-7B-RL

Parameters

Inputs

• prompt - (text ): User-provided input can be a question, a request, or the context to be addressed. The system uses the prompt to analyze, infer, and generate a corresponding response.

Output

• output - (text): The content generated after the token is the final answer that the model returns to the user.

Usage for developers

Please find below the details to track the information and access the code for processing the model on our platform.

Requirements

pip install -r requirements.txt

Code based on AIOZ structure

import os
from transformers import AutoModelForCausalLM, AutoTokenizer

...

def do_ai_task(
        prompt: Union[str, str],
        model_storage_directory: Union[str, Path],
        device: Literal["cpu", "cuda", "gpu"] = "cpu",
        *args, **kwargs) -> Any:
    """Define AI task: load model, pre-process, post-process, etc ..."""
    # Define AI task workflow. Below is an example

    model_weight_path = os.path.join(model_storage_directory, "MiMo-7B-Base")

    ...

    output = tokenizer.decode(output.tolist()[0])
    return output

Reference

This repository is based on and inspired by XiaomiMiMo's work. We sincerely appreciate their generosity in sharing the code.

License

We respect and comply with the terms of the author's license cited in the Reference section.

Citation

@misc{coreteam2025mimounlockingreasoningpotential,
      title={MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining}, 
      author={LLM-Core-Team Xiaomi},
      year={2025},
      eprint={2505.07608},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.07608}, 
}