Phi-4

Summary

• Introduction
• Model Overview
• Intended Use
• Benchmark Performance
• Parameters
• Usage for Developers
• Reference
• License
• Citation

Introduction

Microsoft has released Phi-4, a 14-billion-parameter dense decoder-only Transformer language model built on a curated blend of synthetic datasets, filtered public-domain web data, and licensed academic books and Q&A corpora. Its training recipe emphasizes data quality and advanced reasoning over raw scale, allowing the model to reach performance levels typically associated with much larger systems. Key characteristics include:

• Quality-First Data Curation: Training relies on a blend of textbook-quality synthetic data covering mathematics, code, and chain-of-thought reasoning, combined with filtered public-domain documents and acquired academic Q&A datasets, with approximately 8% multilingual coverage.
• Strong Reasoning at 14B Scale: On OpenAI's SimpleEval suite, Phi-4 matches or exceeds substantially larger open and proprietary models on GPQA (science) and MATH (mathematics), while maintaining competitive scores on MMLU, HumanEval, and DROP.
• Alignment for Instruction Following: The base model is further refined with supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) to improve instruction adherence, response formatting, and safety behavior.

Model Overview

Phi-4 has the following characteristics:

• Type: Causal Language Model (decoder-only Transformer)
• Architecture Lineage: Minimal changes from the Phi-3 architecture
• Number of Parameters: 14B
• Context Length: 16,384 tokens
• Training Tokens: ~9.8T tokens
• Training Compute: 1,920 NVIDIA H100-80GB GPUs over 21 days
• Training Window: October–November 2024
• Data Cutoff: June 2024 (publicly available data)
• Primary Language: English (with limited multilingual coverage)
• Chat Template: ChatML-style tokens (<|im_start|>, <|im_sep|>, <|im_end|>)
• Release Date: December 12, 2024

Intended Use

Primary Use Cases

Phi-4 is designed to accelerate research on language models and to serve as a building block for generative AI features in general-purpose systems and applications (primarily English). It is well-suited for:

1. Memory- and compute-constrained environments.
2. Latency-sensitive deployments.
3. Tasks requiring reasoning, logic, and structured problem solving.

Out-of-Scope Use Cases

Phi-4 has not been specifically designed or evaluated for all downstream purposes. Developers should observe the following:

1. Evaluate and mitigate for accuracy, safety, and fairness before adopting the model in any downstream use case, particularly in high-risk scenarios.
2. Comply with applicable laws and regulations (including privacy and trade compliance) relevant to the deployment context, noting the model's English-language focus.
3. The contents of this model card do not constitute a modification of, or exception to, the license under which the model is released.

Benchmark Performance

The following results compare Phi-4 against representative open and proprietary models on OpenAI's SimpleEval benchmark suite. Higher values indicate better performance.

Category	Benchmark	Phi-4 (14B)	Phi-3 (14B)	Qwen 2.5 (14B Instruct)	GPT-4o-mini	Llama-3.3 (70B Instruct)	Qwen 2.5 (72B Instruct)	GPT-4o
Popular Aggregated Benchmark	MMLU	84.8	77.9	79.9	81.8	86.3	85.3	88.1
Science	GPQA	56.1	31.2	42.9	40.9	49.1	49.0	50.6
Math	MGSM MATH	80.6 80.4	53.5 44.6	79.6 75.6	86.5 73.0	89.1 66.3*	87.3 80.0	90.4 74.6
Code Generation	HumanEval	82.6	67.8	72.1	86.2	78.9*	80.4	90.6
Factual Knowledge	SimpleQA	3.0	7.6	5.4	9.9	20.9	10.2	39.4
Reasoning	DROP	75.5	68.3	85.5	90.2	90.2	76.7	80.9

Note: Scores marked with * differ from those reported by Meta. The SimpleEval framework imposes strict response-formatting requirements that Llama models struggle to satisfy. SimpleEval is used here for reproducibility; Meta reports 77.0 on MATH and 88.0 on HumanEval for Llama-3.3-70B.

Parameters

Input

• input_prompt (text): User-provided input (a question, instruction, or context) that the model analyzes and conditions on to generate a response.

Output

• output (text): The text generated by the model in response to the input prompt.

Usage for Developers

The following details describe how to access and run the model on our platform.

Requirements

pip install -r requirements.txt

Code based on AIOZ structure

import torch
import transformers

...

def do_ai_task(
        input_prompt: str,
        model_storage_directory: Union[str, Path],
        device: Literal["cpu", "cuda", "gpu"] = "cpu",
        *args, **kwargs) -> Any:
    """Define AI task: load model, pre-process, post-process, etc."""
    # Define AI task workflow. Below is an example.
    messages = [
        {"role": "system", "content": "You are a medieval knight and must provide explanations to modern people."},
        {"role": "user", "content": input_prompt},
    ]

    ...

    outputs = pipeline(messages, max_new_tokens=128)
    output = outputs[0]["generated_text"][-1]

    return output

Reference

This repository is based on and inspired by Microsoft's work. We sincerely appreciate their generosity in sharing the model and accompanying technical report.

License

We respect and comply with the terms of the original author's license cited in the Reference section.

Citation

@techreport{abdin2024phi4,
  title        = {Phi-4 Technical Report},
  author       = {Abdin, Marah and Aneja, Jyoti and Behl, Harkirat and Bubeck, S{\'e}bastien and Eldan, Ronen and Gunasekar, Suriya and Harrison, Michael and Hewett, Russell J. and Javaheripi, Mojan and Kauffmann, Piero and Lee, James R. and Lee, Yin Tat and Li, Yuanzhi and Liu, Weishung and Mendes, Caio C. T. and Nguyen, Anh and Price, Eric and de Rosa, Gustavo and Saarikivi, Olli and Salim, Adil and Shah, Shital and Wang, Xin and Ward, Rachel and Wu, Yue and Yu, Dingli and Zhang, Cyril and Zhang, Yi},
  institution  = {Microsoft Research},
  year         = {2024},
  eprint       = {2412.08905},
  archivePrefix= {arXiv},
  primaryClass = {cs.CL},
  url          = {https://arxiv.org/abs/2412.08905}
}