LongBench

Summary

• Introduction
• Dataset Structure
• Task statistics
• Task description
• LongBench-E statistics
• Reference
• License
• Citation

Introduction

LongBench is the first benchmark for bilingual, multitask, and comprehensive assessment of long context understanding capabilities of large language models. LongBench includes different languages (Chinese and English) to provide a more comprehensive evaluation of the large models' multilingual capabilities on long contexts. In addition, LongBench is composed of six major categories and twenty-one different tasks, covering key long-text application scenarios such as single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion.

We are fully aware of the potentially high costs involved in the model evaluation process, especially in the context of long context scenarios (such as manual annotation costs or API call costs). Therefore, we adopt a fully automated evaluation method, aimed at measuring and evaluating the model's ability to understand long contexts at the lowest cost.

LongBench includes 14 English tasks, 5 Chinese tasks, and 2 code tasks, with the average length of most tasks ranging from 5k to 15k, and a total of 4,750 test data. For detailed statistics and construction methods of LongBench tasks, please refer to the GitHub repository. In addition, we provide LongBench-E, a test set with a more uniform length distribution constructed by uniform sampling, with comparable amounts of data in the 0-4k, 4k-8k, and 8k+ length intervals to provide an analysis of the model's performance variations at different input lengths.

GitHub repo for LongBench. arXiv paper for LongBench.

Dataset Structure

Data Instances

An example of a data instance looks as follows:

{
    "input": "The input/command for the task, usually short, such as questions in QA, queries in Few-shot tasks, etc",
    "context": "The long context required for the task, such as documents, cross-file code, few-shot examples in Few-shot tasks",
    "answers": "A List of all true answers",
    "length": "Total length of the first three items (counted in characters for Chinese and words for English)",
    "dataset": "The name of the dataset to which this piece of data belongs",
    "language": "The language of this piece of data",
    "all_classes": "All categories in classification tasks, null for non-classification tasks",
    "_id": "Random id for each piece of data"
}

Data Fields

The dataset has the following fields:

• input (str): The input/command for the task, usually short, such as questions in QA or queries in few-shot tasks.
• context (str): The long context required for the task, such as documents, cross-file code, or few-shot examples.
• answers (list): A list of all true answers.
• length (int): Total length of the input, context, and answers (counted in characters for Chinese and words for English).
• dataset (str): The name of the dataset to which this piece of data belongs.
• language (str): The language of this piece of data.
• all_classes (list): All categories in classification tasks; null for non-classification tasks.
• _id (str): Random id for each piece of data.

Task statistics

Task	Task Type	Eval metric	Avg len	Language	#Sample
HotpotQA	Multi-doc QA	F1	9,151	EN	200
2WikiMultihopQA	Multi-doc QA	F1	4,887	EN	200
MuSiQue	Multi-doc QA	F1	11,214	EN	200
DuReader	Multi-doc QA	Rouge-L	15,768	ZH	200
MultiFieldQA-en	Single-doc QA	F1	4,559	EN	150
MultiFieldQA-zh	Single-doc QA	F1	6,701	ZH	200
NarrativeQA	Single-doc QA	F1	18,409	EN	200
Qasper	Single-doc QA	F1	3,619	EN	200
GovReport	Summarization	Rouge-L	8,734	EN	200
QMSum	Summarization	Rouge-L	10,614	EN	200
MultiNews	Summarization	Rouge-L	2,113	EN	200
VCSUM	Summarization	Rouge-L	15,380	ZH	200
TriviaQA	Few shot	F1	8,209	EN	200
SAMSum	Few shot	Rouge-L	6,258	EN	200
TREC	Few shot	Accuracy	5,177	EN	200
LSHT	Few shot	Accuracy	22,337	ZH	200
PassageRetrieval-en	Synthetic	Accuracy	9,289	EN	200
PassageCount	Synthetic	Accuracy	11,141	EN	200
PassageRetrieval-zh	Synthetic	Accuracy	6,745	ZH	200
LCC	Code	Edit Sim	1,235	Python/C#/Java	500
RepoBench-P	Code	Edit Sim	4,206	Python/Java	500

Note: In order to avoid discrepancies caused by different tokenizers, we use the word count (using Python's split function) to calculate the average length of English datasets and code datasets, and use the character count to calculate the average length of Chinese datasets.

Task description

Task	Task Description
HotpotQA	Answer related questions based on multiple given documents
2WikiMultihopQA	Answer related questions based on multiple given documents
MuSiQue	Answer related questions based on multiple given documents
DuReader	Answer related Chinese questions based on multiple retrieved documents
MultiFieldQA-en	Answer English questions based on a long article, which comes from a relatively diverse field
MultiFieldQA-zh	Answer Chinese questions based on a long article, which comes from a relatively diverse field
NarrativeQA	Answer questions based on stories or scripts, including understanding of important elements such as characters, plots, themes, etc.
Qasper	Answer questions based on an NLP research paper; questions proposed and answered by NLP practitioners
GovReport	A summarization task that requires summarizing government work reports
MultiNews	A multi-doc summarization task that requires summarizing over multiple news articles
QMSum	A summarization task that requires summarizing meeting records based on user queries
VCSUM	A summarization task that requires summarizing Chinese meeting records
SAMSum	A dialogue summarization task, providing several few-shot examples
TriviaQA	Single document question answering task, providing several few-shot examples
TREC	A classification task that requires categorizing questions, includes 50 categories in total
LSHT	A Chinese classification task that requires categorizing news, includes 24 categories in total
PassageRetrieval-en	Given 30 English Wikipedia paragraphs, determine which paragraph the given summary corresponds to
PassageCount	Determine the total number of different paragraphs in a given repetitive article
PassageRetrieval-zh	Given several Chinese paragraphs from the C4 data set, determine which paragraph the given abstract corresponds to
LCC	Given a long piece of code, predict the next line of code
RepoBench-P	Given code in multiple files within a GitHub repository (including cross-file dependencies), predict the next line of code

LongBench-E statistics

Task	Task Type	#data in 0-4k	#data in 4-8k	#data in 8k+
HotpotQA	Multi-doc QA	100	100	100
2WikiMultihopQA	Multi-doc QA	100	100	100
MultiFieldQA-en	Single-doc QA	67	70	13
Qasper	Single-doc QA	100	100	24
GovReport	Summarization	100	100	100
MultiNews	Summarization	100	100	94
TriviaQA	Few shot	100	100	100
SAMSum	Few shot	100	100	100
TREC	Few shot	100	100	100
PassageRetrieval-en	Synthetic	100	100	100
PassageCount	Synthetic	100	100	100
LCC	Code	100	100	100
RepoBench-P	Code	100	100	100

Reference

We would like to acknowledge Yushi Bai and Xin Lv et al. for creating and maintaining the LongBench dataset as a valuable resource for the natural language processing and machine learning research community. For more information about the LongBench dataset and its creators, please visit the LongBench repository.

License

The dataset has been released under the MIT license.

Citation

@misc{bai2023longbench,
      title={LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding}, 
      author={Yushi Bai and Xin Lv and Jiajie Zhang and Hongchang Lyu and Jiankai Tang and Zhidian Huang and Zhengxiao Du and Xiao Liu and Aohan Zeng and Lei Hou and Yuxiao Dong and Jie Tang and Juanzi Li},
      year={2023},
      eprint={2308.14508},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}