TextVQA

Summary

Introduction
Dataset Structure
Reference
License
Citation

Introduction

TextVQA requires models to read and reason about text in images to answer questions about them. Specifically, models need to incorporate a new modality of text present in the images and reason over it to answer TextVQA questions. TextVQA dataset contains 45,336 questions over 28,408 images from the OpenImages dataset. The dataset uses VQA accuracy metric for evaluation.

Dataset Structure

Data Instances

A typical sample mainly contains the question in question field, an image object in image field, OpenImage image id in image_id and lot of other useful metadata. 10 answers per questions are contained in the answers attribute. For test set, 10 empty strings are contained in the answers field as the answers are not available for it.

An example look like below:

{
  'question': 'who is this copyrighted by?',
   'image_id': '00685bc495504d61',
   'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=384x512 at 0x276021C5EB8>,
   'image_classes': ['Vehicle', 'Tower', 'Airplane', 'Aircraft'],
   'flickr_original_url': 'https://farm2.staticflickr.com/5067/5620759429_4ea686e643_o.jpg',
   'flickr_300k_url': 'https://c5.staticflickr.com/6/5067/5620759429_f43a649fb5_z.jpg',
   'image_width': 786,
   'image_height': 1024,
   'answers': ['simon clancy',
    'simon ciancy',
    'simon clancy',
    'simon clancy',
    'the brand is bayard',
    'simon clancy',
    'simon clancy',
    'simon clancy',
    'simon clancy',
    'simon clancy'],
   'question_tokens': ['who', 'is', 'this', 'copyrighted', 'by'],
   'question_id': 3,
   'set_name': 'train'
},

Data Fields

question: string, the question that is being asked about the image
image_id: string, id of the image which is same as the OpenImages id
image: A PIL.Image.Image object containing the image about which the question is being asked. Note that when accessing the image column: dataset[0]["image"] the image file is automatically decoded. Decoding of a large number of image files might take a significant amount of time. Thus it is important to first query the sample index before the "image" column, i.e. dataset[0]["image"] should always be preferred over dataset["image"][0].
image_classes: List[str], The OpenImages classes to which the image belongs to.
flickr_original_url: string, URL to original image on Flickr
flickr_300k_url: string, URL to resized and low-resolution image on Flickr.
image_width: int, Width of the original image.
image_height: int, Height of the original image.
question_tokens: List[str], A pre-tokenized list of question.
answers: List[str], List of 10 human-annotated answers for the question. These 10 answers are collected from 10 different users. The list will contain empty strings for test set for which we don't have the answers.
question_id: int, Unique id of the question.
set_name: string, the set to which this question belongs.

Data Splits

There are three splits. train, validation and test. The train and validation sets share images with OpenImages train set and have their answers available. For test set answers, we return a list of ten empty strings. To get inference results and numbers on test set, you need to go to the EvalAI leaderboard and upload your predictions there. Please see instructions at https://textvqa.org/challenge/.

Reference

We would like to acknowledge Singh, Amanpreet et al. for creating and maintaining the TextVQA dataset as a valuable resource for the computer vision and machine learning research community. For more information about the TextVQA dataset and its creator, please visit the TextVQA website.

License

The dataset has been released under the CC by 4.0 License.

Citation

@inproceedings{singh2019towards,
    title={Towards VQA Models That Can Read},
    author={Singh, Amanpreet and Natarjan, Vivek and Shah, Meet and Jiang, Yu and Chen, Xinlei and Batra, Dhruv and Parikh, Devi and Rohrbach, Marcus},
    booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
    pages={8317-8326},
    year={2019}
}