This report introduces a dataset named Microsoft Research Asia Multimedia (MSRA-MM) that aims to encourage research in multimedia information retrieval and the related areas. The images and videos in the dataset are collected from Microsoft Live Search and the performance of state-of-the-art industrial techniques can be evaluated accordingly. Therefore, conducting research using the dataset is able to demonstrate the practical usefulness of the studied algorithms. The data have been comprehensively annotated, such as their relevance, concept, category and quality, and different research scenarios can be implemented on the dataset.