Neu-IR: The SIGIR 2016 Workshop on Neural Information Retrieval

Neu-IR: The SIGIR 2016 Workshop on Neural Information Retrieval

Summary

(The final report on the workshop is available here.)

In recent years, deep neural networks have yielded significant performance improvements on speech recognition and computer vision tasks, as well as led to exciting breakthroughs in novel application areas such as automatic voice translation, image captioning, and conversational agents. Despite demonstrating good performance on natural language processing (NLP) tasks, the performance of deep neural networks on IR tasks has had relatively less scrutiny.

The lack of many positive results in the area of information retrieval is partially due to the fact that IR tasks such as ranking are fundamentally different from NLP tasks, but also because the IR and neural network communities are only beginning to focus on the application of these techniques to core information retrieval problems. Given that deep learning has made such a big impact, first on speech processing and computer vision and now, increasingly, also on computational linguistics, it seems clear that deep learning will have a major impact on information retrieval and that this is an ideal time for a workshop in this area. Our focus is on the applicability of deep neural networks to information retrieval: demonstrating performance improvements on public or private information retrieval datasets, identifying key modelling challenges and best practices, and thinking about what insights deep neural network architectures give us about information retrieval problems.

Neu-IR 2016 will be a highly interactive full day workshop that will provide a forum for academic and industrial researchers working at the intersection of IR and neural networks. The purpose is to provide an opportunity for people to present new work and early results, compare notes on neural network toolkits, share best practices, and discuss the main challenges facing this line of research.

Please use the tabs above to navigate to see the program, the accepted papers and other details of this workshop.

Program

Neu-IR will be a highly interactive full day workshop, featuring a mix of presentation and interaction formats. The full schedule is presented below.

Morning Session I
09:00 – 10:30

Welcome and opening announcements [slides]
Bhaskar Mitra
15 mins

Keynote: Recurrent Networks and Beyond [slides]
Tomas Mikolov, Facebook AI Research
45 mins

Paper: Query Expansion with Locally-Trained Word Embeddings [slides]
Fernando Diaz, Bhaskar Mitra and Nick Craswell
15 mins

Paper: Uncertainty in Neural Network Word Embedding Exploration of Potential Threshold [slides]
Navid Rekabsaz, Mihai Lupu and Allan Hanbury
15 mins

Coffee Break
10:30 – 11:00

Morning Session II
11:00 – 12:30

Lessons from the Trenches [slides]
45 mins

Poster presentations
45 mins

Lunch Break
12:30 – 14:00

Afternoon Session I
14:00 – 15:30

Keynote: Does IR Need Deep Learning? [slides]
Hang Li, Huawei Technologies
45 mins

Paper: Modelling User Preferences using Word Embeddings for Context-Aware Venue Recommendation [slides]
Jarana Manotumruksa, Craig Macdonald and Iadh Ounis
15 mins

Paper: A Study of MatchPyramid Models on Ad-hoc Retrieval [slides]
Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu and Xueqi Cheng
15 mins

Paper: Emulating Human Conversations using Convolutional Neural Network-based IR [slides]
Abhay Prakash, Chris Brockett and Puneet Agrawal
15 mins

Coffee Break
15:30 – 16:00

Afternoon Session II
16:00 – 17:45

Breakout session
45 mins

Breakout session retrospective
45 mins

Concluding remarks
15 mins

Keynotes

Recurrent Networks and Beyond

Tomas Mikolov, Facebook AI Research

mikolovAbstract: In this talk, I will give a brief overview of recurrent networks and their applications. I will then present several extensions that aim to help these powerful models to learn more patterns from training data. This will include a simple modification of the architecture that allows to capture longer context information, and an architecture that allows to learn complex algorithmic patterns. The talk will be concluded with a discussion of a long term research plan on how to advance machine learning techniques towards development of artificial intelligence.

Bio: Tomas Mikolov is a research scientist at Facebook AI Research since May 2014. Previously he has been a member of the Google Brain team, where he developed and implemented efficient algorithms for computing distributed representations of words (word2vec project). He obtained his PhD from Brno University of Technology (Czech Republic) for his work on recurrent neural network based language models (RNNLM). His long term research goal is to develop intelligent machines capable of learning and communication with people using natural language.

 

Does IR Need Deep Learning?

Hang Li, Huawei Technologies

HangLiAbstract: In recent years, deep learning has become the key technology of state-of-the-art systems in many areas of computer science, such as computer vision, speech processing, and natural language processing. A question naturally arises, that is, can deep learning also bring breakthrough into IR (information retrieval)? In fact, there has been a large amount of effort made to address the question and significant progress has been achieved. Yet there is still doubt about whether it is the case.

In this talk, I will argue that, if we take a broad view on IR, then we arrive at a conclusion that deep learning can indeed greatly boost IR. Actually it has been observed that deep learning can make great improvements on some hard problems in IR such as question answering from knowledge base, image retrieval, etc; on the other hand, for some traditional IR tasks, in some sense easy tasks, such as document retrieval, the improvements might not be so notable. I will introduce some of the work on deep learning for IR conducted at Huawei Noah’s Ark Lab, to support my claim. I will also make discussions on the strength and limitation of deep learning, IR problems on which deep learning can potentially make significant contributions, as well as future directions of research on IR.

Bio: Hang Li is director of the Noah’s Ark Lab of Huawei Technologies, adjunct professors of Peking University and Nanjing University. He is ACM Distinguished Scientist. His research areas include information retrieval, natural language processing, statistical machine learning, and data mining. Hang graduated from Kyoto University in 1988 and earned his PhD from the University of Tokyo in 1998. He worked at the NEC lab as researcher during 1991 and 2001, and Microsoft Research Asia as senior researcher and research manager during 2001 and 2012. He joined Huawei Technologies in 2012. Hang has published three technical books, and more than 120 technical papers at top international conferences including SIGIR, WWW, WSDM, ACL, EMNLP, ICML, NIPS, SIGKDD, AAAI, IJCAI, and top international journals including CL, NLE, JMLR, TOIS, IRJ, IPM, TKDE, TWEB, TIST. He and his colleagues’ papers received the SIGKDD’08 best application paper award, the SIGIR’08 best student paper award, the ACL’12 best student paper award. Hang worked on the development of several products such as Microsoft SQL Server 2005, Office 2007, Live Search 2008, Bing 2009, Office 2010, Bing 2010, Office 2012, Huawei Smartphones 2014. He has 42 granted US patents. Hang is also very active in the research communities and has served or is serving top international conferences as PC chair, Senior PC member, or PC member, including SIGIR, WWW, WSDM, ACL, NACL, EMNLP, NIPS, SIGKDD, ICDM, IJCAI, ACML, and top international journals as associate editor or editorial board member, including CL, IRJ, TIST, JASIST, JCST.

Accepted Papers

We had 27 submissions (excluding three incomplete submissions). Every paper was reviewed by at least two members of the program committee and finally 19 submission were accepted (acceptance rate of 73%). Among the accepted papers, there were a few popular themes. 8 papers were related to learning and applications of word embeddings. 10 papers focused on applications of deep neural networks for different IR tasks. The accepted papers also covered a broad range of tasks, including question/answering, proactive IR, knowledge-based IR, conversational models and text-to-image, but document ranking was a popular choice with 7 papers using it as the evaluation task. The word cloud summary (generated using http://www.wordle.net) of the abstracts of the accepted papers highlights additional themes across all the submissions.

wordcloud-abstracts

Geographically, the accepted papers (based on the first author) ranged from 9 countries and 3 continents (FR: 4, IN: 4, CN: 2, DK: 2, UK: 2, US: 2, AT: 1, FI: 1 and IT: 1). Based on the first author’s affiliation, 2 of the accepted papers came from the industry and the rest from academia.

 

The full list of accepted papers is below:

An empirical study on large scale text classification with skip-gram embeddings Pdf_icon
Georgios Balikas and Massih-Reza Amini

Deep Feature Fusion Network for Answer Quality Prediction in Community Question Answering Pdf_icon
Sai Praneeth Suggu, Kushwanth N. Goutham T, Manoj K. Chinnakotla and Manish Shrivastava

Selective Term Proximity Scoring Via BP-ANN Pdf_icon
Ju Yang, Rebecca Stones, Gang Wang and Xiaoguang Liu

Adaptability of Neural Networks on Varying Granularity IR Tasks Pdf_icon
Daniel Cohen, Qingyao Ai and W. Bruce Croft

Emulating Human Conversations using Convolutional Neural Network-based IR Pdf_icon
Abhay Prakash, Chris Brockett and Puneet Agrawal

A Study of MatchPyramid Models on Ad-hoc Retrieval Pdf_icon
Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu and Xueqi Cheng

Learning text representation using recurrent convolutional neural network with highway layers Pdf_icon
Ying Wen, Weinan Zhang, Rui Luo and Jun Wang

Toward Word Embedding for Personalized Information Retrieval Pdf_icon
Nawal Ould Amer, Philippe Mulhem and Mathias Géry

Toward a Deep Neural Approach for Knowledge-Based IR Pdf_icon
Gia-Hung Nguyen, Lynda Tamine, Laure Soulier and Nathalie Bricon-Souf

Query Expansion with Locally-Trained Word Embeddings Pdf_icon
Fernando Diaz, Bhaskar Mitra and Nick Craswell

LSTM-Based Predictions for Proactive Information Retrieval Pdf_icon
Petri Luukkonen, Markus Koskela and Patrik Floréen

Picture It In Your Mind: Generating High Level Visual Representations From Textual Descriptions Pdf_icon
Fabio Carrara, Andrea Esuli, Tiziano Fagni, Fabrizio Falchi and Alejandro Moreo Fernández

Uncertainty in Neural Network Word Embedding Exploration of Potential Threshold Pdf_icon
Navid Rekabsaz, Mihai Lupu and Allan Hanbury

Deep Learning Relevance: Creating Relevant Information (as Opposed to Retrieving it) Pdf_icon
Christina Lioma, Birger Larsen, Casper Petersen and Jakob Grue Simonsen

Learning Dynamic Classes of Events using Stacked Multilayer Perceptron Networks Pdf_icon
Nattiya Kanhabua, Huamin Ren and Thomas B. Moeslund

Representing Documents and Queries as Sets of Word Embedded Vectors for Information Retrieval Pdf_icon
Debasis Ganguly, Dwaipayan Roy, Mandar Mitra and Gareth Jones

Using Word Embeddings for Automatic Query Expansion Pdf_icon
Dwaipayan Roy, Debjyoti Paul and Mandar Mitra

Modelling User Preferences using Word Embeddings for Context-Aware Venue Recommendation Pdf_icon
Jarana Manotumruksa, Craig Macdonald and Iadh Ounis

Using Word Embeddings in Twitter Election Classification Pdf_icon
Xiao Yang, Craig Macdonald and Iadh Ounis

 

Lessons from the Trenches

The Lessons from the Trenches will be a series of “lightning talks” by researchers who are actively working in the intersection of information retrieval and neural networks who want to share their personal insights and learning with the broader community. In particular, we are hoping to hear about,

  • Key challenges faced in making neural models work effectively for IR tasks
  • Best practices and related insights
  • Negative results

 

The following people have signed-up to present at this session.

  • Sergey Nikolenko
  • Qingyao Ai
  • Debasis Ganguly
  • Alessandro Moschitti
  • Jun Xu
  • Grady Simon
  • Alexey Borisov
  • Bhaska Mitra

Call for Papers

We solicit submission of papers of two to six pages (excluding references), representing reports of original research, preliminary research results, proposals for new work, descriptions of neural network based toolkits tailored for IR, and position papers. Papers presented at the workshop will be required to be uploaded to arXiv.org but will be considered non-archival, and may be submitted elsewhere (modified or not), although the workshop site will maintain a link to the arXiv versions. This makes the workshop a forum for the presentation and discussion of current work, without preventing the work from being published elsewhere.

We are interested in submissions relevant to the following main themes:

  1. The application of neural network models in IR tasks, including but not limited to:
    • Full text document retrieval, passage retrieval, question answering
    • Web search, searching social media, distributed information retrieval, entity ranking
    • Learning to rank combined with neural network based representation learning
    • User and task modelling, personalized search, diversity
    • Query formulation assistance, query recommendation, conversational search
    • Multimedia retrieval
  2. Fundamental modelling challenges faced in such applications, including but not limited to:
    • Learning dense representations for long documents
    • Dealing with rare queries and rare words
    • Modelling text at different granularities (character, word, passage, document)
    • Compositionality of vector representations
    • Jointly modelling queries, documents, entities and other structured/knowledge data
  3. Best practices for research and development in the area, dealing with concerns such as:
    • Finding sufficient publicly-available training data
    • Baselines, test data, avoiding overfitting
    • Neural network toolkits
    • Real-world use cases, deployment at scale

All papers will be peer reviewed (single-blind) by the program committee and judged by their relevance to the workshop, especially to the main themes identified above, and their potential to generate discussion. All submissions must be formatted according to the ACM SIG proceedings template. Please note that at least one of the authors of each accepted paper must register for the workshop and present the paper in-person.

Submission url: https://easychair.org/conferences/?conf=neuir2016

Organization

Organizers

Nick Craswell, Microsoft, Bellevue, US
W. Bruce Croft, University of Massachusetts, Amherst, US
Jiafeng Guo, Chinese Academy of Sciences, Beijing, China
Bhaskar Mitra, Microsoft, Cambridge, UK
Maarten de Rijke, University of Amsterdam, Amsterdam, The Netherlands

 

Program Committee

Carsten Eickhoff, ETH Zurich
Debasis Ganguly, Dublin City University
Katja Hoffman, Microsoft Research
Hang Li, Huawei Technologies
Piotr Mirowski, Google DeepMind
Alessandro Moschitti, Qatar Computing Research Institute, HKBU
Pavel Serdyukov, Yandex
Fabrizio Silvestri, Yahoo Labs
Alessandro Sordoni, University of Montreal