(The final report on the workshop is available here.)
In recent years, deep neural networks have yielded significant performance improvements on speech recognition and computer vision tasks, as well as led to exciting breakthroughs in novel application areas such as automatic voice translation, image captioning, and conversational agents. Despite demonstrating good performance on natural language processing (NLP) tasks, the performance of deep neural networks on IR tasks has had relatively less scrutiny.
The lack of many positive results in the area of information retrieval is partially due to the fact that IR tasks such as ranking are fundamentally different from NLP tasks, but also because the IR and neural network communities are only beginning to focus on the application of these techniques to core information retrieval problems. Given that deep learning has made such a big impact, first on speech processing and computer vision and now, increasingly, also on computational linguistics, it seems clear that deep learning will have a major impact on information retrieval and that this is an ideal time for a workshop in this area. Our focus is on the applicability of deep neural networks to information retrieval: demonstrating performance improvements on public or private information retrieval datasets, identifying key modelling challenges and best practices, and thinking about what insights deep neural network architectures give us about information retrieval problems.
Neu-IR 2016 will be a highly interactive full day workshop that will provide a forum for academic and industrial researchers working at the intersection of IR and neural networks. The purpose is to provide an opportunity for people to present new work and early results, compare notes on neural network toolkits, share best practices, and discuss the main challenges facing this line of research.
Please use the tabs above to navigate to see the program, the accepted papers and other details of this workshop.
Neu-IR will be a highly interactive full day workshop, featuring a mix of presentation and interaction formats. The full schedule is presented below.
Morning Session I
09:00 – 10:30
Welcome and opening announcements [slides]
Keynote: Recurrent Networks and Beyond [slides]
Tomas Mikolov, Facebook AI Research
Paper: Query Expansion with Locally-Trained Word Embeddings [slides]
Fernando Diaz, Bhaskar Mitra and Nick Craswell
Paper: Uncertainty in Neural Network Word Embedding Exploration of Potential Threshold [slides]
Navid Rekabsaz, Mihai Lupu and Allan Hanbury
10:30 – 11:00
Morning Session II
11:00 – 12:30
Lessons from the Trenches [slides]
12:30 – 14:00
Afternoon Session I
14:00 – 15:30
Keynote: Does IR Need Deep Learning? [slides]
Hang Li, Huawei Technologies
Paper: Modelling User Preferences using Word Embeddings for Context-Aware Venue Recommendation [slides]
Jarana Manotumruksa, Craig Macdonald and Iadh Ounis
Paper: A Study of MatchPyramid Models on Ad-hoc Retrieval [slides]
Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu and Xueqi Cheng
Paper: Emulating Human Conversations using Convolutional Neural Network-based IR [slides]
Abhay Prakash, Chris Brockett and Puneet Agrawal
15:30 – 16:00
Afternoon Session II
16:00 – 17:45
Breakout session retrospective
Recurrent Networks and Beyond
Tomas Mikolov, Facebook AI Research
Abstract: In this talk, I will give a brief overview of recurrent networks and their applications. I will then present several extensions that aim to help these powerful models to learn more patterns from training data. This will include a simple modification of the architecture that allows to capture longer context information, and an architecture that allows to learn complex algorithmic patterns. The talk will be concluded with a discussion of a long term research plan on how to advance machine learning techniques towards development of artificial intelligence.
Bio: Tomas Mikolov is a research scientist at Facebook AI Research since May 2014. Previously he has been a member of the Google Brain team, where he developed and implemented efficient algorithms for computing distributed representations of words (word2vec project). He obtained his PhD from Brno University of Technology (Czech Republic) for his work on recurrent neural network based language models (RNNLM). His long term research goal is to develop intelligent machines capable of learning and communication with people using natural language.
Does IR Need Deep Learning?
Hang Li, Huawei Technologies
Abstract: In recent years, deep learning has become the key technology of state-of-the-art systems in many areas of computer science, such as computer vision, speech processing, and natural language processing. A question naturally arises, that is, can deep learning also bring breakthrough into IR (information retrieval)? In fact, there has been a large amount of effort made to address the question and significant progress has been achieved. Yet there is still doubt about whether it is the case.
In this talk, I will argue that, if we take a broad view on IR, then we arrive at a conclusion that deep learning can indeed greatly boost IR. Actually it has been observed that deep learning can make great improvements on some hard problems in IR such as question answering from knowledge base, image retrieval, etc; on the other hand, for some traditional IR tasks, in some sense easy tasks, such as document retrieval, the improvements might not be so notable. I will introduce some of the work on deep learning for IR conducted at Huawei Noah’s Ark Lab, to support my claim. I will also make discussions on the strength and limitation of deep learning, IR problems on which deep learning can potentially make significant contributions, as well as future directions of research on IR.
Bio: Hang Li is director of the Noah’s Ark Lab of Huawei Technologies, adjunct professors of Peking University and Nanjing University. He is ACM Distinguished Scientist. His research areas include information retrieval, natural language processing, statistical machine learning, and data mining. Hang graduated from Kyoto University in 1988 and earned his PhD from the University of Tokyo in 1998. He worked at the NEC lab as researcher during 1991 and 2001, and Microsoft Research Asia as senior researcher and research manager during 2001 and 2012. He joined Huawei Technologies in 2012. Hang has published three technical books, and more than 120 technical papers at top international conferences including SIGIR, WWW, WSDM, ACL, EMNLP, ICML, NIPS, SIGKDD, AAAI, IJCAI, and top international journals including CL, NLE, JMLR, TOIS, IRJ, IPM, TKDE, TWEB, TIST. He and his colleagues’ papers received the SIGKDD’08 best application paper award, the SIGIR’08 best student paper award, the ACL’12 best student paper award. Hang worked on the development of several products such as Microsoft SQL Server 2005, Office 2007, Live Search 2008, Bing 2009, Office 2010, Bing 2010, Office 2012, Huawei Smartphones 2014. He has 42 granted US patents. Hang is also very active in the research communities and has served or is serving top international conferences as PC chair, Senior PC member, or PC member, including SIGIR, WWW, WSDM, ACL, NACL, EMNLP, NIPS, SIGKDD, ICDM, IJCAI, ACML, and top international journals as associate editor or editorial board member, including CL, IRJ, TIST, JASIST, JCST.
We had 27 submissions (excluding three incomplete submissions). Every paper was reviewed by at least two members of the program committee and finally 19 submission were accepted (acceptance rate of 73%). Among the accepted papers, there were a few popular themes. 8 papers were related to learning and applications of word embeddings. 10 papers focused on applications of deep neural networks for different IR tasks. The accepted papers also covered a broad range of tasks, including question/answering, proactive IR, knowledge-based IR, conversational models and text-to-image, but document ranking was a popular choice with 7 papers using it as the evaluation task. The word cloud summary (generated using http://www.wordle.net) of the abstracts of the accepted papers highlights additional themes across all the submissions.
Geographically, the accepted papers (based on the first author) ranged from 9 countries and 3 continents (FR: 4, IN: 4, CN: 2, DK: 2, UK: 2, US: 2, AT: 1, FI: 1 and IT: 1). Based on the first author’s affiliation, 2 of the accepted papers came from the industry and the rest from academia.
The full list of accepted papers is below:
Lessons from the Trenches
The Lessons from the Trenches will be a series of “lightning talks” by researchers who are actively working in the intersection of information retrieval and neural networks who want to share their personal insights and learning with the broader community. In particular, we are hoping to hear about,
- Key challenges faced in making neural models work effectively for IR tasks
- Best practices and related insights
- Negative results
The following people have signed-up to present at this session.
- Sergey Nikolenko
- Qingyao Ai
- Debasis Ganguly
- Alessandro Moschitti
- Jun Xu
- Grady Simon
- Alexey Borisov
- Bhaska Mitra
Call for Papers
We solicit submission of papers of two to six pages (excluding references), representing reports of original research, preliminary research results, proposals for new work, descriptions of neural network based toolkits tailored for IR, and position papers. Papers presented at the workshop will be required to be uploaded to arXiv.org but will be considered non-archival, and may be submitted elsewhere (modified or not), although the workshop site will maintain a link to the arXiv versions. This makes the workshop a forum for the presentation and discussion of current work, without preventing the work from being published elsewhere.
We are interested in submissions relevant to the following main themes:
- The application of neural network models in IR tasks, including but not limited to:
- Full text document retrieval, passage retrieval, question answering
- Web search, searching social media, distributed information retrieval, entity ranking
- Learning to rank combined with neural network based representation learning
- User and task modelling, personalized search, diversity
- Query formulation assistance, query recommendation, conversational search
- Multimedia retrieval
- Fundamental modelling challenges faced in such applications, including but not limited to:
- Learning dense representations for long documents
- Dealing with rare queries and rare words
- Modelling text at different granularities (character, word, passage, document)
- Compositionality of vector representations
- Jointly modelling queries, documents, entities and other structured/knowledge data
- Best practices for research and development in the area, dealing with concerns such as:
- Finding sufficient publicly-available training data
- Baselines, test data, avoiding overfitting
- Neural network toolkits
- Real-world use cases, deployment at scale
All papers will be peer reviewed (single-blind) by the program committee and judged by their relevance to the workshop, especially to the main themes identified above, and their potential to generate discussion. All submissions must be formatted according to the ACM SIG proceedings template. Please note that at least one of the authors of each accepted paper must register for the workshop and present the paper in-person.
Submission url: https://easychair.org/conferences/?conf=neuir2016
Nick Craswell, Microsoft, Bellevue, US
W. Bruce Croft, University of Massachusetts, Amherst, US
Jiafeng Guo, Chinese Academy of Sciences, Beijing, China
Bhaskar Mitra, Microsoft, Cambridge, UK
Maarten de Rijke, University of Amsterdam, Amsterdam, The Netherlands
Carsten Eickhoff, ETH Zurich
Debasis Ganguly, Dublin City University
Katja Hoffman, Microsoft Research
Hang Li, Huawei Technologies
Piotr Mirowski, Google DeepMind
Alessandro Moschitti, Qatar Computing Research Institute, HKBU
Pavel Serdyukov, Yandex
Fabrizio Silvestri, Yahoo Labs
Alessandro Sordoni, University of Montreal