Microsoft Research India Summer Workshop on Artificial Social Intelligence


Breakthroughs in Artificial Intelligence (AI) have typically shown that AI systems are good at solving specific tasks that have a well-defined goal, such as Speech Recognition, Image Captioning, Games like Poker/Go/Jeopardy, among others. However, as AI systems become ubiquitous, it is not enough for them to solve specific tasks; rather they will have to continuously interact with human-users as well as other AI systems in a rapidly evolving environment. These systems will have to continuously review and evolve their interaction strategies during the ongoing interaction. Goals may not be defined in advance, and might evolve dynamically. The systems have to ensure that apart from solving the primary task, the user receives a pleasant and professional experience that is ”socio-culturally appropriate”. In other words, we are quickly moving towards a world where AI systems have to go far beyond functional intelligence  – they have to be socio-culturally adept and behaviourally intelligent. We refer to this phenomenon as “Artificial Social Intelligence” and the consequent systems as Socially Intelligent Agents.

As one can imagine, ASI is an extremely multi-disciplinary endeavor, where one needs inputs not only from AI and Machine-Learning researchers, but also linguists, social scientists, HCI, design and vision researchers.

The above figure shows a hypothetical interaction between a chatbot “botty” and a young boy “chhota bheem” in Hinglish to demonstrate the importance of ASI. While botty is able to interpret sentences and generate responses perfectly, it misses the fact that “a princess hat” is not a culturally appropriate birthday gift for chhota bheem’s mother. Thus, recommender systems (if you imagine botty had a gift recommender system embedded within it) need to take into account larger as well as user-specific socio-cultural contexts into account while making recommendations. Further, one might observe that botty has used “uske” (non-honorific pronoun) for Chhota Bheem’s mother and “unko” (the honorific pronoun) for referring to his younger sister, though the conversation etiquettes and pragmatics of Hindi demands the pronouns to be used the other way round.

ASI is an emergent field. While there has been research on some specific aspects of ASI, the parts are yet to come together and coalesce into a field or an interactive AI agent. We believe there are four fundamental sets of problems within the broad scope of ASI, which though can be dealt with independently, at the end should feed into each other:

  1. Discovery of Principles of Socio-cultural Interactions: Linguists, psychologists and social-scientists have been studying human behavior to understand the norms and aberrations, their biological, social and cultural origins and needs. In order to formulate the principles of socio-culturally enriching interactions between human and AI systems, it is not only necessary to gain insights from these fields, but also to conduct large scale data-driven studies that aim at validating the principles and deciphering new behavioral traits. Such studies are now possible, thanks to the large scale availability of socially grounded user data from social media, and due to advances in machine learning and other data-analysis techniques (see [1,2] for examples).  Targeted Human-human and Human-machine interaction studies would also be of great importance.
  2. Design and Development of ASI Systems: The learnt principles could then be used to design interaction policies for ASI systems such as chatbots [9,10], recommender systems [5], search engines, self-driving cars, multimodal agents, or some completely new form of interactive agents. Developing these agents would require one to solve yet another set of engineering and research problems. One example of such a system is the virtual receptionist developed by Dr. Dan Bohus from Microsoft Research Redmond, which keeps track of users attention and engagement through visual cues (such as gaze tracking, head orientation etc.) to initiate the interaction at the most appropriate moment. Further, it can also make use of hesitation (e.g., “hmmm… uhhh”) to attract the attention of the user, buy time for processing or even to indicate uncertainty in the response [3].
  3. Evaluation of ASI: It is easy to evaluate systems which has a well-defined end-goal. For instance, image recognition systems can be evaluated on standard metrics like precision and recall on a certain class of images. However, it is extremely difficult to evaluate socio-cultural intelligence of a system because these traits are neither directly measurable, nor leads to any measurable outcome. We believe this is one of the most challenging open problem of ASI.
  4. Techniques and Resources for enabling ASI: Generic techniques such as learning of unbiased models from potentially biased data [4], platforms for prototyping dialogue systems [6-8] and chatbots with ASI, models of pragmatics, politeness, multilingual interactions, etc. are useful and important for enabling further research and system development in ASI. Large datasets of human-human and human-machine interactions are crucial for building such models and systems.

Proposals spanning any of the above sub-areas of ASI are welcome.


[1] Mark my words! Linguistic style accommodation in social media.

[2] Understanding Language Preference for Expression of Opinion and Sentiment: What do Hindi-English Speakers do on Twitter?

[3] Managing human-robot engagement with forecasts and … um … Hesitations.

[4] Fairness, Accountability, and Transparency in Machine Learning Workshop Series


[6] Strategy and policy learning for non-task oriented conversational bots

[7] On data driven parametric backchannel synthesis for expressing attentiveness in conversational agents

[8] Deciphering the Silent Participant: On the Use of Audio-Visual Cues for the Classification of Listener Categories in Group Discussions

[9] Conversational involvement and synchronous nonverbal behaviour

[10] Towards the automatic detection of involvement in conversation

[11] Modeling ethnicity in/with technology: Using virtual agents to understand sociolinguistic variation

[12] Negotiated Collusion: Modeling Social Language and its Relationship Effects in Intelligent Agents

[13] ‘How about this weather?’ Social Dialog with Embodied Conversational Agents

Application Guidelines

Update:  We have shortlisted the final proposals along with the proposers (faculty) and the students for the Summer workshop. We will not accept any more proposals/student nominations.

Call for Proposals

The MSR Summer Workshop on Artificial Social Intelligence (ASI) will be an intense project-based research endeavor to enable intelligent systems to be more socially and culturally aware. Proposals are invited from faculty members in Indian universities and Indian start-ups in topics including but not limited to:

  1. Computational Social Science
  2. Socially and culturally aware agents
  3. Speech and Language Systems for ASI
  4. Multi-modality and ASI

Proposals could suggest building a Socially Intelligent Agent or generic platforms for architecting such agents; we also invite proposals that seek to conduct large-scale socio-cultural studies using computational methods that will guide architecting ASI systems. Projects done in the summer workshop should lead to either a working system, dataset or an in-depth, large-scale study leading to publishable work.

All faculty and students selected for the workshop are expected not to pursue parallel work during the workshop, since this is intended to be an intense 4 week effort leading to publishable work and significant progress in the field.

Selection Process

The submitted proposals along with the nominations for PhD and/or Postdocs will be selected through a thorough evaluation and revision process by the Program Committee. In parallel, the undergrad and masters students for the workshop will be selected by the Organizing Committee through a different process.

MSR India 2017 Summer Workshop Selection Process

The proposals will be evaluated by the members of the program committee on the basis of following criteria: innovativeness, generality, usability (of the proposed system/study), feasibility (of completing the project in 4 weeks), and preciseness of the goals and success metrics.

After initial round of evaluation, proposals will be shortlisted and the proposers will be invited to Microsoft Research India to give a presentation. Post this 4 to 6 projects will be selected for the summer workshop. Proposals will be refined and may be merged with other proposals before the final selection.

Submission Guidelines

Proposals are invited from faculty in Indian institutions in the following format:

  • Name, Affiliation and contact of the faculty submitting the proposal
  • Name of collaborators (if any): Up to ONE collaborator (could be a faculty member from outside India or a researcher in some non-academic lab or company, including start-ups) can be suggested.
  • Aim of the project
  • Context, challenges and usefulness
  • Proposed approach
  • Resources required: PhD/Masters/UG students, engineers, computational resources, data
  • Success metrics: When would you call this project a success (think of only what can be achieved during the summer workshop – 4 weeks)
  • Future plans: if you were to continue the work beyond the workshop, what would you do?

Proposals should be no longer than 2000 words (including diagrams, tables etc.), and can be sent in doc or pdf format.

Along with the proposal, please also send

  • A brief CV of the proposer highlighting the expertise in the proposed area.
  • Student Nominations: You can recommend names (along with affiliation, emails and CV) of up to
    • 1 Post-doc fellow/PhD student
    • 2 PG/UG students

The completed proposals should be emailed to, cc-ing Monojit Choudhury at no later than 11:59 pm IST 28th February 2017.

Please note that we cannot guarantee the selection of the students nominated in the proposal even if the proposal is selected, because we are seeking UG/masters student nominations through other means as well.

Important dates

Deadline for submitting proposals: Feb 28, 2017
Notification of shortlisted proposals: Mar 20, 2017
Presentation at MSRI: 1st week of April 2017
Notification of selection: April 30, 2017
Summer School: June 5-30, 2017 (tentative)

Who can participate and How?

Faculty/Researchers: Faculty members from universities in India are invited to submit proposals. They will be asked to present the proposal in person at Bangalore, if shortlisted. Projects can also have ONE collaborator from industry/other academic institutions from India or abroad. Selected faculty should be willing to work with other faculty/researchers if their proposals get merged. Faculty members can nominate ONE Post Docs/PhD student and up to TWO UG/master students in their proposal. Faculty members should also be willing to give lectures and tutorials in their areas of expertise during the workshop.

The Principal proposer of each selected project will receive an amount of Rs. 1.5 Lakhs as remuneration for the research efforts in the project. This amount is also expected to cover their travel to/from Bangalore, meals outside of those provided during the workshop and other incidental expenses. Faculty will have to make their own travel arrangements. Microsoft Research will provide accommodation with breakfast, and meals during working hours.

Important: The above remuneration will only be paid to the principal faculty proposer. Collaborators, if any, will receive accommodation and meals.

Post Doc/PhD students: Post Doc and PhD students CANNOT apply directly. They need to be nominated by the faculty member writing the proposal. Each proposal can have ONE Post Doc or PhD student nomination, from which the program committee will make the final selection.

Postdoc/PhD students will be provided with accommodation with breakfast, and meals during working hours. For outstation students, we will provide a travel allowance that should cover a large part of the estimated airfare (if booked well in advance), meals outside working hours and any other incidental expenses. They do not need to submit any bills or tickets to get the travel allowance. It will be a flat amount based on the city where their college is located. If they spend less than the allowance, they do not need to return the funds to us and if they spend more, we hope they can put in money from their own sources to cover the balance.

Masters/Undergraduate students: Masters and Undergraduate students will be selected by the program committee and assigned to projects based on their interests and skills. There are two routes through which these students can apply:

  1. A faculty submitting a proposal nominates up to TWO UG/masters students in their proposal.
  2. We will also seek UG/masters nomination from HoDs of a number of Indian institutes.

UG/Masters students will be provided with accommodation (on twin sharing basis) with breakfast, and meals during working hours. For outstation students, we will provide a travel allowance that should cover a large part of the estimated airfare (if booked well in advance), meals outside working hours and any other incidental expenses. They do not need to submit any bills or tickets to get the travel allowance. It will be a flat amount based on the city where their college is located. If they spend less than the allowance, they do not need to return the funds to us and if they spend more, we hope they can put in money from their own sources to cover the balance.

Computational Infrastructure

Garage style workstations will be provided to each team in Microsoft Research India premises. Team members are expected to work on their own laptops. However, we will provide Internet connection as well as Azure accounts to each team for running large scale experiments. Access to certain MS proprietary datasets and development environment could also be provided based on need and availability.

Each project will be assigned a research collaborator as well as an engineer from MSR/Microsoft, who will actively involve in the research as well as the development aspects of the project.

Best Project Award

All projects will be presented and there will be a competition at the end of the workshop to choose the winning project. The winning project will be awarded seed funding of INR 700,000/- from Microsoft Research India to continue the work.

Based on their performance in the workshop, the UG/PG students participating in the workshop will be considered for internship at MSR, if they wish so.


For any queries on the proposal / eligibility, etc. please write to :, cc-ing Monojit Choudhury at


Program Committee

Organizing committee


For any queries on the proposal / eligibility, etc. please write to :, cc-ing Monojit Choudhury at

Selected Projects

 1. Accent Adaptation in ASR Systems.  

Proposer –  Prof. Preethi Jyothi, IIT Bombay

Abstract: Voice-driven automated agents such as personal assistants are becoming increasingly popular. However, in a multi-lingual and multi-cultural country like India, deploying such agents to engage with large sections of the population is highly challenging. A major hindrance in this regard is the difficulty the agents would face in understanding varying speech accents of the users. Even when the language of interaction with the underlying automatic speech recognition (ASR) system is restricted to a lingua franca (such as English), the accent of the speaker can vary dramatically based on their cultural and linguistic background, posing a fundamental challenge for ASR systems. Tackling this challenge will be a necessary first step towards building socially accepted and commercially successful agents in the Indian context.

The main focus of this project will be to take this first step, by improving state-of-the-art performance of ASR systems on accented speech – specifically, speech with Indian accents. We shall develop  deep neural network based acoustic models that will be trained using not only accented speech data but also speech in the native languages associated with the accent. We shall also develop a tool that will be trained to identify various Indian accents automatically. Finally, we shall investigate how accented-speech-ASR can be effectively incorporated into intelligent agents to help them act in socio-culturally appropriate ways.

2.  HollyChat: Domain Specific Conversation Systems.

Proposer – Prof. Mitesh Khapra, IIT Madras

Abstract: Most of the AI systems today are driven by three key components (i) data (ii) common sense knowledge and (iii) powerful learning algorithms which can harness this data and knowledge to learn task specific meaningful patterns. Recently there has been a lot of interest in domain-specific dialog systems with several downstream use cases such as shopping assistants, customer support, tour guides, etc. Most of the existing dialog systems are partly in line with the trend mentioned above – in the sense that they are data driven and use powerful algorithms (deep recurrent neural networks and their variants). However, we are nowhere close to building deployable domain-specific conversation systems. One of the primary reasons for this shortfall is that such agents do not exploit any common sense or real-world knowledge, and thereby are unable to maintain a richer context of the conversation. We propose to focus on domain specific conversation systems which use domain specific knowledge graphs as external memory. The idea is to use a domain-specific knowledge graph to discover the latent intent of the user and keep the conversation coherent with this intent. For example, the knowledge graph driven intention network could map the user’s utterance \textit{“I really liked the action sequences in Inception (movie)”} to all tuples containing the entity \textit{“Inception”} and keep the conversation restricted to concepts linked to this entity. An appropriate response in this case could be \textit{“Yes, indeed, movies directed by Christopher Nolan are known for their action”} which contains entities and predicates linked to \textit{“Inception”}. This would help in the task of dialog planning and also address the problem of keeping track of large contexts (which would be required for longer conversations containing many turns). In this case, the model could learn to abstract out the context in terms of entities and predicates in the knowledge base and just track these and their immediate neighborhood in the knowledge graph. 

3. Detection of Aggressive Behavior on Social Media.

Proposer: Prof. Ritesh Kumar, Ambedkar University

Abstract: As the interaction over the web has increased, incidents of aggression and related events like trolling, cyberbullying, flaming, hate speech, etc. too have increased manifold across the globe. While most of these behaviour like bullying or hate speech have predated the Internet, the reach and extent of the Internet has given these an unprecedented power and influence to affect the lives of billions of people. It has been observed that the incidents of aggression and unratified verbal behaviour has not remained just a minor nuisance but has acquired the form of a major criminal activity that affects a large number of people. These have not only created mental and psychological agony to the users of the web but has in fact forced people to deactivate their accounts and in rare instances also commit suicides. So it is of utmost significance and importance that some preventive measures be taken to safeguard the interests of the people using the web as well as of the web itself such that it remains a viable medium of communication and connection, in general.

The aim of the project is to develop a prototype that could automatically tell ratified (both aggressive as well as non-aggressive) linguistic behaviour from unratified (aggressive) ones (recognised by varied names like flaming, aggression, trolling, hate speech, cyberbullying, etc.) on the online forums (especially social media and news/opinion websites/blogs). I propose to develop the system using supervised text classification methods combined with sequence models that would be trained using a dataset annotated with labels pertaining to aggression in Hindi and Hindi-English code-mixed data collected from different kinds of Facebook Pages including those of news/media organisations, support/help groups, celebrity pages and other similar pages as well as from certain focussed topics/themes on Twitter.

4. Utilising Social Media for Disaster Relief: Linguistic Analysis of Resource Requests and Offers on Twitter.

Proposer: Prof. Saptarshi Ghosh, IIT Kharagpur

Abstract: Effective coordination of post-disaster relief operations depends critically on the availability of reliable situational information, as well as on citizen participation in the operations. The advent of online social media (e.g., Twitter, Facebook) and the widespread availability of mobile Internet today enable regular citizens to contribute to the relief operations, even if they are themselves stuck in the disaster effected zones. The aim of this project is to develop mechanisms for utilizing online social media for helping post-disaster relief operations. Specifically, our goal is to develop tools that help coordinate resource requests and resource offerings, to ensure optimal resource utilization during the disaster.
To this end, we first propose to analyze the linguistic characteristics of resource requests and resource offerings posted on Twitter during various disaster scenarios. This analysis is likely to yield insights into how different people phrase requests and offers for resources, in various languages. Next, we plan to utilize the insights obtained from the linguistic analysis, to build systems that will help coordinate the resource requests and offerings. Specifically, we envision building an automated bot that responds appropriately to resource requests and offers, and then matches corresponding requests and offers.


Schedule of Lectures / Tutorials

Date 09:30 – 11:00  11:30 – 13:00 14:30 – 17:00
5-Jun  Welcome & Introductions ASI: Monojit Choudhury Tutorial: Azure & Azure ML: Gopal Srinivasa
6-Jun Dialogue Systems: Alan Black Chat Bots: Alan Black Tutorial – Building Chatbots: Mitesh Khapra
7-Jun NLP & Social Media: Monojit Choudhury
8-Jun User Studies: Indrani Medhi Data Ethics & Privacy: Kalika Bali
9-Jun Speech recognition: Preethi Jyothi & Sriram Ganapathy Tutorial – Kaldi: Preethi Jyothi & Sriram Ganapathy
12-Jun Computational Sociolinguistics: A. Seza Doğruöz
13-Jun Multiligualism: Kalika Bali
14-Jun Computational Pragmatics: Ritesh Kumar & Monojit Choudhury
15-Jun Text Analytics: Thamar Solorio
16-Jun Social Media Analytics: Saptarshi Ghosh
19-Jun Status Update
20-Jun Error Analysis: Sunayana Sitaram
21-Jun ML for ASI – 1: Prateek Jain
22-Jun ML for ASI – 2: Praneeth Netrapalli
23-Jun Ethnography for Artificial Intelligence: Jacki O’Neill
29-Jun Final Presentations


Note: The above schedule is subject to change.