![\"\"](\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2021\/01\/interacting-with-instructional-videos-using-knowledge-supervision-1.png\")
Figure 1. This figure demonstrates a scenario where a question is asked on an instructional video, and the video is skipped directly to a related segment for effective teaching. Currently, YouTube relies on manual annotation for this feature.<\/p><\/div>\n
![\"\"](\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2021\/01\/interacting-with-instructional-videos-using-knowledge-supervision-2.png\")
Figure 2. Research challenges for asking \u201chow to\u201d questions, known as non-factoid questions with long answers for text and video modality.<\/p><\/div>\n
Professor Seung-won Hwang from Yonsei University in South Korea and his collaborators, Microsoft Research Asia\u2019s Nan Duan and Lei Ji, and Yonsei PhD students, Kyungjae Lee, Hojae Han, and Seungtaek Choi, have been investigating this topic. Despite active research conducted on Question Answering (QA), answering \u201chow to\u201d questions is an under-studied topic, and existing solutions targeting factoid QA cannot satisfactorily answer \u201chow to\u201d questions with long answers, especially for multi-modal QA. Research challenges are summarized in Figure 2, which show how to support non-factoid QA for text and video modality. For text modality, the researchers found that the length difference between short questions and long answers hinders neural matching between two representations. They addressed this issue using the MICRON framework (published at EMNLP 2019), which combines the advantages of representation- and interaction-based matching. Their proposed model combines contextual representations and also considers interactions between n-grams of multiple granularities.<\/p>\n
Their finding is extended to video QA. For this, there are two additional challenges (marked in red in Figure 2). Though documents are often semantically pre-segmented by paragraphs, segmenting them into answer units itself is a challenge in video QA. In addition, videos have multimodality, where semantics represented in both text and video modalities need to be aggregated. The researchers proposed a two-stage method, where Segmenter uses multimodal-BERT to generate segmentation candidates by predicting likely start\/end positions. Ranker then prioritizes candidates based on length-adaptive gating to overcome the length gap challenge discussed above. This idea is described in an AAAI 2020 paper entitled \u201cSegment-then-Rank: Non-factoid Question Answering on Instructional Videos.\u201d See Figure 3.<\/p>\n
![\"diagram\"](\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2021\/01\/interacting-with-instructional-videos-using-knowledge-supervision-3.png\")
Figure 3. Segment-then-rank framework for non-factoid video QA<\/p><\/div>\n
The team is continuing efforts to extract key segments from a full video, as Figure 4 illustrates. Existing video summarization extracts highlights but cannot support chaptering. The distinction with the current research is that it seeks to leverage external knowledge, such as recipes, to annotate key steps, which would help students discover key questions to ask for learning. A demo has been published at ECML 2020, available at: http:\/\/pcdeepred.yonsei.ac.kr\/IVSKS\/<\/p>\n
![\"graphical](\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2021\/01\/interacting-with-instructional-videos-using-knowledge-supervision-4.png\")