{"id":931014,"date":"2023-03-29T02:15:45","date_gmt":"2023-03-29T09:15:45","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-event&p=931014"},"modified":"2023-09-17T23:48:37","modified_gmt":"2023-09-18T06:48:37","slug":"the-workshop-on-understanding-and-evaluating-big-models-for-human-intelligence-and-learning","status":"publish","type":"msr-event","link":"https:\/\/www.microsoft.com\/en-us\/research\/event\/the-workshop-on-understanding-and-evaluating-big-models-for-human-intelligence-and-learning\/","title":{"rendered":"The Workshop on Understanding and Evaluating Big Models for Human Intelligence and Learning"},"content":{"rendered":"\n\n\n\n\n

Big models have revolutionized various domains, including natural language processing, computer vision, and speech recognition. However, their complexity poses significant challenges for understanding their relationship to human cognition and behaviour, and their implications on important fields such as education. Microsoft Research Asia invites leading experts from psychology, education, and computer science to this closed-door workshop.<\/p>\n\n\n\n

This exclusive event provides an opportunity to delve into the latest developments in big models, discuss their potential implicationas for psychology and education, and discover new research directions and challenges critical for the current era. The workshop aims to build a community of experts and leverage the benefits of multiple disciplines for promising collaborations.<\/p>\n\n\n\n

Organizing Committee<\/strong><\/p>\n\n\n\n

Xing Xie<\/a> (Chair), Fang Luo (opens in new tab)<\/span><\/a> (Chair), Xiting Wang<\/a> (Chair), Luning Sun (opens in new tab)<\/span><\/a> (Chair), Beibei Shi<\/a> (Co-Chair), Weizhe Shi<\/p>\n\n\n\n

<\/p>\n\n\n\n\n\n

Date: <\/strong>April 11, 2023<\/p>\n\n\n\n

Timetable:<\/strong><\/p>\n\n\n\n

\n\n\n\n\n\n\n\n\n\n\n\n\n
China Standard Time (UTC+8)<\/th>\nCentral European Summer Time (UTC+2)<\/th>\nEastern Daylight Time (UTC-4)<\/th>\nItem<\/th>\nSpeaker<\/th>\n<\/tr>\n<\/thead>\n
20:00 \u2013 20:05<\/td>\n14:00 \u2013 14:05<\/td>\n08:00 \u2013 08:05<\/td>\nOpening<\/td>\nHost:
Xiting Wang, Microsoft Research Asia

\nSpeaker:
\nFang Luo, Beijing Normal University\n<\/td>\n<\/tr>\n
Session 1: Assessing Human and AI Capabilities: Convergence and Integration<\/strong><\/td>\n<\/tr>\n
20:05 \u2013 20:25<\/td>\n14:05 \u2013 14:25<\/td>\n08:05 \u2013 08:25<\/td>\nKeynote: Integrating the Evaluation of Artificial and Natural Intelligence: Are We Ready Yet?\n
\n[slides<\/a>, video (opens in new tab)<\/span><\/a>]\n

\nAbstract: Despite a long-standing history of psychological theories and methods, the prevailing evaluation approach in AI is still based on observable task performance metrics. Yet this leaderboard culture has led to a crisis of predictability in AI. In this presentation we will briefly cover over two decades of efforts to incorporate ideas from psychology and other disciplines into AI evaluation, such as identifying difficulty levels, leveraging item response theory, conceptualising generality and employing a cognitive approach to infer capabilities. \n
\nThese methods have been carefully deployed under the umbrella of a universal comparative psychology, championed as the remedy to anthropocentrism in AI evaluation. Today, with the advent of large language models and other foundation models exhibiting a fascinating blend of general-purpose and human-like behaviour, it comes increasingly evident that task-oriented evaluation based on aggregate benchmark performance is not fit for purpose. Are we ready for a paradigm shift? \n
\nCan we assess the latest and rarest kinds of intelligence without the populational compass of human intelligence? This presentation will explore the most promising pathways for integrating AI and human evaluation, with a particular focus on a future where humans are augmented by AI.\n<\/td>\n		Host:
Xiting Wang, Microsoft Research Asia

\nSpeaker:
\nJos\u00e9 Hern\u00e1ndez-Orallo, Universitat Polit\u00e8cnica de Val\u00e8ncia, Leverhulme Centre for the Future of Intelligence, Centre for the Study of Existential Risk<\/td>\n<\/tr>\n
20:25 \u2013 21:55<\/td>\n14:25 \u2013 15:55<\/td>\n08:25 \u2013 09:55<\/td>\nPanel Discussion: Assessing Human and AI Capabilities: Convergence and Integration\n

\nAbstract: We will examine how psychological theories and methods can inform the evaluation and improvement of big models’ performance and robustness, and how the evaluation of big models may provide valuable insights for psychology.\n<\/td>\n		Host:
Xiting Wang, Microsoft Research Asia

\nPanelists:
\n
    \n
  • Jinyan Fan, Auburn University<\/li>\n
  • Jos\u00e9 Hern\u00e1ndez-Orallo, Universitat Polit\u00e8cnica de Val\u00e8ncia, Leverhulme Centre for the Future of Intelligence, Centre for the Study of Existential Risk<\/li>\n
  • Marija Slavkovik, University of Bergen<\/li>\n
  • Clemens Stachl, University of St. Gallen<\/li>\n
  • David Stillwell, University of Cambridge<\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
Session 2: The Future of Education with Big Models<\/strong><\/td>\n<\/tr>\n
22:00 \u2013 22:20<\/td>\n16:00 \u2013 16:20<\/td>\n10:00 \u2013 10:20<\/td>\nKeynote: \u201cAre we there yet?\u201d \u2014The promise of the Holodeck for the future of education\n
\n[slides<\/a>, video (opens in new tab)<\/span><\/a>]\n

\nAbstract: Since its introduction in 1974 in the Star Wars television franchise, visionaries of education dreamed of the virtual and malleable space that can teach, train, assess by immersion, the Holodeck. The recent launches of large multimodal computational models made us all dream of the modern Socratic tutor accessible to all, the immersive experience of learning and training, and of virtual in-situ multimodal assessments. So, are we there yet?
\nIn this presentation I\u2019ll focus on three areas of interest: 1. Construct definition: what and how should we teach in the times of AI? 2. Assessment design & development: what and how to assess the relevant skills in the New World; 3. Social context: what type of guardrails do we need to protect and support students, teachers and integrity of the educational experience.\n
\nI will conclude with my personal thoughts on the value of humanity in a fast world of technological prowess and on the dream of the Holodeck.\n<\/td>\n		Host:
Luning Sun, University of Cambridge

\nSpeaker:
\nAlina A von Davier, Duolingo, EdAstra Tech, University of Oxford, Carnegie Mellon University\n<\/td>\n<\/tr>\n
22:20 \u2013 23:55<\/td>\n16:20 \u2013 17:55<\/td>\n10:20 \u2013 11:55<\/td>\nPanel Discussion: The Future of Education with Big Models\n

\nAbstract: We will address the educational implications of big model development and deployment, including preparing the next generation for a world where AI is ubiquitous and influential.\n<\/td>\n		Host:
\nLuning Sun, University of Cambridge\n

\nPanelists:
\n
    \n
  • Xiangen Hu, The University of Memphis, Central China Normal University<\/li>\n
  • Yu Lu, Beijing Normal University<\/li>\n
  • Bryan Maddox, University of Cambridge, University of East Anglia, University of Oslo, Assessment MicroAnalytics Ltd<\/li>\n
  • Alina A von Davier, Duolingo, EdAstra Tech, University of Oxford, Carnegie Mellon University<\/li>\n
  • Mengxiao Zhu, University of Science and Technology of China<\/li>\n<\/ul>\n<\/td><\/tr>\n
23:55-24:00<\/td>\n17:55 \u2013 18:00<\/td>\n11:55 \u2013 12:00<\/td>\nClosing<\/td>\nXing Xie, Microsoft Research Asia<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n\n\n\n\n\n
\"Jinyan<\/figure>
\n

Jinyan Fan (opens in new tab)<\/span><\/a>, Auburn University<\/p>\n\n\n\n

Dr. Jinyan Fan earned a bachelor\u2019s degree in psychology in 1994 and a master\u2019s degree in applied psychology in 1997, both from East China Normal University, and then earned a Ph.D. in industrial\/organizational psychology from Ohio State University in 2004. Dr. Fan currently is a full professor at the Dept. of Psychological Sciences of Auburn University. His research interests are in the domains of artificial intelligence, personnel selection, newcomer orientation and socialization, and cross-cultural adjustment and training. His work has appeared in premier journals in the field of I\/O Psychology such as Journal of Applied Psychology, Journal of Management<\/em>, Journal of Organizational Behavior, and Journal of Vocational Behavior<\/em>. Dr. Fan has received several research awards and funds from the Society for Industrial and Organizational Psychology and the Academy of Management. He served as an Associate Editor of Journal of Vocational Behavior<\/em> between 2019 and 2021. Dr. Fan has also developed a series of talent assessment tools and has been active in providing related service to various organizations.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"a<\/figure>
\n

Jos\u00e9 Hern\u00e1ndez-Orallo (opens in new tab)<\/span><\/a>, Universitat Polit\u00e8cnica de Val\u00e8ncia, Leverhulme Centre for the Future of Intelligence, Centre for the Study of Existential Risk<\/p>\n\n\n\n

Jos\u00e9 Hern\u00e1ndez-Orallo is Professor at the Universitat Polit\u00e8cnica de Val\u00e8ncia, Spain and Senior Research Fellow at the Leverhulme Centre for the Future of Intelligence, University of Cambridge, UK. He received a B.Sc. and a M.Sc. in Computer Science from UPV, partly completed at the \u00c9cole Nationale Sup\u00e9rieure de l’\u00c9lectronique et de ses Applications (France), and a Ph.D. in Logic and Philosophy of Science with a doctoral extraordinary prize from the University of Valencia. His academic and research activities have spanned several areas of artificial intelligence, machine learning, data science and intelligence measurement, with a focus on a more insightful analysis of the capabilities, generality, progress, impact and risks of artificial intelligence. He has published five books and more than two hundred journal articles and conference papers on these topics. His research in the area of machine intelligence evaluation has been covered by several popular outlets, such as The Economist, New Scientist or Nature. He keeps exploring a more integrated view of the evaluation of natural and artificial intelligence, as vindicated in his book “The Measure of All Minds” (Cambridge University Press, 2017, PROSE Award 2018). He is a member of AAAI, CLAIRE and ELLIS, and a EurAI Fellow.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Xiangen<\/figure>
\n

Xiangen Hu, (opens in new tab)<\/span><\/a> The University of Memphis, Central China Normal University<\/p>\n\n\n\n

Dr. Xiangen Hu is a professor who specializes in Psychology, Electrical and Computer Engineering, and Computer Science. He currently holds positions at both The University of Memphis (UofM) and Central China Normal University (CCNU), where he serves as the Dean of the School of Psychology. Dr. Hu has completed a MS in applied mathematics, a MA in social sciences, and a Ph.D. in Cognitive Sciences.<\/p>\n\n\n\n

Dr. Hu’s research interests are broad and include Mathematical Psychology, Research Design and Statistics, and Cognitive Psychology. He is the Director of the Advanced Distributed Learning (ADL) Partnership Laboratory at the UofM, and he has specific interests in General Processing Tree (GPT) models, categorical data analysis, knowledge representation, computerized tutoring, and advanced distributed learning.<\/p>\n\n\n\n

Dr. Hu has secured funding for his research from multiple sources, including the US National Science Foundation (NSF), US Institute of Education Sciences (IES), ADL of the US Department of Defense (DoD), US Army Medical Research Acquisition Activity (USAMRAA), and US Army Research Laboratories (ARL). Dr. Hu’s exceptional research background and funding support have enabled him to make significant contributions to his field. He continues to conduct research and mentor students in his areas of expertise, and his work has been recognized by scholars around the world.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Yu<\/figure>
\n

Yu Lu (opens in new tab)<\/span><\/a>, Beijing Normal University<\/p>\n\n\n\n

Yu Lu received the Ph.D. from National University of Singapore in computer engineering. He is currently an Associate Professor with the School of Educational Technology, Faculty of Education, Beijing Normal University (BNU), where he also serves as the director of AI Lab at the advanced innovation center for future education (AICFE). He has published more than 80 academic papers in the prestigious journals and conferences (e.g., IJAIED, IEEE TKDE, IEEE TLT), and serves as the PC member or track chair for multiple international conferences (e.g., AIED, EDM, AAAI, ACL, EMNLP, IJCAI). Before joining BNU, he was a research scientist and principle investigator at the Institute for Infocomm Research (I2R), A*STAR, Singapore. His research interests mainly sit at the intersection field of artificial intelligence and education.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Fang<\/figure>
\n

Fang Luo (opens in new tab)<\/span><\/a>, Beijing Normal University<\/p>\n\n\n\n

Fang Luo, Professor, Faculty of Psychology, Beijing Normal University. My field is Psychometrics, and my recent research are focused on these aspects, they are research on developing human-computer interactive test and data mining of process data, the method of identifying cheating in large-scale exams, and data analysis methods and applications of large-scale educational testing. Using multimedia technology, we have developed interactive tests of creativity, critical thinking and complex problem solving to evaluate higher-order thinking ability. We draw on artificial intelligence and big data mining technology to try to evaluate personality traits from the daily performance of an individual, such as network traces, conversation content, free painting and so on.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Bryan<\/figure>
\n

Bryan Maddox, (opens in new tab)<\/span><\/a> University of Cambridge, University of East Anglia, University of Oslo, Assessment MicroAnalytics Ltd<\/p>\n\n\n\n

Professor Bryan Maddox leads the assessment strand at the Digital Education Futures Initiative (DEFI), Hughes Hall, University of Cambridge.  A social anthropologist by training, his research focuses on validity, diversity, and inclusion in large-scale educational assessments.  He is skilled in observational research methods including digital ethnography, verbal interaction, eye tracking and gesture analysis. Bryan is also Professor of Educational Assessment at the University of East Anglia, Visiting Professor at the Centre for Educational Measurement, University of Oslo, and Executive Director of Assessment MicroAnalytics Ltd.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Marija<\/figure>
\n

Marija Slavkovik (opens in new tab)<\/span><\/a>, University of Bergen<\/p>\n\n\n\n

Marija Slavkovik is a full Professor with the Faculty for Social Sciences of the University of Bergen and chair of the Department of Information Science and Media Studies. Her background is in computer science and artificial intelligence. She has been doing research in machine ethics since 2012. Machine ethics studies how moral reasoning can or should be automated.  Marija works on formalising ethical collective decision-making. She serves in the boards of several AI associations in Norway and in Europe.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"a<\/figure>
\n

Clemens Stachl (opens in new tab)<\/span><\/a>, University of St. Gallen<\/p>\n\n\n\n

Clemens Stachl is an Associate Professor of Behavioral Science at the University of St. Gallen, who uses computational technologies to study human behavior, experiences, and preferences. His research focuses on recognizing individual differences through digital footprints, understanding the implications of digital behavioral data in algorithmic decision-making, and designing intelligent systems and services that consider stable user traits and momentary states. Stachl holds a PhD in Psychology and has worked as a postdoctoral scholar at Stanford University and Ludwig-Maximilians-Universit\u00e4t M\u00fcnchen. He has published numerous papers in leading journals and conferences in the fields of Personality Psychology, Behavioral Science, and Human-Computer Interaction, and his work has been recognized with the best paper award from the German Society for Online-Research and the price for digital assessment from the German Psychological Society. His research has also been featured in worldwide media outlets, including Forbes, Harvard Business Review, Fast Company, and Die Zeit. Stachl’s work lies at the intersection of behavioral and computational sciences, and he aims to use technology to objectively quantify and study everyday human behavior and psychological processes. He is interested in how AI and machine learning can be used to better understand human behavior and experiences, and how intelligent systems and services can support people in their daily lives.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"David<\/figure>
\n

David Stillwell (opens in new tab)<\/span><\/a>, University of Cambridge<\/p>\n\n\n\n

David studies the links between big data and psychology; his research with 6 million social media users found that the computer can predict a user\u2019s personality as accurately as their spouse can. Follow up research found that personalizing an advert to the recipient\u2019s psychology is more effective than generic ads. David has also published research using various big data sources to show that spending money on products and services that match one\u2019s personality leads to greater life satisfaction, that people tend to date others who have a similar personality, and that people who swear seem to be more honest.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"a<\/figure>
\n

Luning Sun (opens in new tab)<\/span><\/a>, University of Cambridge<\/p>\n\n\n\n

Dr Luning Sun is a Research Associate at Judge Business School in the University of Cambridge. He is also Research Director of the Psychometrics Centre. Luning is interested in the new forms of assessment and their applications in different contexts.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Alina<\/figure>
\n

Alina A von Davier (opens in new tab)<\/span><\/a>, Duolingo, EdAstra Tech, University of Oxford, Carnegie Mellon University<\/p>\n\n\n\n

Alina A von Davier is a researcher, innovator, and an executive leader with over 20 years of experience in EdTech and in the assessment industry. She is the Chief of Assessment at Duolingo (opens in new tab)<\/span><\/a>, where she leads the Duolingo English Test (opens in new tab)<\/span><\/a> research and development area. She is also the Founder and CEO of EdAstra Tech, a service-oriented EdTech company. In 2022, she joined the University of Oxford (opens in new tab)<\/span><\/a> as an Honorary Research Fellow, and Carnegie Mellon University (opens in new tab)<\/span><\/a> as a Senior Research Fellow. Her research interests include computational psychometrics, and AI and ML in learning and assessment.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"a<\/figure>
\n

Xiting Wang<\/a>, Microsoft Research Asia<\/p>\n\n\n\n

Xiting Wang is a principal researcher at Microsoft Research Asia. Her research interest is explainable and trustworthy AI. She has published around 50 papers on top conferences or journals. Two of her papers were selected as the spotlight article by a top journal IEEE TVCG. The technologies developed by Xiting have been applied in multiple products like Microsoft Bing and Microsoft News, impacting millions of users each month. Xiting is an area chair of IJCAI 2023 and was awarded Best SPC by AAAI 2021.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Xing<\/figure>
\n

Xing Xie<\/a>, Microsoft Research Asia<\/p>\n\n\n\n

Dr. Xing Xie is a senior principal research manager of Microsoft Research Asia. He received his B.S. and Ph.D. degrees in Computer Science from the University of Science and Technology of China in 1996 and 2001, respectively. He joined Microsoft Research Asia in July 2001, working on data mining, social computing and ubiquitous computing. During the past years, he has published over 300 papers, won the ACM SIGKDD 2022 test-of-time award, the ACM SIGKDD China 2021 test of time award, the 10-year impact award honorable mention in ACM SIGSPATIAL 2020, the 10-year impact award in ACM SIGSPATIAL 2019, the best student paper award in KDD 2016, and the best paper awards in ICDM 2013 and UIC 2010. He is a Fellow of China Computer Federation (CCF) and the IEEE, and a Distinguished Member of ACM.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Xiaozhu<\/figure>
\n

Mengxiao Zhu (opens in new tab)<\/span><\/a>, University of Science and Technology of China<\/p>\n\n\n\n

Dr. Mengxiao Zhu is a Distinguished Research Professor in the Department of Science and Technology Communication and School of Humanities and Social Sciences, at University of Science and Technology of China. She earned her Ph.D. Degree from the Department of Industrial Engineering and Management Sciences at Northwestern University. She holds degrees in Communication (M.A.) from the University of Illinois at Urbana-Champaign, Computer Science (M.E. & B.E.) and Science and English (B.S.) from the University of Science and Technology of China. Before joining USTC, she worked as a Research Scientist in the Research and Development division at Educational Testing Service (ETS) for over seven years. Prior to that, she also worked as a Post-doctoral Research Associate in the School of Communication and Information at Rutgers University, and a Graduate Research Assistant in the Science of Networks in Communities (SONIC) Research Group led by Professor Noshir Contractor. She has been involved in several NSF and NIH -funded projects focusing on computer-mediated communication in emergency response teams, and on the development of knowledge networks and the dynamics of collaborations both in real world, such as research institutions, and in virtual worlds, such as Second Life and online role-playing games. Her current research interests include computational methods in communication, social networks and social media, and the interactions of AI and human in communication and education.<\/p>\n<\/div><\/div>\n\n\n","protected":false},"excerpt":{"rendered":"

Big models have revolutionized various domains, including natural language processing, computer vision, and speech recognition. However, their complexity poses significant challenges for understanding their relationship to human cognition and behaviour, and their implications on important fields such as education. Microsoft Research Asia invites leading experts from psychology, education, and computer science to this closed-door workshop. […]<\/p>\n","protected":false},"featured_media":932628,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr_startdate":"2023-04-11","msr_enddate":"2023-04-11","msr_location":"Virtual","msr_expirationdate":"","msr_event_recording_link":"","msr_event_link":"","msr_event_link_redirect":false,"msr_event_time":"","msr_hide_region":false,"msr_private_event":false,"msr_hide_image_in_river":0,"footnotes":""},"research-area":[13556],"msr-region":[256048],"msr-event-type":[197944],"msr-video-type":[],"msr-locale":[268875],"msr-program-audience":[],"msr-post-option":[],"msr-impact-theme":[],"class_list":["post-931014","msr-event","type-msr-event","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-region-global","msr-event-type-hosted-by-microsoft","msr-locale-en_us"],"msr_about":"\n\n\n\n\n

Big models have revolutionized various domains, including natural language processing, computer vision, and speech recognition. However, their complexity poses significant challenges for understanding their relationship to human cognition and behaviour, and their implications on important fields such as education. Microsoft Research Asia invites leading experts from psychology, education, and computer science to this closed-door workshop.<\/p>\n\n\n\n

This exclusive event provides an opportunity to delve into the latest developments in big models, discuss their potential implicationas for psychology and education, and discover new research directions and challenges critical for the current era. The workshop aims to build a community of experts and leverage the benefits of multiple disciplines for promising collaborations.<\/p>\n\n\n\n

Organizing Committee<\/strong><\/p>\n\n\n\n

Xing Xie<\/a> (Chair), Fang Luo<\/a> (Chair), Xiting Wang<\/a> (Chair), Luning Sun<\/a> (Chair), Beibei Shi<\/a> (Co-Chair), Weizhe Shi<\/p>\n\n\n\n

<\/p>\n\n\n\n\n\n

Date: <\/strong>April 11, 2023<\/p>\n\n\n\n

Timetable:<\/strong><\/p>\n\n\n\n

\n\n\n\n\n\n\n\n\n\n\n\n\n
China Standard Time (UTC+8)<\/th>\nCentral European Summer Time (UTC+2)<\/th>\nEastern Daylight Time (UTC-4)<\/th>\nItem<\/th>\nSpeaker<\/th>\n<\/tr>\n<\/thead>\n
20:00 \u2013 20:05<\/td>\n14:00 \u2013 14:05<\/td>\n08:00 \u2013 08:05<\/td>\nOpening<\/td>\nHost:
Xiting Wang, Microsoft Research Asia

\nSpeaker:
\nFang Luo, Beijing Normal University\n<\/td>\n<\/tr>\n
Session 1: Assessing Human and AI Capabilities: Convergence and Integration<\/strong><\/td>\n<\/tr>\n
20:05 \u2013 20:25<\/td>\n14:05 \u2013 14:25<\/td>\n08:05 \u2013 08:25<\/td>\nKeynote: Integrating the Evaluation of Artificial and Natural Intelligence: Are We Ready Yet?\n
\n[slides<\/a>, video<\/a>]\n

\nAbstract: Despite a long-standing history of psychological theories and methods, the prevailing evaluation approach in AI is still based on observable task performance metrics. Yet this leaderboard culture has led to a crisis of predictability in AI. In this presentation we will briefly cover over two decades of efforts to incorporate ideas from psychology and other disciplines into AI evaluation, such as identifying difficulty levels, leveraging item response theory, conceptualising generality and employing a cognitive approach to infer capabilities. \n
\nThese methods have been carefully deployed under the umbrella of a universal comparative psychology, championed as the remedy to anthropocentrism in AI evaluation. Today, with the advent of large language models and other foundation models exhibiting a fascinating blend of general-purpose and human-like behaviour, it comes increasingly evident that task-oriented evaluation based on aggregate benchmark performance is not fit for purpose. Are we ready for a paradigm shift? \n
\nCan we assess the latest and rarest kinds of intelligence without the populational compass of human intelligence? This presentation will explore the most promising pathways for integrating AI and human evaluation, with a particular focus on a future where humans are augmented by AI.\n<\/td>\n		Host:
Xiting Wang, Microsoft Research Asia

\nSpeaker:
\nJos\u00e9 Hern\u00e1ndez-Orallo, Universitat Polit\u00e8cnica de Val\u00e8ncia, Leverhulme Centre for the Future of Intelligence, Centre for the Study of Existential Risk<\/td>\n<\/tr>\n
20:25 \u2013 21:55<\/td>\n14:25 \u2013 15:55<\/td>\n08:25 \u2013 09:55<\/td>\nPanel Discussion: Assessing Human and AI Capabilities: Convergence and Integration\n

\nAbstract: We will examine how psychological theories and methods can inform the evaluation and improvement of big models' performance and robustness, and how the evaluation of big models may provide valuable insights for psychology.\n<\/td>\n		Host:
Xiting Wang, Microsoft Research Asia

\nPanelists:
\n
    \n
  • Jinyan Fan, Auburn University<\/li>\n
  • Jos\u00e9 Hern\u00e1ndez-Orallo, Universitat Polit\u00e8cnica de Val\u00e8ncia, Leverhulme Centre for the Future of Intelligence, Centre for the Study of Existential Risk<\/li>\n
  • Marija Slavkovik, University of Bergen<\/li>\n
  • Clemens Stachl, University of St. Gallen<\/li>\n
  • David Stillwell, University of Cambridge<\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
Session 2: The Future of Education with Big Models<\/strong><\/td>\n<\/tr>\n
22:00 \u2013 22:20<\/td>\n16:00 \u2013 16:20<\/td>\n10:00 \u2013 10:20<\/td>\nKeynote: \u201cAre we there yet?\u201d \u2014The promise of the Holodeck for the future of education\n
\n[slides<\/a>, video<\/a>]\n

\nAbstract: Since its introduction in 1974 in the Star Wars television franchise, visionaries of education dreamed of the virtual and malleable space that can teach, train, assess by immersion, the Holodeck. The recent launches of large multimodal computational models made us all dream of the modern Socratic tutor accessible to all, the immersive experience of learning and training, and of virtual in-situ multimodal assessments. So, are we there yet?
\nIn this presentation I\u2019ll focus on three areas of interest: 1. Construct definition: what and how should we teach in the times of AI? 2. Assessment design & development: what and how to assess the relevant skills in the New World; 3. Social context: what type of guardrails do we need to protect and support students, teachers and integrity of the educational experience.\n
\nI will conclude with my personal thoughts on the value of humanity in a fast world of technological prowess and on the dream of the Holodeck.\n<\/td>\n		Host:
Luning Sun, University of Cambridge

\nSpeaker:
\nAlina A von Davier, Duolingo, EdAstra Tech, University of Oxford, Carnegie Mellon University\n<\/td>\n<\/tr>\n
22:20 \u2013 23:55<\/td>\n16:20 \u2013 17:55<\/td>\n10:20 \u2013 11:55<\/td>\nPanel Discussion: The Future of Education with Big Models\n

\nAbstract: We will address the educational implications of big model development and deployment, including preparing the next generation for a world where AI is ubiquitous and influential.\n<\/td>\n		Host:
\nLuning Sun, University of Cambridge\n

\nPanelists:
\n
    \n
  • Xiangen Hu, The University of Memphis, Central China Normal University<\/li>\n
  • Yu Lu, Beijing Normal University<\/li>\n
  • Bryan Maddox, University of Cambridge, University of East Anglia, University of Oslo, Assessment MicroAnalytics Ltd<\/li>\n
  • Alina A von Davier, Duolingo, EdAstra Tech, University of Oxford, Carnegie Mellon University<\/li>\n
  • Mengxiao Zhu, University of Science and Technology of China<\/li>\n<\/ul>\n<\/td><\/tr>\n
23:55-24:00<\/td>\n17:55 \u2013 18:00<\/td>\n11:55 \u2013 12:00<\/td>\nClosing<\/td>\nXing Xie, Microsoft Research Asia<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n\n\n\n\n\n
\"Jinyan<\/figure>
\n

Jinyan Fan<\/a>, Auburn University<\/p>\n\n\n\n

Dr. Jinyan Fan earned a bachelor\u2019s degree in psychology in 1994 and a master\u2019s degree in applied psychology in 1997, both from East China Normal University, and then earned a Ph.D. in industrial\/organizational psychology from Ohio State University in 2004. Dr. Fan currently is a full professor at the Dept. of Psychological Sciences of Auburn University. His research interests are in the domains of artificial intelligence, personnel selection, newcomer orientation and socialization, and cross-cultural adjustment and training. His work has appeared in premier journals in the field of I\/O Psychology such as Journal of Applied Psychology, Journal of Management<\/em>, Journal of Organizational Behavior, and Journal of Vocational Behavior<\/em>. Dr. Fan has received several research awards and funds from the Society for Industrial and Organizational Psychology and the Academy of Management. He served as an Associate Editor of Journal of Vocational Behavior<\/em> between 2019 and 2021. Dr. Fan has also developed a series of talent assessment tools and has been active in providing related service to various organizations.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"a<\/figure>
\n

Jos\u00e9 Hern\u00e1ndez-Orallo<\/a>, Universitat Polit\u00e8cnica de Val\u00e8ncia, Leverhulme Centre for the Future of Intelligence, Centre for the Study of Existential Risk<\/p>\n\n\n\n

Jos\u00e9 Hern\u00e1ndez-Orallo is Professor at the Universitat Polit\u00e8cnica de Val\u00e8ncia, Spain and Senior Research Fellow at the Leverhulme Centre for the Future of Intelligence, University of Cambridge, UK. He received a B.Sc. and a M.Sc. in Computer Science from UPV, partly completed at the \u00c9cole Nationale Sup\u00e9rieure de l'\u00c9lectronique et de ses Applications (France), and a Ph.D. in Logic and Philosophy of Science with a doctoral extraordinary prize from the University of Valencia. His academic and research activities have spanned several areas of artificial intelligence, machine learning, data science and intelligence measurement, with a focus on a more insightful analysis of the capabilities, generality, progress, impact and risks of artificial intelligence. He has published five books and more than two hundred journal articles and conference papers on these topics. His research in the area of machine intelligence evaluation has been covered by several popular outlets, such as The Economist, New Scientist or Nature. He keeps exploring a more integrated view of the evaluation of natural and artificial intelligence, as vindicated in his book \"The Measure of All Minds\" (Cambridge University Press, 2017, PROSE Award 2018). He is a member of AAAI, CLAIRE and ELLIS, and a EurAI Fellow.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Xiangen<\/figure>
\n

Xiangen Hu,<\/a> The University of Memphis, Central China Normal University<\/p>\n\n\n\n

Dr. Xiangen Hu is a professor who specializes in Psychology, Electrical and Computer Engineering, and Computer Science. He currently holds positions at both The University of Memphis (UofM) and Central China Normal University (CCNU), where he serves as the Dean of the School of Psychology. Dr. Hu has completed a MS in applied mathematics, a MA in social sciences, and a Ph.D. in Cognitive Sciences.<\/p>\n\n\n\n

Dr. Hu's research interests are broad and include Mathematical Psychology, Research Design and Statistics, and Cognitive Psychology. He is the Director of the Advanced Distributed Learning (ADL) Partnership Laboratory at the UofM, and he has specific interests in General Processing Tree (GPT) models, categorical data analysis, knowledge representation, computerized tutoring, and advanced distributed learning.<\/p>\n\n\n\n

Dr. Hu has secured funding for his research from multiple sources, including the US National Science Foundation (NSF), US Institute of Education Sciences (IES), ADL of the US Department of Defense (DoD), US Army Medical Research Acquisition Activity (USAMRAA), and US Army Research Laboratories (ARL). Dr. Hu's exceptional research background and funding support have enabled him to make significant contributions to his field. He continues to conduct research and mentor students in his areas of expertise, and his work has been recognized by scholars around the world.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Yu<\/figure>
\n

Yu Lu<\/a>, Beijing Normal University<\/p>\n\n\n\n

Yu Lu received the Ph.D. from National University of Singapore in computer engineering. He is currently an Associate Professor with the School of Educational Technology, Faculty of Education, Beijing Normal University (BNU), where he also serves as the director of AI Lab at the advanced innovation center for future education (AICFE). He has published more than 80 academic papers in the prestigious journals and conferences (e.g., IJAIED, IEEE TKDE, IEEE TLT), and serves as the PC member or track chair for multiple international conferences (e.g., AIED, EDM, AAAI, ACL, EMNLP, IJCAI). Before joining BNU, he was a research scientist and principle investigator at the Institute for Infocomm Research (I2R), A*STAR, Singapore. His research interests mainly sit at the intersection field of artificial intelligence and education.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Fang<\/figure>
\n

Fang Luo<\/a>, Beijing Normal University<\/p>\n\n\n\n

Fang Luo, Professor, Faculty of Psychology, Beijing Normal University. My field is Psychometrics, and my recent research are focused on these aspects, they are research on developing human-computer interactive test and data mining of process data, the method of identifying cheating in large-scale exams, and data analysis methods and applications of large-scale educational testing. Using multimedia technology, we have developed interactive tests of creativity, critical thinking and complex problem solving to evaluate higher-order thinking ability. We draw on artificial intelligence and big data mining technology to try to evaluate personality traits from the daily performance of an individual, such as network traces, conversation content, free painting and so on.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Bryan<\/figure>
\n

Bryan Maddox,<\/a> University of Cambridge, University of East Anglia, University of Oslo, Assessment MicroAnalytics Ltd<\/p>\n\n\n\n

Professor Bryan Maddox leads the assessment strand at the Digital Education Futures Initiative (DEFI), Hughes Hall, University of Cambridge.  A social anthropologist by training, his research focuses on validity, diversity, and inclusion in large-scale educational assessments.  He is skilled in observational research methods including digital ethnography, verbal interaction, eye tracking and gesture analysis. Bryan is also Professor of Educational Assessment at the University of East Anglia, Visiting Professor at the Centre for Educational Measurement, University of Oslo, and Executive Director of Assessment MicroAnalytics Ltd.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Marija<\/figure>
\n

Marija Slavkovik<\/a>, University of Bergen<\/p>\n\n\n\n

Marija Slavkovik is a full Professor with the Faculty for Social Sciences of the University of Bergen and chair of the Department of Information Science and Media Studies. Her background is in computer science and artificial intelligence. She has been doing research in machine ethics since 2012. Machine ethics studies how moral reasoning can or should be automated.  Marija works on formalising ethical collective decision-making. She serves in the boards of several AI associations in Norway and in Europe.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"a<\/figure>
\n

Clemens Stachl<\/a>, University of St. Gallen<\/p>\n\n\n\n

Clemens Stachl is an Associate Professor of Behavioral Science at the University of St. Gallen, who uses computational technologies to study human behavior, experiences, and preferences. His research focuses on recognizing individual differences through digital footprints, understanding the implications of digital behavioral data in algorithmic decision-making, and designing intelligent systems and services that consider stable user traits and momentary states. Stachl holds a PhD in Psychology and has worked as a postdoctoral scholar at Stanford University and Ludwig-Maximilians-Universit\u00e4t M\u00fcnchen. He has published numerous papers in leading journals and conferences in the fields of Personality Psychology, Behavioral Science, and Human-Computer Interaction, and his work has been recognized with the best paper award from the German Society for Online-Research and the price for digital assessment from the German Psychological Society. His research has also been featured in worldwide media outlets, including Forbes, Harvard Business Review, Fast Company, and Die Zeit. Stachl's work lies at the intersection of behavioral and computational sciences, and he aims to use technology to objectively quantify and study everyday human behavior and psychological processes. He is interested in how AI and machine learning can be used to better understand human behavior and experiences, and how intelligent systems and services can support people in their daily lives.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"David<\/figure>
\n

David Stillwell<\/a>, University of Cambridge<\/p>\n\n\n\n

David studies the links between big data and psychology; his research with 6 million social media users found that the computer can predict a user\u2019s personality as accurately as their spouse can. Follow up research found that personalizing an advert to the recipient\u2019s psychology is more effective than generic ads. David has also published research using various big data sources to show that spending money on products and services that match one\u2019s personality leads to greater life satisfaction, that people tend to date others who have a similar personality, and that people who swear seem to be more honest.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"a<\/figure>
\n

Luning Sun<\/a>, University of Cambridge<\/p>\n\n\n\n

Dr Luning Sun is a Research Associate at Judge Business School in the University of Cambridge. He is also Research Director of the Psychometrics Centre. Luning is interested in the new forms of assessment and their applications in different contexts.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Alina<\/figure>
\n

Alina A von Davier<\/a>, Duolingo, EdAstra Tech, University of Oxford, Carnegie Mellon University<\/p>\n\n\n\n

Alina A von Davier is a researcher, innovator, and an executive leader with over 20 years of experience in EdTech and in the assessment industry. She is the Chief of Assessment at Duolingo<\/a>, where she leads the Duolingo English Test<\/a> research and development area. She is also the Founder and CEO of EdAstra Tech, a service-oriented EdTech company. In 2022, she joined the University of Oxford<\/a> as an Honorary Research Fellow, and Carnegie Mellon University<\/a> as a Senior Research Fellow. Her research interests include computational psychometrics, and AI and ML in learning and assessment.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"a<\/figure>
\n

Xiting Wang<\/a>, Microsoft Research Asia<\/p>\n\n\n\n

Xiting Wang is a principal researcher at Microsoft Research Asia. Her research interest is explainable and trustworthy AI. She has published around 50 papers on top conferences or journals. Two of her papers were selected as the spotlight article by a top journal IEEE TVCG. The technologies developed by Xiting have been applied in multiple products like Microsoft Bing and Microsoft News, impacting millions of users each month. Xiting is an area chair of IJCAI 2023 and was awarded Best SPC by AAAI 2021.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Xing<\/figure>
\n

Xing Xie<\/a>, Microsoft Research Asia<\/p>\n\n\n\n

Dr. Xing Xie is a senior principal research manager of Microsoft Research Asia. He received his B.S. and Ph.D. degrees in Computer Science from the University of Science and Technology of China in 1996 and 2001, respectively. He joined Microsoft Research Asia in July 2001, working on data mining, social computing and ubiquitous computing. During the past years, he has published over 300 papers, won the ACM SIGKDD 2022 test-of-time award, the ACM SIGKDD China 2021 test of time award, the 10-year impact award honorable mention in ACM SIGSPATIAL 2020, the 10-year impact award in ACM SIGSPATIAL 2019, the best student paper award in KDD 2016, and the best paper awards in ICDM 2013 and UIC 2010. He is a Fellow of China Computer Federation (CCF) and the IEEE, and a Distinguished Member of ACM.<\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n
\"Xiaozhu<\/figure>
\n

Mengxiao Zhu<\/a>, University of Science and Technology of China<\/p>\n\n\n\n

Dr. Mengxiao Zhu is a Distinguished Research Professor in the Department of Science and Technology Communication and School of Humanities and Social Sciences, at University of Science and Technology of China. She earned her Ph.D. Degree from the Department of Industrial Engineering and Management Sciences at Northwestern University. She holds degrees in Communication (M.A.) from the University of Illinois at Urbana-Champaign, Computer Science (M.E. & B.E.) and Science and English (B.S.) from the University of Science and Technology of China. Before joining USTC, she worked as a Research Scientist in the Research and Development division at Educational Testing Service (ETS) for over seven years. Prior to that, she also worked as a Post-doctoral Research Associate in the School of Communication and Information at Rutgers University, and a Graduate Research Assistant in the Science of Networks in Communities (SONIC) Research Group led by Professor Noshir Contractor. She has been involved in several NSF and NIH -funded projects focusing on computer-mediated communication in emergency response teams, and on the development of knowledge networks and the dynamics of collaborations both in real world, such as research institutions, and in virtual worlds, such as Second Life and online role-playing games. Her current research interests include computational methods in communication, social networks and social media, and the interactions of AI and human in communication and education.<\/p>\n<\/div><\/div>\n\n\n","tab-content":[],"msr_startdate":"2023-04-11","msr_enddate":"2023-04-11","msr_event_time":"","msr_location":"Virtual","msr_event_link":"","msr_event_recording_link":"","msr_startdate_formatted":"April 11, 2023","msr_register_text":"Watch now","msr_cta_link":"","msr_cta_text":"","msr_cta_bi_name":"","featured_image_thumbnail":"\"banner\"","event_excerpt":"Big models have revolutionized various domains, including natural language processing, computer vision, and speech recognition. However, their complexity poses significant challenges for understanding their relationship to human cognition and behaviour, and their implications on important fields such as education. Microsoft Research Asia invites leading experts from psychology, education, and computer science to this closed-door workshop. This exclusive event provides an opportunity to delve into the latest developments in big models, discuss their potential implicationas for…","msr_research_lab":[199560,1012650],"related-researchers":[],"msr_impact_theme":[],"related-academic-programs":[],"related-groups":[],"related-projects":[995412],"related-opportunities":[],"related-publications":[],"related-videos":[],"related-posts":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event\/931014","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-event"}],"version-history":[{"count":47,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event\/931014\/revisions"}],"predecessor-version":[{"id":936504,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event\/931014\/revisions\/936504"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/932628"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=931014"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=931014"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=931014"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=931014"},{"taxonomy":"msr-video-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video-type?post=931014"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=931014"},{"taxonomy":"msr-program-audience","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-program-audience?post=931014"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=931014"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=931014"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}