{"id":601884,"date":"2019-08-05T17:47:32","date_gmt":"2019-08-06T00:47:32","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-blog-post&p=601884"},"modified":"2019-08-13T10:42:55","modified_gmt":"2019-08-13T17:42:55","slug":"trustworthy-ai-through-user-interface-design","status":"publish","type":"msr-blog-post","link":"https:\/\/www.microsoft.com\/en-us\/research\/articles\/trustworthy-ai-through-user-interface-design\/","title":{"rendered":"Trustworthy AI through user interface design"},"content":{"rendered":"

By Isaac Naveh-Benjamin, Software Engineer<\/a><\/p>\n

The rise of artificial intelligence has led to a growing problem of\u00a0<\/span>transparency<\/span><\/i>. It is increasingly difficult for the layperson<\/span>, or even for the developer of an AI system<\/span>, to understand how\u00a0<\/span>such<\/span>\u00a0systems work. Sometimes, this\u00a0<\/span>is\u00a0<\/span>due to the use of proprietary algorithms \u2013 but more often, it\u2019s because modern AI systems\u00a0<\/span>are<\/span> simply too complex <\/span>for humans to\u00a0<\/span>understand. Such systems are \u201cblack boxes\u201d whose functioning depends on millions of internal interdependent variables.<\/span>\u00a0<\/span><\/p>\n

As\u00a0<\/span>AI<\/span>\u00a0systems are deployed\u00a0<\/span>in the real world, questions of\u00a0<\/span>transparency and\u00a0<\/span>accountability<\/span>\u00a0<\/span>have become increasingly important<\/span>. With AI being used in applications like\u00a0<\/span>criminal sentencing<\/span> (opens in new tab)<\/span><\/a>\u00a0or\u00a0<\/span>loan approval<\/span> (opens in new tab)<\/span><\/a>, it\u2019s important\u00a0<\/span>to<\/span>\u00a0be able to answer questions like \u201c<\/span>Why did input A result in output B<\/span>?\u201d or \u201cWhat should I do differently to get a favorable outcome?\u201d<\/span>\u00a0<\/span>\u00a0<\/span><\/p>\n

When it comes to\u00a0<\/span>the\u00a0<\/span>medical\u00a0<\/span>domain<\/span>, the stakes are\u00a0<\/span>even higher<\/span>. Doctors<\/span>\u00a0<\/span>can\u2019t afford to trust the recommendations of an AI system whose workings\u00a0<\/span>they don\u2019t understand<\/span>.\u00a0<\/span>While they don’t<\/span>\u00a0need to understand the\u00a0<\/span>precise technical<\/span>\u00a0<\/span>details<\/span>\u00a0<\/span>of the algorithms, they do need<\/span>\u00a0to have an intuitive\u00a0<\/span>grasp<\/span>\u00a0of\u00a0<\/span>how the system works (<\/span>so that they<\/span>\u00a0can understand its limitations)<\/span>. They also\u00a0<\/span>need an \u201caudit trail\u201d that explains how a machine came to a specific conclusion.<\/span>\u00a0<\/span><\/p>\n

At EmpowerMD, a Microsoft incubation project, we\u2019re developing a<\/span>n\u00a0<\/span>Intelligent Scribe Service<\/span><\/a>\u00a0that\u00a0<\/span>listens to doctor-patient interactions and automatically generates a medical note.<\/span>\u00a0This note is then vetted for accuracy by the doctor, who can edit it as she sees fit. This design is\u00a0<\/span>known as<\/span>\u00a0human in the\u00a0<\/span><\/i>loop<\/span><\/i>, and<\/span>\u00a0is one of the main ways of making AI systems more transparent. By incorporating human feedback into the process, we can be more confident in the\u00a0<\/span>overall\u00a0<\/span>integrity of the system.\u00a0<\/span>(Note that it is also a legal imperative, since by law, medical notes must be reviewed for accuracy and signed by a doctor<\/span>).<\/span>\u00a0<\/span><\/p>\n

In designing the ISS\u2019s user interface, we added several additional features that help make it more transparent. Below, you can see a\u00a0<\/span>screenshot<\/span>\u00a0of the Intelligent Scribe, with\u00a0<\/span>the\u00a0<\/span>transcript of conversation on the right<\/span>-hand side<\/span>, and the\u00a0<\/span>machine-<\/span>generated note on the left.<\/span>\u00a0<\/span>Whenever<\/span>\u00a0the user clicks any part of the medical note, we give a visual indication of\u00a0<\/span>where in<\/span>\u00a0the conversation it came<\/span>\u00a0from by highlighting the corresponding parts of the transcript. This makes it easy for the user to determine the provenance of any part of the medical note.<\/span>\u00a0<\/span><\/p>\n

[Click to enlarge]<\/span><\/p>\n