Dissecting Racial Bias in an Algorithm that Guides Health Decisions for Millions

For millions of patients across the US, hospitals use commercial risk scores to target those needing extra help with complex health needs. We examine a widely used commercial algorithm for racial bias. Thanks to a unique dataset, we also study the algorithm’s construction, gaining a rare window into the mechanisms of bias. We find significant racial bias: at the same risk score, blacks are considerably sicker than whites. Removing bias would double the number of high-risk blacks auto-identified for extra help, from 17.7% to 46.5%. We isolate the problem to the algorithm’s objective function: it predicts costs, and since blacks incur lower costs than whites conditional on health, accurate cost predictions produce racially biased health predictions. We find suggestive evidence of a “problem formulation error”: as algorithmic prediction is in a nascent stage, convenient choices of proxy labels to predict (in this case, cost) can inadvertently produce biases at scale.

[SLIDES]

Speaker Details

Ziad Obermeyer is an Associate Professor (Acting) at UC Berkeley who works at the intersection of machine learning and medicine. His research seeks to understand and improve decision making in public policy and clinical medicine, and drive innovations in health research. Previously, he was an Assistant Professor at Harvard Medical School, where he received the Early Independence Award, the National Institutes of Health’s most prestigious award for exceptional junior scientists. He continues to practice emergency medicine in underserved parts of the US. Prior to his career in medicine, he worked as a consultant to pharmaceutical and global health clients at McKinsey & Co. in New Jersey, Geneva, and Tokyo. He is a graduate of Harvard College (magna cum laude) and Harvard Medical School (magna cum laude), and earned an M.Phil. from Cambridge.