Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming
Alex Chouldechova, A. Feder Cooper, Solon Barocas, Abhinav Palia, Dan Vann, Hanna Wallach
NeurIPS 2025 | October 2025
Hanna Wallach
Vice President and Distinguished Scientist
论文与出版物
-
-
-
Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
-
Taxonomizing Representational Harms using Speech Act Theory
Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alex Chouldechova, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
March 2025
-
Validating LLM-as-a-Judge Systems under Rating Indeterminacy
Luke Guerdan, Solon Barocas, Kenneth Holstein, Hanna Wallach, Zhiwei Steven Wu, Alex Chouldechova
NeurIPS 2025 | March 2025
-
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
ICML 2025 | January 2025
-
-
-
Machine Unlearning Doesn’t Do What You Think: Lessons for Generative AI Policy, Research, and Practice
A. F. Cooper, Christopher A. Choquette-Choo, Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alex Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, Ilia Shumailov, Eleni Triantafillou, Peter Kairouz, N. Mitchell, Percy Liang, Daniel E. Ho, Yejin Choi, Sanmi Koyejo, Fernando Delgado, James Grimmelmann, Vitaly Shmatikov, Christopher De Sa, Solon Barocas, A. Cyphert, Mark A. Lemley, danah boyd, Jennifer Wortman Vaughan, M. Brundage, David Bau, Seth Neel, Abigail Z. Jacobs, Andreas Terzis, Hanna Wallach, Nicolas Papernot, Katherine Lee
NeurIPS 2025 | December 2024
-
A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts
Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach
December 2024
-
Microsoft New Future of Work Report 2024
Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le
MSR-TR-2024-56 | December 2024
作者：Microsoft
-
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
November 2024
-
Dimensions of Generative AI Evaluation Design
Alex Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach
November 2024
-
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
November 2024
-
Understanding the Impacts of Language Technologies’ Performance Disparities on African American Language Speakers
Jay Cunningham, Su Lin Blodgett, Hal Daumé III, Christina Harrington, Hanna Wallach, Michael Madaio
Findings of the Association for Computational Linguistics: ACL 2024 | August 2024
-
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
-
Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists
Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan
Computer-Supported Cooperative Work | April 2024
-
-
-
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
ArXiv | October 2023, 第 abs/2310.17750 卷
-
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
October 2023
-
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach
ACL 2023 | July 2023
-
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
-
-
-
Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
2022 Conference on Computer Supported Cooperative Work | November 2022
-
Measuring Representational Harms in Image Captioning
Kristen Laird, Solon Barocas, Hanna Wallach, Angelina Wang
June 2022
-
REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
FAccT 2022 | May 2022
-
Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support
Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022
-
-
-
A Human-Centered Agenda for Intelligible Machine Learning
Jennifer Wortman Vaughan, Hanna Wallach
In Machines We Trust: Perspectives on Dependable AI | 作者：MIT Press | 2021
-
Datasheets for Datasets
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
Communications of the ACM | December 2021, 第 64 卷(12): pp. 86-92
-
Summarize with Caution: Comparing Global Feature Attributions
Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan
Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021
-
From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence
David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan
HCOMP 2021 | September 2021
-
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
-
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, 第 67 卷(7): pp. 30-32
https://doi.org/10.1145/3466612
-
Doubly Non-Central Beta Matrix Factorization for DNA Methylation Data
Aaron Schein, Anjali Nagulpally, Hanna Wallach, Patrick Flaherty
Conference on Uncertainty in Artificial Intelligence (UAI) 2021 | June 2021
-
Manipulating and Measuring Model Interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2021 | May 2021
-
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
AIES 2021 | May 2021
-
Bridging Human–Computer Interaction and Natural Language Processing
Su Lin Blodgett, Michael Madaio, Brendan O’Connor, Hanna Wallach, Qian Yang
EACL 2021 Workshop | April 2021
-
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
-
-
-
Assessing and mitigating unfairness in credit models with the Fairlearn toolkit
Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach
MSR-TR-2020-34 | September 2020
作者：Microsoft
-
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
-
Fairlearn: A toolkit for assessing and improving fairness in AI
Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker
MSR-TR-2020-32 | May 2020
作者：Microsoft
-
On the Relationships Between the Grammatical Genders of Inanimate Nouns and their Co-Occurring Adjectives and Verbs
Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damian Blasi, Hanna Wallach
May 2020
-
A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach
April 2020
-
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
CHI 2020 | April 2020
CHI 2020 Honorable Mention Award
-
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
-
-
-
Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer
Zach Harned, Hanna Wallach
Florida State University Law Review, Forthcoming | December 2019
-
Weight of Evidence as a Basis for Human-Oriented Explanations
David Alvarez-Melis, Hal Daumé III, Jennifer Wortman Vaughan, Hanna Wallach
Workshop on Human-Centric Machine Learning at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. | October 2019
-
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
-
Improving fairness in machine learning systems: What do industry practitioners need?
Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
-
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Yin, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2019 | May 2019
Honorable Mention Award at CHI 2019
-
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
Honorable Mention Award at CHI 2019
-
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
-
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Adam Tauman Kalai
FAT* ’19: Conference on Fairness, Accountability, and Transparency | January 2019
-
-
-
The Hyperedge Event Model
Bomin Kim, Aaron Schein, Bruce A. Desmarais, Hanna Wallach
July 2018
-
A Reductions Approach to Fair Classification
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, Hanna Wallach
FATML’17 | March 2018
-
-
-
Auditing Search Engines for Differential Satisfaction Across Demographics
Rishabh Mehrotra, Ashton Anderson, Fernando Diaz, Amit Sharma, Hanna Wallach, Emine Yilmaz
Proceedings of the 26th International Conference on World Wide Web (Industry Track) | April 2017
-
Transparency by Conformity: A Field Experiment Evaluating Openness in Local Governments
James ben-Aaron, Matthew Denny, Bruce Desmarais, Hanna Wallach
Public Administration Review | February 2017, 第 77 卷: pp. 68-77
-
-
-
Detecting and Characterizing Events
Allison June-Barlow Chaney, Hanna Wallach, Matthew Connelly, David M. Blei
EMNLP | November 2016
-
Flexible Models for Microclustering with Application to Entity Resolution
Glacomo Zanella, Brenda Betancourt, Hanna Wallach, Jeffrey Miller, Abbas Zaidi, Rebecca C. Steorts
30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain | October 2016
-
The Social Dynamics of Language Change in Online Networks
Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Fernando Diaz, Hanna Wallach, Jacob Eisenstein
International Conference on Social Informatics: SocInfo 2016 | October 2016
-
Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI
Sarah Bird, Solon Barocas, Kate Crawford, Hanna Wallach
Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), New York University | October 2016
-
Poisson–Gamma Dynamical Systems
Aaron Schein, Hanna Wallach, Mingyuan Zhou
Advances in Neural Information Processing Systems 29 (NIPS 2016) | May 2016
-
Bag of What? Simple Noun Phrase Extraction for Text Analysis
Abram Handler, Matthew Denny, Hanna Wallach, Brendan T. O'Connor
NLP+CSS@EMNLP | April 2016
-
Bayesian Poisson Tucker Decomposition for Learning the Structure of International Relations
Aaron Schein, Mingyuan Zhou, David M. Blei, Hanna Wallach
33rd International Conference on Machine Learning (ICML 2016) | April 2016
-
Computational Social Science: Toward a Collaborative Future
Computational Social Science: Discovery and Prediction | 作者：Cambridge University Press | 2016 | Computational Social Science: Discovery and Prediction edition
-
-
-
Bayesian Poisson Tensor Factorization for Inferring Multilateral Relations from Sparse Dyadic Event Counts
Aaron Schein, John Paisley, David M. Blei, Hanna Wallach
Proceedings of the Twenty-First ACM SIGKDD Conference on Knowledge Discovery and Data Mining | August 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015
-
The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation
F. Guo, C. Blundell, Hanna Wallach, K. Heller
Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics | May 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015; the Fifth Annual Text as Data Conference, 2014
-
-
-
Big Data, Machine Learning, and the Social Sciences: Fairness, Accountability, and Transparency
Medium | December 2014
-
Computational Social Science: Computer-Supported Cooperative Work in the Social Media Era
Scott Counts, Munmun De Choudhury, Jana Diesner, Eric Gilbert, Marta C. Gonzalez, Brian Keegan, Mor Naaman, Hanna Wallach
CSCW'14 | February 2014
-
Summarizing Topics: From Word Lists to Phrases
Lauren A. Hannah, Hanna Wallach
NIPS Workshop on Modern Machine Learning and Natural Language Processing | January 2014
-
-
-
Computational social science and social computing
Winter Mason, Jennifer Wortman Vaughan, Hanna Wallach
Mach Learn | November 2013, 第 95 卷: pp. 257-260
-
Efficient Nearest-Neighbor Search in the Probability Simplex
Kriste Krstovski, David A. Smith, Hanna Wallach, Andrew McGregor
2013 Conference on the Theory of Information Retrieval | September 2013
-
Inferring Multilateral Relations from Dynamic Pairwise Interactions
Aaron Schein, Juston Moore, Hanna Wallach
NIPS 2013 Workshop on Frontiers of Network Analysis | September 2013
-
Seven Things I Wish I’d Known About Derby
DerbyLife | March 2013
-
The Benefits of Double-Blind Review
ICML 2013 | January 2013
-
-
-
Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming
Alex Chouldechova, A. Feder Cooper, Solon Barocas, Abhinav Palia, Dan Vann, Hanna Wallach
NeurIPS 2025 | October 2025
-
Taxonomizing Representational Harms using Speech Act Theory
Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alex Chouldechova, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
March 2025
-
Validating LLM-as-a-Judge Systems under Rating Indeterminacy
Luke Guerdan, Solon Barocas, Kenneth Holstein, Hanna Wallach, Zhiwei Steven Wu, Alex Chouldechova
NeurIPS 2025 | March 2025
-
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
ICML 2025 | January 2025
-
Machine Unlearning Doesn’t Do What You Think: Lessons for Generative AI Policy, Research, and Practice
A. F. Cooper, Christopher A. Choquette-Choo, Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alex Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, Ilia Shumailov, Eleni Triantafillou, Peter Kairouz, N. Mitchell, Percy Liang, Daniel E. Ho, Yejin Choi, Sanmi Koyejo, Fernando Delgado, James Grimmelmann, Vitaly Shmatikov, Christopher De Sa, Solon Barocas, A. Cyphert, Mark A. Lemley, danah boyd, Jennifer Wortman Vaughan, M. Brundage, David Bau, Seth Neel, Abigail Z. Jacobs, Andreas Terzis, Hanna Wallach, Nicolas Papernot, Katherine Lee
NeurIPS 2025 | December 2024
-
Microsoft New Future of Work Report 2024
Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le
MSR-TR-2024-56 | December 2024
作者：Microsoft
-
A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts
Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach
December 2024
-
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
November 2024
-
Dimensions of Generative AI Evaluation Design
Alex Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach
November 2024
-
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
November 2024
-
Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists
Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan
Computer-Supported Cooperative Work | April 2024
-
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
ArXiv | October 2023, 第 abs/2310.17750 卷
-
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
October 2023
-
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach
ACL 2023 | July 2023
-
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
-
Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
2022 Conference on Computer Supported Cooperative Work | November 2022
-
REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
FAccT 2022 | May 2022
-
Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support
Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022
-
A Human-Centered Agenda for Intelligible Machine Learning
Jennifer Wortman Vaughan, Hanna Wallach
In Machines We Trust: Perspectives on Dependable AI | 作者：MIT Press | 2021
-
Summarize with Caution: Comparing Global Feature Attributions
Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan
Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021
-
Datasheets for Datasets
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
Communications of the ACM | December 2021, 第 64 卷(12): pp. 86-92
-
From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence
David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan
HCOMP 2021 | September 2021
-
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, 第 67 卷(7): pp. 30-32
https://doi.org/10.1145/3466612
-
Doubly Non-Central Beta Matrix Factorization for DNA Methylation Data
Aaron Schein, Anjali Nagulpally, Hanna Wallach, Patrick Flaherty
Conference on Uncertainty in Artificial Intelligence (UAI) 2021 | June 2021
-
Manipulating and Measuring Model Interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2021 | May 2021
-
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
AIES 2021 | May 2021
-
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
-
Assessing and mitigating unfairness in credit models with the Fairlearn toolkit
Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach
MSR-TR-2020-34 | September 2020
作者：Microsoft
-
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
-
Fairlearn: A toolkit for assessing and improving fairness in AI
Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker
MSR-TR-2020-32 | May 2020
作者：Microsoft
-
A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach
April 2020
-
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
CHI 2020 | April 2020
CHI 2020 Honorable Mention Award
-
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
-
Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer
Zach Harned, Hanna Wallach
Florida State University Law Review, Forthcoming | December 2019
-
Weight of Evidence as a Basis for Human-Oriented Explanations
David Alvarez-Melis, Hal Daumé III, Jennifer Wortman Vaughan, Hanna Wallach
Workshop on Human-Centric Machine Learning at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. | October 2019
-
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
-
Improving fairness in machine learning systems: What do industry practitioners need?
Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
-
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
Honorable Mention Award at CHI 2019
-
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
-
A Reductions Approach to Fair Classification
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, Hanna Wallach
FATML’17 | March 2018
-
Flexible Models for Microclustering with Application to Entity Resolution
Glacomo Zanella, Brenda Betancourt, Hanna Wallach, Jeffrey Miller, Abbas Zaidi, Rebecca C. Steorts
30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain | October 2016
-
The Social Dynamics of Language Change in Online Networks
Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Fernando Diaz, Hanna Wallach, Jacob Eisenstein
International Conference on Social Informatics: SocInfo 2016 | October 2016
-
Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI
Sarah Bird, Solon Barocas, Kate Crawford, Hanna Wallach
Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), New York University | October 2016
-
Poisson–Gamma Dynamical Systems
Aaron Schein, Hanna Wallach, Mingyuan Zhou
Advances in Neural Information Processing Systems 29 (NIPS 2016) | May 2016
-
Summarizing Topics: From Word Lists to Phrases
Lauren A. Hannah, Hanna Wallach
NIPS Workshop on Modern Machine Learning and Natural Language Processing | January 2014
-
Efficient Nearest-Neighbor Search in the Probability Simplex
Kriste Krstovski, David A. Smith, Hanna Wallach, Andrew McGregor
2013 Conference on the Theory of Information Retrieval | September 2013
-
-
-
Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming
Alex Chouldechova, A. Feder Cooper, Solon Barocas, Abhinav Palia, Dan Vann, Hanna Wallach
NeurIPS 2025 | October 2025
-
Microsoft New Future of Work Report 2024
Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le
MSR-TR-2024-56 | December 2024
作者：Microsoft
-
The Hyperedge Event Model
Bomin Kim, Aaron Schein, Bruce A. Desmarais, Hanna Wallach
July 2018
-
-
-
Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming
Alex Chouldechova, A. Feder Cooper, Solon Barocas, Abhinav Palia, Dan Vann, Hanna Wallach
NeurIPS 2025 | October 2025
-
Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI
Sarah Bird, Solon Barocas, Kate Crawford, Hanna Wallach
Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), New York University | October 2016
-
-
-
Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
-
Understanding the Impacts of Language Technologies’ Performance Disparities on African American Language Speakers
Jay Cunningham, Su Lin Blodgett, Hal Daumé III, Christina Harrington, Hanna Wallach, Michael Madaio
Findings of the Association for Computational Linguistics: ACL 2024 | August 2024
-
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
-
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
-
Bridging Human–Computer Interaction and Natural Language Processing
Su Lin Blodgett, Michael Madaio, Brendan O’Connor, Hanna Wallach, Qian Yang
EACL 2021 Workshop | April 2021
-
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
-
On the Relationships Between the Grammatical Genders of Inanimate Nouns and their Co-Occurring Adjectives and Verbs
Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damian Blasi, Hanna Wallach
May 2020
-
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
-
The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation
F. Guo, C. Blundell, Hanna Wallach, K. Heller
Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics | May 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015; the Fifth Annual Text as Data Conference, 2014
-
-
-
Microsoft New Future of Work Report 2024
Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le
MSR-TR-2024-56 | December 2024
作者：Microsoft
-
Understanding the Impacts of Language Technologies’ Performance Disparities on African American Language Speakers
Jay Cunningham, Su Lin Blodgett, Hal Daumé III, Christina Harrington, Hanna Wallach, Michael Madaio
Findings of the Association for Computational Linguistics: ACL 2024 | August 2024
-
Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
2022 Conference on Computer Supported Cooperative Work | November 2022
-
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, 第 67 卷(7): pp. 30-32
https://doi.org/10.1145/3466612
-
Bridging Human–Computer Interaction and Natural Language Processing
Su Lin Blodgett, Michael Madaio, Brendan O’Connor, Hanna Wallach, Qian Yang
EACL 2021 Workshop | April 2021
-
On the Relationships Between the Grammatical Genders of Inanimate Nouns and their Co-Occurring Adjectives and Verbs
Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damian Blasi, Hanna Wallach
May 2020
-
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
CHI 2020 | April 2020
CHI 2020 Honorable Mention Award
-
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
-
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
-
Improving fairness in machine learning systems: What do industry practitioners need?
Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
-
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Yin, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2019 | May 2019
Honorable Mention Award at CHI 2019
-
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
Honorable Mention Award at CHI 2019
-
-
-
Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
-
Taxonomizing Representational Harms using Speech Act Theory
Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alex Chouldechova, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
March 2025
-
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
ICML 2025 | January 2025
-
Microsoft New Future of Work Report 2024
Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le
MSR-TR-2024-56 | December 2024
作者：Microsoft
-
A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts
Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach
December 2024
-
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
November 2024
-
Dimensions of Generative AI Evaluation Design
Alex Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach
November 2024
-
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
November 2024
-
Understanding the Impacts of Language Technologies’ Performance Disparities on African American Language Speakers
Jay Cunningham, Su Lin Blodgett, Hal Daumé III, Christina Harrington, Hanna Wallach, Michael Madaio
Findings of the Association for Computational Linguistics: ACL 2024 | August 2024
-
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
ArXiv | October 2023, 第 abs/2310.17750 卷
-
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, 第 67 卷(7): pp. 30-32
https://doi.org/10.1145/3466612
-
Manipulating and Measuring Model Interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2021 | May 2021
-
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
-
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
-
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
-
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Adam Tauman Kalai
FAT* ’19: Conference on Fairness, Accountability, and Transparency | January 2019
-
Transparency by Conformity: A Field Experiment Evaluating Openness in Local Governments
James ben-Aaron, Matthew Denny, Bruce Desmarais, Hanna Wallach
Public Administration Review | February 2017, 第 77 卷: pp. 68-77
-
Detecting and Characterizing Events
Allison June-Barlow Chaney, Hanna Wallach, Matthew Connelly, David M. Blei
EMNLP | November 2016
-
Bayesian Poisson Tucker Decomposition for Learning the Structure of International Relations
Aaron Schein, Mingyuan Zhou, David M. Blei, Hanna Wallach
33rd International Conference on Machine Learning (ICML 2016) | April 2016
-
Bag of What? Simple Noun Phrase Extraction for Text Analysis
Abram Handler, Matthew Denny, Hanna Wallach, Brendan T. O'Connor
NLP+CSS@EMNLP | April 2016
-
Computational Social Science: Toward a Collaborative Future
Computational Social Science: Discovery and Prediction | 作者：Cambridge University Press | 2016 | Computational Social Science: Discovery and Prediction edition
-
Bayesian Poisson Tensor Factorization for Inferring Multilateral Relations from Sparse Dyadic Event Counts
Aaron Schein, John Paisley, David M. Blei, Hanna Wallach
Proceedings of the Twenty-First ACM SIGKDD Conference on Knowledge Discovery and Data Mining | August 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015
-
The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation
F. Guo, C. Blundell, Hanna Wallach, K. Heller
Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics | May 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015; the Fifth Annual Text as Data Conference, 2014
-
Big Data, Machine Learning, and the Social Sciences: Fairness, Accountability, and Transparency
Medium | December 2014
-
Computational Social Science: Computer-Supported Cooperative Work in the Social Media Era
Scott Counts, Munmun De Choudhury, Jana Diesner, Eric Gilbert, Marta C. Gonzalez, Brian Keegan, Mor Naaman, Hanna Wallach
CSCW'14 | February 2014
-
Computational social science and social computing
Winter Mason, Jennifer Wortman Vaughan, Hanna Wallach
Mach Learn | November 2013, 第 95 卷: pp. 257-260
-
Inferring Multilateral Relations from Dynamic Pairwise Interactions
Aaron Schein, Juston Moore, Hanna Wallach
NIPS 2013 Workshop on Frontiers of Network Analysis | September 2013
-
Seven Things I Wish I’d Known About Derby
DerbyLife | March 2013
-
The Benefits of Double-Blind Review
ICML 2013 | January 2013
-
-
-
Microsoft New Future of Work Report 2024
Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le
MSR-TR-2024-56 | December 2024
作者：Microsoft
-
-
-
Microsoft New Future of Work Report 2024
Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le
MSR-TR-2024-56 | December 2024
作者：Microsoft
-
-
-
Measuring Representational Harms in Image Captioning
Kristen Laird, Solon Barocas, Hanna Wallach, Angelina Wang
June 2022
-
-
-
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
-
-
-
Auditing Search Engines for Differential Satisfaction Across Demographics
Rishabh Mehrotra, Ashton Anderson, Fernando Diaz, Amit Sharma, Hanna Wallach, Emine Yilmaz
Proceedings of the 26th International Conference on World Wide Web (Industry Track) | April 2017
-
-
-
Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming
Alex Chouldechova, A. Feder Cooper, Solon Barocas, Abhinav Palia, Dan Vann, Hanna Wallach
NeurIPS 2025 | October 2025
-
Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
-
Validating LLM-as-a-Judge Systems under Rating Indeterminacy
Luke Guerdan, Solon Barocas, Kenneth Holstein, Hanna Wallach, Zhiwei Steven Wu, Alex Chouldechova
NeurIPS 2025 | March 2025
-
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
ICML 2025 | January 2025
-
Machine Unlearning Doesn’t Do What You Think: Lessons for Generative AI Policy, Research, and Practice
A. F. Cooper, Christopher A. Choquette-Choo, Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alex Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, Ilia Shumailov, Eleni Triantafillou, Peter Kairouz, N. Mitchell, Percy Liang, Daniel E. Ho, Yejin Choi, Sanmi Koyejo, Fernando Delgado, James Grimmelmann, Vitaly Shmatikov, Christopher De Sa, Solon Barocas, A. Cyphert, Mark A. Lemley, danah boyd, Jennifer Wortman Vaughan, M. Brundage, David Bau, Seth Neel, Abigail Z. Jacobs, Andreas Terzis, Hanna Wallach, Nicolas Papernot, Katherine Lee
NeurIPS 2025 | December 2024
-
Understanding the Impacts of Language Technologies’ Performance Disparities on African American Language Speakers
Jay Cunningham, Su Lin Blodgett, Hal Daumé III, Christina Harrington, Hanna Wallach, Michael Madaio
Findings of the Association for Computational Linguistics: ACL 2024 | August 2024
-
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
-
Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists
Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan
Computer-Supported Cooperative Work | April 2024
-
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach
ACL 2023 | July 2023
-
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
-
Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
2022 Conference on Computer Supported Cooperative Work | November 2022
-
REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
FAccT 2022 | May 2022
-
Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support
Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022
-
From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence
David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan
HCOMP 2021 | September 2021
-
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
-
Doubly Non-Central Beta Matrix Factorization for DNA Methylation Data
Aaron Schein, Anjali Nagulpally, Hanna Wallach, Patrick Flaherty
Conference on Uncertainty in Artificial Intelligence (UAI) 2021 | June 2021
-
Manipulating and Measuring Model Interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2021 | May 2021
-
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
AIES 2021 | May 2021
-
Bridging Human–Computer Interaction and Natural Language Processing
Su Lin Blodgett, Michael Madaio, Brendan O’Connor, Hanna Wallach, Qian Yang
EACL 2021 Workshop | April 2021
-
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
-
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
-
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
CHI 2020 | April 2020
CHI 2020 Honorable Mention Award
-
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
-
Weight of Evidence as a Basis for Human-Oriented Explanations
David Alvarez-Melis, Hal Daumé III, Jennifer Wortman Vaughan, Hanna Wallach
Workshop on Human-Centric Machine Learning at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. | October 2019
-
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
-
Improving fairness in machine learning systems: What do industry practitioners need?
Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
-
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Yin, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2019 | May 2019
Honorable Mention Award at CHI 2019
-
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
Honorable Mention Award at CHI 2019
-
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
-
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Adam Tauman Kalai
FAT* ’19: Conference on Fairness, Accountability, and Transparency | January 2019
-
A Reductions Approach to Fair Classification
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, Hanna Wallach
FATML’17 | March 2018
-
Auditing Search Engines for Differential Satisfaction Across Demographics
Rishabh Mehrotra, Ashton Anderson, Fernando Diaz, Amit Sharma, Hanna Wallach, Emine Yilmaz
Proceedings of the 26th International Conference on World Wide Web (Industry Track) | April 2017
-
Detecting and Characterizing Events
Allison June-Barlow Chaney, Hanna Wallach, Matthew Connelly, David M. Blei
EMNLP | November 2016
-
Flexible Models for Microclustering with Application to Entity Resolution
Glacomo Zanella, Brenda Betancourt, Hanna Wallach, Jeffrey Miller, Abbas Zaidi, Rebecca C. Steorts
30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain | October 2016
-
The Social Dynamics of Language Change in Online Networks
Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Fernando Diaz, Hanna Wallach, Jacob Eisenstein
International Conference on Social Informatics: SocInfo 2016 | October 2016
-
Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI
Sarah Bird, Solon Barocas, Kate Crawford, Hanna Wallach
Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), New York University | October 2016
-
Poisson–Gamma Dynamical Systems
Aaron Schein, Hanna Wallach, Mingyuan Zhou
Advances in Neural Information Processing Systems 29 (NIPS 2016) | May 2016
-
Bayesian Poisson Tucker Decomposition for Learning the Structure of International Relations
Aaron Schein, Mingyuan Zhou, David M. Blei, Hanna Wallach
33rd International Conference on Machine Learning (ICML 2016) | April 2016
-
Bag of What? Simple Noun Phrase Extraction for Text Analysis
Abram Handler, Matthew Denny, Hanna Wallach, Brendan T. O'Connor
NLP+CSS@EMNLP | April 2016
-
Bayesian Poisson Tensor Factorization for Inferring Multilateral Relations from Sparse Dyadic Event Counts
Aaron Schein, John Paisley, David M. Blei, Hanna Wallach
Proceedings of the Twenty-First ACM SIGKDD Conference on Knowledge Discovery and Data Mining | August 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015
-
The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation
F. Guo, C. Blundell, Hanna Wallach, K. Heller
Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics | May 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015; the Fifth Annual Text as Data Conference, 2014
-
Computational Social Science: Computer-Supported Cooperative Work in the Social Media Era
Scott Counts, Munmun De Choudhury, Jana Diesner, Eric Gilbert, Marta C. Gonzalez, Brian Keegan, Mor Naaman, Hanna Wallach
CSCW'14 | February 2014
-
Summarizing Topics: From Word Lists to Phrases
Lauren A. Hannah, Hanna Wallach
NIPS Workshop on Modern Machine Learning and Natural Language Processing | January 2014
-
Efficient Nearest-Neighbor Search in the Probability Simplex
Kriste Krstovski, David A. Smith, Hanna Wallach, Andrew McGregor
2013 Conference on the Theory of Information Retrieval | September 2013
-
Inferring Multilateral Relations from Dynamic Pairwise Interactions
Aaron Schein, Juston Moore, Hanna Wallach
NIPS 2013 Workshop on Frontiers of Network Analysis | September 2013
-
The Benefits of Double-Blind Review
ICML 2013 | January 2013
-
-
-
Taxonomizing Representational Harms using Speech Act Theory
Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alex Chouldechova, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
March 2025
-
A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts
Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach
December 2024
-
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
November 2024
-
Dimensions of Generative AI Evaluation Design
Alex Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach
November 2024
-
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
November 2024
-
On the Relationships Between the Grammatical Genders of Inanimate Nouns and their Co-Occurring Adjectives and Verbs
Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damian Blasi, Hanna Wallach
May 2020
-
A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach
April 2020
-
The Hyperedge Event Model
Bomin Kim, Aaron Schein, Bruce A. Desmarais, Hanna Wallach
July 2018
-
-
-
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
ArXiv | October 2023, 第 abs/2310.17750 卷
-
Summarize with Caution: Comparing Global Feature Attributions
Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan
Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021
-
Datasheets for Datasets
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
Communications of the ACM | December 2021, 第 64 卷(12): pp. 86-92
-
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, 第 67 卷(7): pp. 30-32
https://doi.org/10.1145/3466612
-
Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer
Zach Harned, Hanna Wallach
Florida State University Law Review, Forthcoming | December 2019
-
Transparency by Conformity: A Field Experiment Evaluating Openness in Local Governments
James ben-Aaron, Matthew Denny, Bruce Desmarais, Hanna Wallach
Public Administration Review | February 2017, 第 77 卷: pp. 68-77
-
Big Data, Machine Learning, and the Social Sciences: Fairness, Accountability, and Transparency
Medium | December 2014
-
Computational social science and social computing
Winter Mason, Jennifer Wortman Vaughan, Hanna Wallach
Mach Learn | November 2013, 第 95 卷: pp. 257-260
-
Seven Things I Wish I’d Known About Derby
DerbyLife | March 2013
-
-
-
Microsoft New Future of Work Report 2024
Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le
MSR-TR-2024-56 | December 2024
作者：Microsoft
-
Assessing and mitigating unfairness in credit models with the Fairlearn toolkit
Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach
MSR-TR-2020-34 | September 2020
作者：Microsoft
-
Fairlearn: A toolkit for assessing and improving fairness in AI
Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker
MSR-TR-2020-32 | May 2020
作者：Microsoft
-
-
-
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
October 2023
-
Measuring Representational Harms in Image Captioning
Kristen Laird, Solon Barocas, Hanna Wallach, Angelina Wang
June 2022
-
-
-
A Human-Centered Agenda for Intelligible Machine Learning
Jennifer Wortman Vaughan, Hanna Wallach
In Machines We Trust: Perspectives on Dependable AI | 作者：MIT Press | 2021
-
Computational Social Science: Toward a Collaborative Future
Computational Social Science: Discovery and Prediction | 作者：Cambridge University Press | 2016 | Computational Social Science: Discovery and Prediction edition
-