Do Machine Teachers Dream of Algorithms?
Gonzalo Ramos, Felicia Ng, Nicole Sultanum, Chris Meek, Jina Suh, Soroush Ghorashi
December 2019
Machine Teaching Group
Publications
Publications
Publications by Year
-
Using Expert Patterns in Assisted Interactive Machine Learning: A Study in Machine Teaching
Emily Wall, Soroush Ghorashi, Gonzalo Ramos
Interact 2019 | September 2019
AnchorViz: Facilitating Semantic Data Exploration and Concept Discovery for Interactive Machine Learning
Jina Suh, Soroush Ghorashi, Gonzalo Ramos, Nan-Chen Chen, Steven Drucker, Johan Verwey, Patrice Simard
ACM Transactions on Interactive Intelligent Systems (TiiS) | August 2019, Vol 10(1)
-
Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Qian Yang, Jina Suh, Nan-Chen Chen, Gonzalo Ramos
Proceeding of the 2018 Designing of Interactive Systems Conference | June 2018
AnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data Exploration
Nan-Chen Chen, Jina Suh, Johan Verwey, Gonzalo Ramos, Steven Drucker, Patrice Simard
IUI ’18 23rd International Conference on Intelligent User Interfaces | March 2018
ISBN: 978-1-4503-4945-1
-
Machine Teaching: A New Paradigm for Building Machine Learning Systems
Patrice Simard, Saleema Amershi, Max Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Chris Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, John Wernsing
MSR-TR-2017-26 | July 2017
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
Joseph Chee Chang, Saleema Amershi, Ece Kamar
Proceedings of the Conference on Human Factors in Computing Systems (CHI 2017) | May 2017
-
A Characterization of Prediction Errors
Chris Meek
MSR-TR-2016-1105 | November 2016
Analysis of a Design Pattern for Teaching with Features and Labels
Chris Meek, Patrice Simard, Xiaojin Zhu
MSR-TR-2016-1104 | November 2016
Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, Jason Williams
IEEE TVCG, VAST | August 2016
Interactive Semantic Featuring for Text Classification
Patrice Simard, Max Chickering, Jina Suh
Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016) | June 2016
The Label Complexity of Mixed-Initiative Classiï¬er Training
Jina Suh, Xiaojin Zhu, Saleema Amershi
Proceedings of the 33rd International Conference on Machine Learning | June 2016
-
FeatureInsight: Visual Support for Error-Driven Feature Ideation in Text Classification
Michael Brooks, Saleema Amershi, Bongshin Lee, Steven Drucker, Ashish Kapoor, Patrice Simard
IEEE VAST | October 2015
Fast and Easy Language Understanding For Dialog Systems With Microsoft Language Understanding Intelligent Service (LUIS)
Jason Williams, Eslam Kamal, Mikhtar Ashour, Hani Amr, Jessica Miller, Geoffrey Zweig
Proceedings of 2015 SIGDIAL Conference, Prague | September 2015
Demonstration at SIGDIAL 2015.
ModelTracker: Redesigning Performance Analysis Tools for Machine Learning
Saleema Amershi, Max Chickering, Steven Drucker, Bongshin Lee, Patrice Simard, Jina Suh
Proceedings of the Conference on Human Factors in Computing Systems (CHI 2015) | April 2015
Rapidly Scaling Dialog Systems with Interactive Learning
Jason Williams, Nobal B. Niraula, Pradeep Dasigi, Aparna Lakshmiratan, Carlos Garcia Jurado Suarez, Mouni Reddy, Geoffrey Zweig
January 2015
-
Power to the People: The Role of Humans in Interactive Machine Learning
Saleema Amershi, Maya Cakmak, W. Bradley Knox, Todd Kulesza
AI Magazine | December 2014
ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
Patrice Simard, Max Chickering, Apama Lakshmiratan, Denis Charles, Leon Bottou, Carlos Garcia Jurado Suarez, David Grangier, Saleema Amershi, Johan Verwey, Jina Suh
MSR-TR-2014-128 | September 2014
Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy
Denny Zhou, Qiang Liu, John Platt, Chris Meek
Proceedings of the 31st International Conference on Machine Learning (ICML) | June 2014
Proceedings of the 31st International Conference on Machine Learning (ICML)
Structured Labeling for Facilitating Concept Evolution in Machine Learning
Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, Denis Charles
Proceedings of the Conference on Human Factors in Computing Systems (CHI 2014) | May 2014
Best Paper Award
-
Learning from Our Mistakes: Towards a Correctable Learning Algorithm
Karthik Raman, Krysta M. Svore, Ran Gilad-Bachrach, Chris J.C. Burges
21st International Conference on Information and Knowledge Management (CIKM) | October 2012
21st International Conference on Information and Knowledge Management (CIKM)
Publications by Research Area
-
Do Machine Teachers Dream of Algorithms?
Gonzalo Ramos, Felicia Ng, Nicole Sultanum, Chris Meek, Jina Suh, Soroush Ghorashi
December 2019
Using Expert Patterns in Assisted Interactive Machine Learning: A Study in Machine Teaching
Emily Wall, Soroush Ghorashi, Gonzalo Ramos
Interact 2019 | September 2019
AnchorViz: Facilitating Semantic Data Exploration and Concept Discovery for Interactive Machine Learning
Jina Suh, Soroush Ghorashi, Gonzalo Ramos, Nan-Chen Chen, Steven Drucker, Johan Verwey, Patrice Simard
ACM Transactions on Interactive Intelligent Systems (TiiS) | August 2019, Vol 10(1)
Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Qian Yang, Jina Suh, Nan-Chen Chen, Gonzalo Ramos
Proceeding of the 2018 Designing of Interactive Systems Conference | June 2018
AnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data Exploration
Nan-Chen Chen, Jina Suh, Johan Verwey, Gonzalo Ramos, Steven Drucker, Patrice Simard
IUI ’18 23rd International Conference on Intelligent User Interfaces | March 2018
ISBN: 978-1-4503-4945-1
Machine Teaching: A New Paradigm for Building Machine Learning Systems
Patrice Simard, Saleema Amershi, Max Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Chris Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, John Wernsing
MSR-TR-2017-26 | July 2017
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
Joseph Chee Chang, Saleema Amershi, Ece Kamar
Proceedings of the Conference on Human Factors in Computing Systems (CHI 2017) | May 2017
A Characterization of Prediction Errors
Chris Meek
MSR-TR-2016-1105 | November 2016
Analysis of a Design Pattern for Teaching with Features and Labels
Chris Meek, Patrice Simard, Xiaojin Zhu
MSR-TR-2016-1104 | November 2016
Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, Jason Williams
IEEE TVCG, VAST | August 2016
Interactive Semantic Featuring for Text Classification
Patrice Simard, Max Chickering, Jina Suh
Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016) | June 2016
The Label Complexity of Mixed-Initiative Classiï¬er Training
Jina Suh, Xiaojin Zhu, Saleema Amershi
Proceedings of the 33rd International Conference on Machine Learning | June 2016
FeatureInsight: Visual Support for Error-Driven Feature Ideation in Text Classification
Michael Brooks, Saleema Amershi, Bongshin Lee, Steven Drucker, Ashish Kapoor, Patrice Simard
IEEE VAST | October 2015
ModelTracker: Redesigning Performance Analysis Tools for Machine Learning
Saleema Amershi, Max Chickering, Steven Drucker, Bongshin Lee, Patrice Simard, Jina Suh
Proceedings of the Conference on Human Factors in Computing Systems (CHI 2015) | April 2015
Rapidly Scaling Dialog Systems with Interactive Learning
Jason Williams, Nobal B. Niraula, Pradeep Dasigi, Aparna Lakshmiratan, Carlos Garcia Jurado Suarez, Mouni Reddy, Geoffrey Zweig
January 2015
Power to the People: The Role of Humans in Interactive Machine Learning
Saleema Amershi, Maya Cakmak, W. Bradley Knox, Todd Kulesza
AI Magazine | December 2014
ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
Patrice Simard, Max Chickering, Apama Lakshmiratan, Denis Charles, Leon Bottou, Carlos Garcia Jurado Suarez, David Grangier, Saleema Amershi, Johan Verwey, Jina Suh
MSR-TR-2014-128 | September 2014
Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy
Denny Zhou, Qiang Liu, John Platt, Chris Meek
Proceedings of the 31st International Conference on Machine Learning (ICML) | June 2014
Proceedings of the 31st International Conference on Machine Learning (ICML)
Structured Labeling for Facilitating Concept Evolution in Machine Learning
Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, Denis Charles
Proceedings of the Conference on Human Factors in Computing Systems (CHI 2014) | May 2014
Best Paper Award
Learning from Our Mistakes: Towards a Correctable Learning Algorithm
Karthik Raman, Krysta M. Svore, Ran Gilad-Bachrach, Chris J.C. Burges
21st International Conference on Information and Knowledge Management (CIKM) | October 2012
21st International Conference on Information and Knowledge Management (CIKM)
-
Do Machine Teachers Dream of Algorithms?
Gonzalo Ramos, Felicia Ng, Nicole Sultanum, Chris Meek, Jina Suh, Soroush Ghorashi
December 2019
Using Expert Patterns in Assisted Interactive Machine Learning: A Study in Machine Teaching
Emily Wall, Soroush Ghorashi, Gonzalo Ramos
Interact 2019 | September 2019
AnchorViz: Facilitating Semantic Data Exploration and Concept Discovery for Interactive Machine Learning
Jina Suh, Soroush Ghorashi, Gonzalo Ramos, Nan-Chen Chen, Steven Drucker, Johan Verwey, Patrice Simard
ACM Transactions on Interactive Intelligent Systems (TiiS) | August 2019, Vol 10(1)
Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Qian Yang, Jina Suh, Nan-Chen Chen, Gonzalo Ramos
Proceeding of the 2018 Designing of Interactive Systems Conference | June 2018
AnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data Exploration
Nan-Chen Chen, Jina Suh, Johan Verwey, Gonzalo Ramos, Steven Drucker, Patrice Simard
IUI ’18 23rd International Conference on Intelligent User Interfaces | March 2018
ISBN: 978-1-4503-4945-1
Machine Teaching: A New Paradigm for Building Machine Learning Systems
Patrice Simard, Saleema Amershi, Max Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Chris Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, John Wernsing
MSR-TR-2017-26 | July 2017
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
Joseph Chee Chang, Saleema Amershi, Ece Kamar
Proceedings of the Conference on Human Factors in Computing Systems (CHI 2017) | May 2017
The Label Complexity of Mixed-Initiative Classiï¬er Training
Jina Suh, Xiaojin Zhu, Saleema Amershi
Proceedings of the 33rd International Conference on Machine Learning | June 2016
ModelTracker: Redesigning Performance Analysis Tools for Machine Learning
Saleema Amershi, Max Chickering, Steven Drucker, Bongshin Lee, Patrice Simard, Jina Suh
Proceedings of the Conference on Human Factors in Computing Systems (CHI 2015) | April 2015
Rapidly Scaling Dialog Systems with Interactive Learning
Jason Williams, Nobal B. Niraula, Pradeep Dasigi, Aparna Lakshmiratan, Carlos Garcia Jurado Suarez, Mouni Reddy, Geoffrey Zweig
January 2015
Power to the People: The Role of Humans in Interactive Machine Learning
Saleema Amershi, Maya Cakmak, W. Bradley Knox, Todd Kulesza
AI Magazine | December 2014
ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
Patrice Simard, Max Chickering, Apama Lakshmiratan, Denis Charles, Leon Bottou, Carlos Garcia Jurado Suarez, David Grangier, Saleema Amershi, Johan Verwey, Jina Suh
MSR-TR-2014-128 | September 2014
Structured Labeling for Facilitating Concept Evolution in Machine Learning
Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, Denis Charles
Proceedings of the Conference on Human Factors in Computing Systems (CHI 2014) | May 2014
Best Paper Award
-
AnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data Exploration
Nan-Chen Chen, Jina Suh, Johan Verwey, Gonzalo Ramos, Steven Drucker, Patrice Simard
IUI ’18 23rd International Conference on Intelligent User Interfaces | March 2018
ISBN: 978-1-4503-4945-1
Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, Jason Williams
IEEE TVCG, VAST | August 2016
FeatureInsight: Visual Support for Error-Driven Feature Ideation in Text Classification
Michael Brooks, Saleema Amershi, Bongshin Lee, Steven Drucker, Ashish Kapoor, Patrice Simard
IEEE VAST | October 2015
-
Fast and Easy Language Understanding For Dialog Systems With Microsoft Language Understanding Intelligent Service (LUIS)
Jason Williams, Eslam Kamal, Mikhtar Ashour, Hani Amr, Jessica Miller, Geoffrey Zweig
Proceedings of 2015 SIGDIAL Conference, Prague | September 2015
Demonstration at SIGDIAL 2015.
-
Fast and Easy Language Understanding For Dialog Systems With Microsoft Language Understanding Intelligent Service (LUIS)
Jason Williams, Eslam Kamal, Mikhtar Ashour, Hani Amr, Jessica Miller, Geoffrey Zweig
Proceedings of 2015 SIGDIAL Conference, Prague | September 2015
Demonstration at SIGDIAL 2015.
-
Learning from Our Mistakes: Towards a Correctable Learning Algorithm
Karthik Raman, Krysta M. Svore, Ran Gilad-Bachrach, Chris J.C. Burges
21st International Conference on Information and Knowledge Management (CIKM) | October 2012
21st International Conference on Information and Knowledge Management (CIKM)
Publications by Type
-
AnchorViz: Facilitating Semantic Data Exploration and Concept Discovery for Interactive Machine Learning
Jina Suh, Soroush Ghorashi, Gonzalo Ramos, Nan-Chen Chen, Steven Drucker, Johan Verwey, Patrice Simard
ACM Transactions on Interactive Intelligent Systems (TiiS) | August 2019, Vol 10(1)
Power to the People: The Role of Humans in Interactive Machine Learning
Saleema Amershi, Maya Cakmak, W. Bradley Knox, Todd Kulesza
AI Magazine | December 2014
-
Using Expert Patterns in Assisted Interactive Machine Learning: A Study in Machine Teaching
Emily Wall, Soroush Ghorashi, Gonzalo Ramos
Interact 2019 | September 2019
Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Qian Yang, Jina Suh, Nan-Chen Chen, Gonzalo Ramos
Proceeding of the 2018 Designing of Interactive Systems Conference | June 2018
AnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data Exploration
Nan-Chen Chen, Jina Suh, Johan Verwey, Gonzalo Ramos, Steven Drucker, Patrice Simard
IUI ’18 23rd International Conference on Intelligent User Interfaces | March 2018
ISBN: 978-1-4503-4945-1
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
Joseph Chee Chang, Saleema Amershi, Ece Kamar
Proceedings of the Conference on Human Factors in Computing Systems (CHI 2017) | May 2017
Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, Jason Williams
IEEE TVCG, VAST | August 2016
Interactive Semantic Featuring for Text Classification
Patrice Simard, Max Chickering, Jina Suh
Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016) | June 2016
The Label Complexity of Mixed-Initiative Classiï¬er Training
Jina Suh, Xiaojin Zhu, Saleema Amershi
Proceedings of the 33rd International Conference on Machine Learning | June 2016
FeatureInsight: Visual Support for Error-Driven Feature Ideation in Text Classification
Michael Brooks, Saleema Amershi, Bongshin Lee, Steven Drucker, Ashish Kapoor, Patrice Simard
IEEE VAST | October 2015
Fast and Easy Language Understanding For Dialog Systems With Microsoft Language Understanding Intelligent Service (LUIS)
Jason Williams, Eslam Kamal, Mikhtar Ashour, Hani Amr, Jessica Miller, Geoffrey Zweig
Proceedings of 2015 SIGDIAL Conference, Prague | September 2015
Demonstration at SIGDIAL 2015.
ModelTracker: Redesigning Performance Analysis Tools for Machine Learning
Saleema Amershi, Max Chickering, Steven Drucker, Bongshin Lee, Patrice Simard, Jina Suh
Proceedings of the Conference on Human Factors in Computing Systems (CHI 2015) | April 2015
Rapidly Scaling Dialog Systems with Interactive Learning
Jason Williams, Nobal B. Niraula, Pradeep Dasigi, Aparna Lakshmiratan, Carlos Garcia Jurado Suarez, Mouni Reddy, Geoffrey Zweig
January 2015
Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy
Denny Zhou, Qiang Liu, John Platt, Chris Meek
Proceedings of the 31st International Conference on Machine Learning (ICML) | June 2014
Proceedings of the 31st International Conference on Machine Learning (ICML)
Structured Labeling for Facilitating Concept Evolution in Machine Learning
Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, Denis Charles
Proceedings of the Conference on Human Factors in Computing Systems (CHI 2014) | May 2014
Best Paper Award
Learning from Our Mistakes: Towards a Correctable Learning Algorithm
Karthik Raman, Krysta M. Svore, Ran Gilad-Bachrach, Chris J.C. Burges
21st International Conference on Information and Knowledge Management (CIKM) | October 2012
21st International Conference on Information and Knowledge Management (CIKM)
-
Do Machine Teachers Dream of Algorithms?
Gonzalo Ramos, Felicia Ng, Nicole Sultanum, Chris Meek, Jina Suh, Soroush Ghorashi
December 2019
-
Machine Teaching: A New Paradigm for Building Machine Learning Systems
Patrice Simard, Saleema Amershi, Max Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Chris Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, John Wernsing
MSR-TR-2017-26 | July 2017
A Characterization of Prediction Errors
Chris Meek
MSR-TR-2016-1105 | November 2016
Analysis of a Design Pattern for Teaching with Features and Labels
Chris Meek, Patrice Simard, Xiaojin Zhu
MSR-TR-2016-1104 | November 2016
ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
Patrice Simard, Max Chickering, Apama Lakshmiratan, Denis Charles, Leon Bottou, Carlos Garcia Jurado Suarez, David Grangier, Saleema Amershi, Johan Verwey, Jina Suh
MSR-TR-2014-128 | September 2014