Beyond Accuracy: Realistic and Diagnostic Evaluation of Code Generation Models
Pareesa Ameneh Golnari, Xiaoyu Liu (lixiaoyu), Gabriel Ryan (ryangabriel), Shengyu Fu (shengyfu)
Deep Learning for Code at NeurIPS | November 2025
Shengyu Fu
Partner Applied Science Manager
Publications
-
-
-
SWE-bench Goes Live!
Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, J. Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin 林庆维, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang
NeurIPS 2025 | May 2025
-
-
-
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang (姜大昕), Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong (YIMING), Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, Shujie Liu
February 2021
-
-
-
GraphCodeBERT: Pre-training Code Representations with Data Flow
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang (姜大昕), Ming Zhou
September 2020
-
Pythia: AI-assisted Code Completion System
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, Neel Sundaresan
25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '19) | September 2020
-
IntelliCode Compose: Code Generation Using Transformer
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, Neel Sundaresan
28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’20) | May 2020
-
-
-
Beyond Accuracy: Realistic and Diagnostic Evaluation of Code Generation Models
Pareesa Ameneh Golnari, Xiaoyu Liu (lixiaoyu), Gabriel Ryan (ryangabriel), Shengyu Fu (shengyfu)
Deep Learning for Code at NeurIPS | November 2025
-
SWE-bench Goes Live!
Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, J. Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin 林庆维, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang
NeurIPS 2025 | May 2025
-
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang (姜大昕), Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong (YIMING), Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, Shujie Liu
February 2021
-
Pythia: AI-assisted Code Completion System
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, Neel Sundaresan
25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '19) | September 2020
-
IntelliCode Compose: Code Generation Using Transformer
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, Neel Sundaresan
28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’20) | May 2020
-
-
-
SWE-bench Goes Live!
Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, J. Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin 林庆维, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang
NeurIPS 2025 | May 2025
-
GraphCodeBERT: Pre-training Code Representations with Data Flow
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang (姜大昕), Ming Zhou
September 2020
-
Pythia: AI-assisted Code Completion System
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, Neel Sundaresan
25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '19) | September 2020
-
IntelliCode Compose: Code Generation Using Transformer
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, Neel Sundaresan
28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’20) | May 2020
-
-
-
Pythia: AI-assisted Code Completion System
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, Neel Sundaresan
25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '19) | September 2020
-
-
-
Beyond Accuracy: Realistic and Diagnostic Evaluation of Code Generation Models
Pareesa Ameneh Golnari, Xiaoyu Liu (lixiaoyu), Gabriel Ryan (ryangabriel), Shengyu Fu (shengyfu)
Deep Learning for Code at NeurIPS | November 2025
-
SWE-bench Goes Live!
Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, J. Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin 林庆维, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang
NeurIPS 2025 | May 2025
-
Pythia: AI-assisted Code Completion System
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, Neel Sundaresan
25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '19) | September 2020
-
IntelliCode Compose: Code Generation Using Transformer
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, Neel Sundaresan
28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’20) | May 2020
-
-
-
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang (姜大昕), Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong (YIMING), Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, Shujie Liu
February 2021
-
GraphCodeBERT: Pre-training Code Representations with Data Flow
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang (姜大昕), Ming Zhou
September 2020
-