Here, we provide a plug-in-and-play implementation of Admin, which stabilizes previously-diverged Transformer training and achieves better performance, without introducing additional hyper-parameters. The design of Admin is half-precision friendly and can be reparameterized into the original Transformer.
XtremeDistil framework for distilling/compressing massive multilingual neural network models to tiny and efficient models for AI at scale.
This repo contains the source code of the Python package loralib and several examples of how to integrate it with PyTorch models, such as those in HuggingFace. We only support PyTorch for now. See our paper for a detailed description of LoRA.
Implementation of MoLeR: a generative model of molecular graphs which supports scaffold-constrained generation. This open-source code accompanies our paper “Learning to Extend Molecular Scaffolds with Structural Motifs”, which has been accepted at the ICLR 2022 conference.
Github link to Iris – pretrained summarization models for structured datasets and cardinality estimation.
Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators. This repository contains the scripts for fine-tuning AMOS pretrained models on GLUE and SQuAD 2.0 benchmarks. Accompanying paper accepted to ICLR 2022: “Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators”
Knowledge Infused Decoding (KID) is a decoding algorithm that infuses knowledge (from Wikipedia) into each step decoding of text generation.
Jigsaw Dataset: Natural language to Python Pandas code. Two datasets (PandasEval1 and PandasEval2) described in our paper, “Jigsaw: Large Language Models meet Program Synthesis”.
Microsoft Collective Communication Library (MSCCL) is a platform to execute custom collective communication algorithms for multiple accelerators supported by Microsoft Azure.
Maximal Update Parametrization (μP) and Hyperparameter Transfer (μTransfer), in association with the paper: Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
