Inside Audrey Tang’s Plan to Align Technology and Democracy
May 20, 2024 | E. Glen Weyl
E. Glen Weyl and Audrey Tang's new book provides a vision of how advanced collaborative technology can bolster democracy.
Microsoft Research Blog
English
-
-
Optimistic Query Routing in Clustering-based Approximate Maximum Inner Product Search
May 19, 2024 | Sebastian Bruch, Aditya Krishnan, and F. M. NardiniClustering-based nearest neighbor search is an effective method in which points are partitioned into geometric shards to form an index, with only a few shards searched during query processing to find a set of top-$k$ vectors. Even though the search efficacy is heavily influenced by…
-
Diffusion for World Modeling: Visual Details Matter in Atari
May 19, 2024World models constitute a promising approach for training reinforcement learning agents in a safe and sample-efficient manner. Recent world models predominantly operate on sequences of discrete latent variables to model environment dynamics. However, this compression into a compact discrete representation may ignore visual details that…
-
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
May 19, 2024Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models. In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize…
-
Synthetic Test Collections for Retrieval Evaluation
May 18, 2024Test collections play a vital role in evaluation of information retrieval (IR) systems. Obtaining a diverse set of user queries for test collection construction can be challenging, and acquiring relevance judgments, which indicate the appropriateness of retrieved documents to a query, is often costly and…
-
MInference: Accelerating Pre-filling for Long-context LLMs via Dynamic Sparse Attention
May 17, 2024MInference 1.0 leverages the dynamic sparse nature of LLMs' attention, which exhibits some static patterns, to speed up the pre-filling for long-context LLMs. It first determines offline which sparse pattern each head belongs to, then approximates the sparse index online and dynamically computes attention with…
-
You Only Need Less Attention Each Stage in Vision Transformers
May 17, 2024 | Shuoxi Zhang, Hanpeng Liu, Stephen Lin, and Kun HeThe advent of Vision Transformers (ViTs) marks a substantial paradigm shift in the realm of computer vision. ViTs capture the global information of images through self-attention modules, which perform dot product computations among patchified image tokens. While self-attention modules empower ViTs to capture long-range dependencies,…
-
Towards Modular LLMs by Building and Reusing a Library of LoRAs
May 17, 2024The growing number of parameter-efficient adaptations of a base large language model (LLM) calls for studying whether we can reuse such trained adapters to improve performance for new tasks. We study how to best build a library of adapters given multi-task data and devise techniques…
-
-
Assouad, Fano, and Le Cam with Interaction: A Unifying Lower Bound Framework and Characterization for Bandit Learnability
May 17, 2024In this paper, we develop a unified framework for lower bound methods in statistical estimation and interactive decision making. Classical lower bound techniques---such as Fano's inequality, Le Cam's method, and Assouad's lemma---have been central to the study of minimax risk in statistical estimation, yet they…
-
Watching the Air Rise: Learning-Based Single-Frame Schlieren Detection
May 17, 2024Detecting air flows caused by phenomena such as heat convection is valuable in multiple scenarios, including leak identification and locating thermal updrafts for extending UAVs’ flight duration. Unfortunately, these flows’ heat signature is often too subtle to be seen by a thermal camera. While convection…
-
A Game-theoretic Framework for Privacy-preserving Federated Learning
May 17, 2024In federated learning, benign participants aim to optimize a global model collaboratively. However, the risk of privacy leakage cannot be ignored in the presence of semi-honest adversaries. Existing research has focused either on designing protection mechanisms or on inventing attacking mechanisms. While the battle between…