Microsoft Research Highlights: Over 100 Papers at NeurIPS 2022

Microsoft Research Focus: NeurIPS 2022 Highlights

This edition of Microsoft Research Focus highlights over 100 papers accepted at NeurIPS 2022, the thirty-sixth annual Conference on Neural Information Processing Systems. The content showcases advancements across various AI and ML domains, including:

Key Research Areas and Papers:

• Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models: This paper introduces AutoDistil, a novel technique that uses Neural Architecture Search (NAS) for knowledge distillation (KD). AutoDistil automates the generation of compressed models with varying computational costs (FLOPs, latency) and achieves significant reductions in FLOPs with minimal performance regression. It addresses customization challenges for deploying models in resource-constrained environments.

• Neuron with Steady Response Leads to Better Generalization: Researchers explored fundamental properties of deep neural networks and identified that stable neuron responses for the same class improve generalization. They developed Neuron Steadiness Regularization (NSR), a computationally efficient method applicable to various architectures and tasks, demonstrating significant improvements in model generalization.

• Long-form Video-Language Pre-training with Multimodal Temporal Contrastive Learning: This work proposes LF-VILA, a model for long-form video representation learning, and introduces a Multimodal Temporal Contrastive (MTC) loss and Hierarchical Temporal Window Attention (HTWA) mechanism. LF-VILA achieves state-of-the-art results on various video-language tasks, including retrieval and action recognition.

• Microsoft Research Causality and ML Team at NeurIPS 2022: The team presented multiple papers and workshops on causality and machine learning, focusing on causal discovery, responsible language models, and soil carbon modeling. They also released open-source tools like ShowWhy for causal discovery and analysis.

  • Main Conference Papers:
    • Simultaneous Missing Value Imputation and Structure Learning with Groups
    • Probing Classifiers are Unreliable for Concept Removal and Detection
    • ML4S: Learning Causal Skeleton from Vicinal Graphs
  • Workshop Papers: Contributions to workshops on Causality for Real-world Impact, Tackling Climate Change with Machine Learning, Distribution Shifts, and Understanding Deep Learning Through Empirical Falsification.
  • Causal AI Software Resources: ShowWhy, PyWhy, Causica.

• New Research on Generative Models: Two papers were presented at NeurIPS 2022:

  • Operationalizing Specifications, In Addition to Test Sets for Evaluating Constrained Generative Models: This paper provides recommendations for evaluating generative models, emphasizing the use of specifications to assess generation quality and adapt evaluation methodologies to the scale and capabilities of modern models.
  • Rank-One Editing of Encoder-Decoder Models: This research explores rank-one editing as a direct intervention method for behavior deletion in encoder-decoder transformer models, demonstrating high efficacy with minimal data for fixing erroneous behaviors.

• Award Winner: A Neural Corpus Indexer for Document Retrieval: This paper, recognized with an Outstanding Paper award at NeurIPS 2022, introduces the Neural Corpus Indexer (NCI). NCI uses a sequence-to-sequence architecture as a model-based index, outperforming traditional sparse and dense retrieval methods and achieving significant improvements in recall and retrieval metrics.

Career Opportunities:

Microsoft Research is hiring for various roles, including internships and researchers. Interested candidates are encouraged to visit the Microsoft booth at NeurIPS 2022 or browse openings online.

Related Publications:

A comprehensive list of related publications is provided, covering areas such as causal reasoning, generative models, document retrieval, and generalization in machine learning.

Continued Reading:

Links to recent Microsoft Research blog posts are included, covering topics from November 2023 to November 2024.

Research Areas and Groups:

Highlights include Artificial Intelligence as a key research area, and research groups such as Multimedia Search and Mining, Machine Translation, Causality and Machine Learning, and Data, Knowledge, and Intelligence.

Follow Microsoft Research:

Links are provided to follow Microsoft Research on X, Facebook, LinkedIn, YouTube, and Instagram, as well as subscribe to their RSS feed.