Microsoft at NeurIPS 2024

Reinforcement Learning Under Latent Dynamics: Toward Statistical and Algorithmic Modularity

Philip Amortila, Dylan Foster, Nan Jiang, Akshay Krishnamurthy, Zakaria Mhammedi

A Large-Scale Human-Centric Benchmark for Referring Expression Comprehension in the LMM Era

Fangyun Wei

Accuracy is Not All You Need

Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, Ramachandran Ramjee

Blind Image Restoration via Fast Diffusion Inversion

Hamadi Chihaoui, Abdelhak Lemkhenter, Paolo Favaro

BPQP: A Differentiable Convex Optimization Framework for Efficient End-to-End Learning

Xiao Yang, Xu Yang, Weiqing Liu, Lewen Wang, Jiang Bian

Compositional Generalization Across Distributional Shifts with Sparse Tree Operations

Paul Smolensky, Jianfeng Gao, Roland Fernandez

Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation

Sahar Abdelnabi, Amr Gomaa, S. Sivaprasad, Lea Schonherr, Mario Fritz

Gorilla: Teaching LLMs to Use Tools

Xin Wang

Implicit Curriculum in Procgen Made Explicit

Kaixin Wang, Xinchao Wang

Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning

Alex Jinpeng Wang, Linjie Li, Yiqi Lin, Min Li, Lijuan Wang, Mike Zheng Shou

Motion Graph Unleashed: A Novel Approach to Video Prediction

Luming Liang, Ilya Zharkov

Semi-Random Matrix Completion via Flow-Based Adaptive Reweighting

Jerry Li

Towards Flexible Visual Relationship Segmentation

Fangrui Zhu, Jianwei Yang, Huaizu Jiang

VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time

Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, Baining Guo

A Generative Approach to Control Complex Physical Systems

Long Wei, Peiyan Hu, Ruiqi Feng, Haodong Feng, Yixuan Du, Tao Zhang, Rui Wang, Yue Wang, Zhi-Ming Ma, Tailin Wu

A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning

Arthur Juliani, Jordan Ash

Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators

Changze Lv, Dongqi Han, Yansen Wang, Xiaoqing Zheng, Xuanjing Huang, Dongsheng Li

Amortized Active Causal Induction with Deep Reinforcement Learning

Yashas Annadani, P. Tigas, Stefan Bauer, Adam Foster, Adam Foster

Approximately Equivariant Neural Processes

Matthew Ashman, Cristiana-Diana Diaconu, Adrian Weller, Wessel Bruinsma, Richard Turner

Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency

Xian Zhang, Fan Yang

Boosting Text-to-Video Generative Model with MLLMs Feedback

Xun Wu, Shaohan Huang, Furu Wei

Can Graph Learning Improve Task Planning?

Xixi Wu, Yifei Shen, Caihua Shan, Kaitao Song, Siwei Wang, Bohang Zhang, Jiarui Feng, Hong Cheng, Wei Chen, Yun Xiong, Dongsheng Li

ElasTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformer

Shun Zheng, Xumeng Wen, Jiang Bian

GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling

Bowen Zhang, Yiji Cheng, Jiaolong Yang, Chunyu Wang, Feng Zhao, Yansong Tang, Dong Chen, Baining Guo

Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations

Hao Chen, Ankit Shah, Jindong Wang, R. Tao, Yidong Wang, Xing Xie, Masashi Sugiyama, Rita Singh, B. Raj

Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation

Yuanhao Zhai, Kevin Lin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Chung-Ching Lin, David Doermann, Junsong Yuan, Lijuan Wang

Multimodal Large Language Models Make Text-to-Image Generative Models Align Better

Xun Wu, Shaohan Huang, Furu Wei

Online Estimation via Offline Estimation: An Information-Theoretic Framework

Dylan Foster, Yanjun Han, Jian Qian, Alexander Rakhlin

Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning

Rui Wang, Junliang Guo, Xu Tan

SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation

Lester Mackey, Miro Dudík, Alex Chouldechova

VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time

Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, Baining Guo

Not All Tokens Are What You Need for Pretraining

Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Jian Jiao, Nan Duan, Weizhu Chen

Bridging Geometric States via Generative Modeling

Shuxin Zheng, Tie-Yan Liu

CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses

Jing Yao, Xiaoyuan Yi, Xing Xie

Crafting Interpretable Embeddings by Asking LLMs Questions

Vinamra Benara, Chandan Singh, John X. Morris, Richard Antonello, Ion Stoica, Alexander Huth, Jianfeng Gao

Efficient Adversarial Training in LLMs with Continuous Attacks

Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, Leo Schwinn

How to Solve Contextual Goal-Oriented Problems with Offline Datasets?

Ying Fan, Jingling Li, Adith Swaminathan, Aditya Modi, Ching-An Cheng

Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions

Minghan Li, Heng Li, Zhi-Qi Cheng, Yifei Dong, Yuxuan Zhou, Jun-Yan He, Qi Dai, Teruko Mitamura, Alexander G. Hauptmann

Identifying Equivalent Training Dynamics

William T. Redman, J. M. Bello-Rivas, M. Fonoberova, Ryan Mohr, I. Kevrekidis, Igor Mezi'c

Improving the Training of Rectified Flows

Sangyun Lee, Zinan Lin, Giulia Fanti

MEDIQ: Question-Asking LLMs for Adaptive and Reliable Clinical Reasoning

Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan Ilgen, Emma Pierson, Pang Wei Koh, Yulia Tsvetkov

Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models

Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, Furu Wei

Neuro-Symbolic Data Generation for Math Reasoning

Xian Zhang, Fan Yang

Noise-Aware Differentially Private Regression via Meta-Learning

Ossi Räisä, Stratis Markou, Matthew Ashman, Wessel Bruinsma, Marlon Tobaben, Antti Honkela, Richard Turner

PromptFix: You Prompt and We Fix the Photo

Yongsheng Yu, Ziyun Zeng, Hang Hua, Jianlong Fu, Jiebo Luo

Protecting Your LLMs with Information Bottleneck

Zichuan Liu, Zefan Wang, Linjie Xu, Jinyu Wang, Lei Song, Tianchun Wang, Chunlin Chen, Wei Cheng, Jiang Bian

ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos

Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen-Chun Chen, Yu-Chiang Frank Wang

SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining

Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, Bamdev Mishra

TaskBench: Benchmarking Large Language Models for Task Automation

Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, Y. Zhuang

The Art of Saying No: Contextual Noncompliance in Language Models

Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, K. Chandu, Jack Hessel, Yulia Tsvetkov, Noah A. Smith, Yejin Choi, Hanna Hajishirzi, Vidhisha Balachandran

VideoGUI: A Benchmark for GUI Automation from Instructional Videos

Kevin Qinghong Lin, Linjie Li, Difei Gao, Qinchen Wu, Mingyi Yan, Zhengyuan Yang, Lijuan Wang, Mike Zheng Shou

CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark

David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, A. Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, D. Adelani, D. Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Ganzorig Batnasan, Gisela Vallejo, Grainne Caulfield, Guido Ivetta, Haiyue Song, Henok Biadglign Ademtew, Hernán Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesús-Germán Ortiz-Barajas, Jinheon Baek, Jocelyn Dunstan, L. A. Alemany, Kumaranage Ravindu Yasas Nagasinghe, Luciana Benotti, L. F. D’Haro, Marcelo Viridiano, Marcos Estecha-Garitagoitia, Maria Camila Buitrago Cabrera, Mario Rodr'iguez-Cantelar, Mélanie Jouitteau, M. Mihaylov, Mohamed Fazli Mohamed Imam, Muhammad Farid Adilazuarda, Munkhjargal Gochoo, Munkh-Erdene Otgonbold, Naome A. Etori, Olivier Niyomugisha, Paula M'onica Silva, Pranjal A. Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryandito Diandaru, Samuel Cahyawijaya, Santiago G'ongora, Soyeong Jeong, Sukannya Purkayastha, Tatsuki Kuribayashi, Thanmay Jayakumar, T. Torrent, Toqeer Ehsan, Vladimir Araujo, Yova Kementchedjhieva, Zara Burzo, Zheng Wei Lim, Zheng-Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea, T. Solorio, Alham Fikri Aji

A Framework for Bilevel Optimization on Riemannian Manifolds

Andi Han, Bamdev Mishra, Pratik Jawanpuria, Akiko Takeda

A Label is Worth a Thousand Images in Dataset Distillation

Tian Qin, Zhiwei Deng, David Alvarez-Melis

Active, anytime-valid risk controlling prediction sets

Nikos Karampatziakis, Paul Mineiro

Aligning LLM Agents by Learning Latent Preference from User Edits

Ge Gao, Alexey Taymanov, Eduardo Salinas, Paul Mineiro, Dipendra Misra

Can large language models explore in-context?

Akshay Krishnamurthy, Keegan Harris, Dylan Foster, Cyril Zhang, Aleksandrs Slivkins

Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement

Tao Yang, Cuiling Lan, Yan Lu, Nanning Zheng

DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents

Peter Alexander Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi, Bodhisattwa Prasad Majumder, Oyvind Tafjord, Peter Clark

Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization

Mucong Ding, Chenghao Deng, Jocelyn Choo, Zichu Wu, Aakriti Agrawal, Avi Schwarzschild, Tianyi Zhou, Tom Goldstein, John Langford, A. Anandkumar, Furong Huang

Infusing Self-Consistency into Density Functional Theory Hamiltonian Prediction via Deep Equilibrium Models

Zun Wang, Chang Liu, Nianlong Zou, He Zhang, Xinran wei, Lin Huang, Lijun Wu, Bin Shao

Instruction-Guided Visual Masking

Jinliang Zheng, Jianxiong Li, Si Cheng, Yinan Zheng, Jiaming Li, Jihao Liu, Yu Liu, Jingjing Liu, Xianyuan Zhan

InversionView: A General-Purpose Method for Reading Information from Neural Activations

Xinting Huang, Madhur Panwar, Navin Goyal, Michael Hahn

Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning

Dylan Foster, Adam Block, Dipendra Misra

Make Your LLM Fully Utilize the Context

Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou

Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs

Jingjing Liu

Meta-Diffu$B$: A Contextualized Sequence-to-Sequence Text Diffusion Model with Meta-Exploration

Kevin Lin

Multi-Head Mixture-of-Experts

Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei

Policy Improvement using Language Feedback Models

Victor Zhong, Dipendra Misra, Xingdi Yuan, Marc-Alexandre Côté

QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, James Hensman, Pashmina Cameron

RedCode: Risky Code Execution and Generation Benchmark for Code Agents

Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, Bo Li

Topic-Conversation Relevance (TCR) Dataset and Benchmarks

Yaran Fan, Jamie Pool, Senja Filipi, Ross Cutler

Understanding Information Storage and Transfer in Multi-modal Large Language Models

Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, S. Feizi, Daniela Massiceti

A Bayesian Approach to Data Point Selection

Xinnuo Xu, Minyoung Kim, Royson Lee, Brais Martinez, Timothy M. Hospedales

Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms

Miaosen Zhang, Yixuan Wei, Zhen Xing, Yifei Ma, Zuxuan Wu, Ji Li, Zheng Zhang, Qi Dai, Chong Luo, Xin Geng, Baining Guo

CARE: a Benchmark Suite for the Classification and Retrieval of Enzymes

Jason Yang, Ariane Mora, Shengchao Liu, Bruce J. Wittmann, A. Anandkumar, Frances H. Arnold, Yisong Yue

C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory

Tianjiao Luo, Tim Pearce, Huayu Chen, Jianfei Chen, Jun Zhu

CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations

Leying Zhang, Yao Qian, Long Zhou, Shujie Liu (shujliu), Dongmei Wang, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Lei He, Sheng Zhao, Michael Zeng

CultureLLM: Incorporating Cultural Differences into Large Language Models

Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, Xing Xie

ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models

Jio Oh, Soyeon Kim, Junseok Seo, Jindong Wang, Ruochen Xu, Xing Xie, Steven Euijong Whang

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

Zhenyu (Allen) Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang

Generalized Linear Bandits with Limited Adaptivity

Ayush Sawarni, Nirjhar Das, Siddharth Barman, Gaurav Sinha

Improving Context-Aware Preference Modeling for Language Models

Silviu Pitis, Ziang Xiao, Nicolas Le Roux, Alessandro Sordoni

Query-Efficient Correlation Clustering with Noisy Oracle

Yuko Kuroki, Atsushi Miyauchi, Francesco Bonchi, Wei Chen

Scaling the Codebook Size of VQ-GAN to 100,000 with a Utilization Rate of 99%

Fangyun Wei, Dong Chen

Slot-VLM: SlowFast Slots for Video-Language Modeling

Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, Yan Lu

The Power of Resets in Online Reinforcement Learning

Zakaria Mhammedi, Dylan Foster, Alexander Rakhlin

Understanding and Improving Training-free Loss-based Diffusion Guidance

Yifei Shen, Xinyang Jiang, Yezhen Wang, Yifan Yang, Dongqi Han, Dongsheng Li

Using Noise to Infer Aspects of Simplicity Without Learning

Lesia Semenova

Voila-A: Aligning Vision-Language Models with User's Gaze Attention

Kun Yan, Lei Ji, Zeyu Wang, Yuntao Wang, Nan Duan, Shuai Ma

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei

ALPINE: Unveiling the Planning Capability of Autoregressive Learning in Language Models

Siwei Wang, Yifei Shen, Shi Feng, Haoran Sun, Shang-Hua Teng, Wei Chen

Can LLMs Learn by Teaching? A Preliminary Study

Xuefei Ning, Zifu Wang, Shiyao Li, Zinan Lin, Peiran Yao, Tianyu Fu, Matthew B. Blaschko, Guohao Dai, Huazhong Yang, Yu Wang

CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models

Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, Wenhao Zheng, Zhaoyang Wang, Xiao Wang, Xuchao Zhang, Chetan Bansal, Marc Niethammer, Junzhou Huang, Hongtu Zhu, Yun Li, Jimeng Sun, Zongyuan Ge, Gang Li, James Zou, Huaxiu Yao

Compositional 3D-aware Video Generation with LLM Director

Hanxin Zhu, Tianyu He, Anni Tang, Junliang Guo, Zhibo Chen, Jiang Bian

CulturePark: Boosting Cross-cultural Understanding in Large Language Models

Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, Jindong Wang

Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition

Edoardo Debenedetti, Javier Rando, Daniel Paleka, Fineas Silaghi, Dragos Albastroiu, Niv Cohen, Yuval Lemberg, Reshmi Ghosh, Ahmed Salem, Rui Wen, Giovanni Cherubin, Santiago Zanella-Béguelin, Robin Schmid, Victor Klemm, Takahiro Miki, Chenhao Li, Stefan Kraft, Mario Fritz, Florian Tramer, Sahar Abdelnabi, Lea Schönherr

Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models

Javier González, Aditya Nori

Don't Just String Tokens, Stack Them! Improving Multimodal Transformers with Layer Stack

Jianwei Yang, Xiyang Dai, Jianfeng Gao

EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals

Yansen Wang, Zilong Wang

Energy-efficient Reinforcement Learning by Discovering Neural Pathways

Alessandro Sordoni

Maia-2: A Unified Model for Human-AI Alignment in Chess

Zhenwei Tang, Difan Jiao, Reid McIlroy-Young, Jon Kleinberg, Siddhartha Sen, Ashton Anderson

Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models

Vibhav Vineet, Xin Wang, Neel Joshi

MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu

Neural P$^3$M: A Long-Range Interaction Modeling Enhancer for Geometric GNNs

Yusong Wang, Chaoran Cheng, Shaoning Li, Yuxuan Ren, Bin Shao, Ge Liu, Pheng-Ann Heng, Nanning Zheng

ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons

Jiawen Zhang, Xumeng Wen, Zhenwei Zhang, Shun Zheng, Jia Li, Jiang Bian

Provably Efficient Interactive-Grounded Learning with Personalized Reward

Mengxiao Zhang, Yuheng Zhang, Haipeng Luo, Paul Mineiro

TorchSpatial: A Location Encoding Framework and Benchmark for Spatial Representation Learning

Nemin Wu, Qian Cao, Zhangyu Wang, Zeping Liu, Yanlin Qi, Jielu Zhang, Joshua Ni, Xiaobai Yao, Hongxu Ma, Lan Mu, Stefano Ermon, Tanuja Ganu, Akshay Nambi, Ni Lao, Gengchen Mai

Trace is the New AutoDiff — Unlocking Efficient Optimization of Computational Workflows

Ching-An Cheng, Allen Nie, Adith Swaminathan

TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

Chenyang Le, Yao Qian, Dongmei Wang, Long Zhou, Shujie Liu, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Sheng Zhao, Michael Zeng

What can Foundation Models’ Embeddings do?

Linjie Li, Jianwei Yang, Lijuan Wang

xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token

Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, Dongyan Zhao

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei