Garbled highlights from ICLR 2025

4.1 Day 1, 27th April

• Workshop on Weight Space Learning

  The recent surge in the number of publicly available neural network models — exceeding a million on platforms like Hugging Face — calls for a shift in how we perceive neural network weights. This workshop aims to establish neural network weights as a new data modality, offering immense potential across various fields.

  We plan to address key dimensions of weight space learning:

  • Weight Space as a Modality: Characterisation of the weight space, symmetries, scaling laws, and model zoo datasets.
  • Model analysis: Inferring model properties, such as test performance or generalisation error, by the inspection of their weights.
  • Model synthesis: Generating model weights for specific datasets and tasks, and improving tasks like pruning, merging, and robustness through weight manipulation.

  This is a great idea. I have not really had time to digest it all though. TBC

• XAI4Science: From Understanding Model Behaviour to Discovering New Scientific Knowledge

  Machine learning (ML) models are impressive when they work but they can also show unreliable, untrustworthy, and harmful behaviour. Yet, such models are widely adopted and deployed, even though we do not understand why they work so well and fail miserably at times. Such rapid dissemination encourages irresponsible use, for example, to spread misinformation or create deep fakes, while hindering the efforts to use them to solve pressing societal problems and advance human knowledge.

  Ideally, we want models to help us improve our understanding of the world and, at the very least, we want them to aid human knowledge and help us to further enrich it. Our goal in this workshop is to take a step in this direction by bringing together researchers working on understanding model behaviour and using it to discover new human knowledge. The workshop will include theoretical topics on understanding model behaviour, namely interpretability and explainability (XAI), but also three distinct scientific application areas: weather and climate, healthcare, and material science (ML4Science).

  Generally a nice idea, but it felt like the tech firms were rather absent from this workshop given their engagement in the area. More worrying signs of the tech firms pulling back from open research?

• BiAlign ICLR 2025

  This workshop focuses on bidirectional Human↔︎AI alignment, a paradigm shift in how we approach the challenge of human-AI alignment, which emphasises the dynamic, complex, and evolving alignment process between humans and AI systems. This is grounded on the “bidirectional human-AI alignment” framework (see Definition and ReadingList) derived from a systematic survey of over 400 interdisciplinary alignment papers in Machine Learning (ML), Human Computer Interaction (HCI), Natural Language Processing (NLP), and more domains. Particularly, it involves two directions to maximise its benefits for human society.

  • Aligning AI with Humans (AI-centred perspective): focuses on integrating human specifications into training, steering, customising, and monitoring AI systems;
  • Aligning Humans with AI (Human-centred perspective): aims to preserve human agency and empower humans to critically evaluate, explain, and collaborate with AI systems.

  I appreciated the goals of this workshop, but the bits I saw didn’t help me so much. However, I did learn about assistance games, which was interesting indeed.

4.2 Day 2, 28th April

• Tackling Climate Change with Machine Learning | Climate Change AI

  Building on our past workshops on this topic, this edition focuses on two aspects related to data-centric approaches to ML for climate action. Data-centric machine learning is not only a timely topic within the ICLR community, as analysing and engineering (pre)training datasets becomes increasingly important, but holds specific challenges and opportunities in climate-related areas, for example in smart grid management or water engineering.

  This was IMO a surprise breakout workshop. I was not expecting much from it, but the workshops were generally IMO starved for content this year, and the climate one was popping with ideas and energy.

• FPI @ ICLR2025

  The Frontiers in Probabilistic Inference: Sampling meets Learning (FPI) workshop at ICLR 2025 focuses on modern approaches to probabilistic inference to address the challenging and under-explored area of sampling from an unnormalised distribution. Sampling spans a wide range of difficult and timely problems from molecular dynamics simulation, and Bayesian posterior inference/inverse problems to sampling from generative models weighted by target density (e.g. finetuning, inference-time alignment). We hope to provide an inclusive and collaborative environment to discuss emerging ML methods for learning samplers and their applications to real-world problems. We aim to facilitate discussions around identifying some key challenges of learning-based approaches, compared to classical sampling approaches, along with techniques to overcome them.

  This was amazing IMO, but somewhat eclipsed by AABI.

• Workshop on Spurious Correlation and Shortcut Learning

  Reliance on spurious correlations due to simplicity bias is a well-known pitfall of deep learning models. This issue stems from the statistical nature of deep learning algorithms and their inductive biases at all stages, including data preprocessing, architectures, and optimisation. Therefore, spurious correlations and shortcut learning are fundamental and common practical problems across all branches of AI. The foundational nature and widespread occurrence of reliance on spurious correlations and shortcut learning make it an important research topic and a gateway to understanding how deep models learn patterns and the underlying mechanisms responsible for their effectiveness and generalisation. This workshop aims to address two aspects of this phenomenon: its foundations and potential solutions.

4.3 Day 3, 29th April

Not technically the same conference, but with a massive overlap: Advances in Approximate Bayesian Inference

In recent years, advances in probabilistic machine learning and approximate inference methods have enabled the development of state-of-the-art methods across many domains, such as in generative modelling, AI-driven scientific discovery, and AI safety, among many others.

This year’s symposium will have an expanded scope and will be focused on the development, analysis, and application of probabilistic machine learning methods broadly construed. In line with this expanded focus, we particularly welcome submissions that explore connections between probabilistic machine learning and other fields such as deep learning, natural language processing, active learning, reinforcement learning, compression, AI safety, AI for scientific discovery, and causal inference. We also encourage contributions that advance the foundations of probabilistic machine learning and machine learning theory.

This was amazing and highly recommended. I learned about q-exponential processes, Jesse Hoogland and Zach Furman got me one step closer to understanding what the sampling problem is in singular learning theory, as evidenced in Lau et al. (2024). Emtiyaz Khan corrected some of my misapprehensions about the Bayes Learning Rule. I did not quite get to the bottom of my MMD problems despite Pierre Alquier’s noble efforts, although the fault here is mine. Christian A. Naesseth explained to me Bartosh, Vetrov, and Naesseth (2025), which was super cool.

Thanks to the AABI organisers for a great workshop!

6.1 Blog posts

Firstly, don’t sleep on the ICLR Blogposts, which demonstrate some next level science-communication. My picks, in alphabetical order:

• A Visual Dive into Conditional Flow Matching
• An Illustrated Guide to Automatic Sparse Differentiation
• Flow With What You Know
• LLMs’ Potential Influences on Our Democracy: Challenges and Opportunities
• Multi-modal Learning: A Look Back and the Road Ahead
• Pitfalls of Evidence-Based AI Policy
• Positional Embeddings in Transformer Models: Evolution from Text to Vision Domains
• The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?
• Understanding Model Calibration - A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)

6.2 Physics I’m interested in

• Physics-Informed Diffusion Models code at jhbastek/PhysicsInformedDiffusionModels: Implementation of Physics-Informed Diffusion Models (Bastek, Sun, and Kochmann 2024)
• Latent-EnSF: A Latent Ensemble Score Filter for High-Dimensional Data Assimilation with Sparse Observation Data (Si and Chen 2024)

6.3 Multi-agent systems + Safety

• Advantage Alignment Algorithms Oral Poster. Code at jduquevan/advantage-alignment (Duque et al. 2024)

  Artificially intelligent agents are increasingly being integrated into human decision-making: from large language model (LLM) assistants to autonomous vehicles. These systems often optimize their individual objective, leading to conflicts, particularly in general-sum games where naive reinforcement learning agents empirically converge to Pareto-suboptimal Nash equilibria. To address this issue, opponent shaping has emerged as a paradigm for finding socially beneficial equilibria in general-sum games. In this work, we introduce Advantage Alignment, a family of algorithms derived from first principles that perform opponent shaping efficiently and intuitively. We achieve this by aligning the advantages of interacting agents, increasing the probability of mutually beneficial actions when their interaction has been positive. We prove that existing opponent shaping methods implicitly perform Advantage Alignment. Compared to these methods, Advantage Alignment simplifies the mathematical formulation of opponent shaping, reduces the computational burden and extends to continuous action domains. We demonstrate the effectiveness of our algorithms across a range of social dilemmas, achieving state-of-the-art cooperation and robustness against exploitation.

• (Meulemans et al. 2024) is related to the previous one, and in fact recommended to me by Juan, one of the authors of the previous paper.

  Promising recent work has shown that in certain tasks cooperation can be established between “learning-aware” agents who model the learning dynamics of each other. Here, we present the first unbiased, higher-derivative-free policy gradient algorithm for learning-aware reinforcement learning, which takes into account that other agents are themselves learning through trial and error based on multiple noisy trials. We then leverage efficient sequence models to condition behaviour on long observation histories that contain traces of the learning dynamics of other agents. Training long-context policies with our algorithm leads to cooperative behaviour and high returns on standard social dilemmas, including a challenging environment where temporally-extended action coordination is required.

• Neural Interactive Proofs / SamAdamDay/neural-interactive-proofs: Experiments for the Neural Interactive Proofs paper (Hammond and Adam-Day 2025)

  In our new paper, we study how a trusted, weak model can learn to interact with one or more stronger but untrusted models in order to solve tasks beyond the weak model’s capabilities. We introduce several new interaction protocols and evaluate them both theoretically and empirically alongside a range of earlier proposals. To facilitate further research on creating and evaluating different protocols for scalable oversight, we also provide a comprehensive open-source codebase.

• Moral Alignment for LLM Agents (Tennant, Hailes, and Musolesi 2024)

  In this work, instead of relying on human feedback, we introduce the design of reward functions that explicitly and transparently encode core human values for Reinforcement Learning-based fine-tuning of foundation agent models. Specifically, we use intrinsic rewards for the moral alignment of LLM agents.

  We evaluate our approach using the traditional philosophical frameworks of Deontological Ethics and Utilitarianism, quantifying moral rewards for agents in terms of actions and consequences on the Iterated Prisoner’s Dilemma (IPD) environment.

  This paper was fun but actually I think their previous paper (Tennant, Hailes, and Musolesi 2023) was more elegant.

• Inverse Attention Agents for Multi-Agent Systems (Long et al. 2024)

  Unlike classical ToM research, which focuses on attributing mental states such as beliefs and desires, our model shifts to the crucial yet less-emphasized component of attention. Our methodology adopts a mentalistic approach, explicitly modelling the internal attentional state of agents using an attention recognition neural network that can be trained end-to-end in combination with components of [Multi-agent reinforcement Learning]. Contrary to traditional ToM modelling approaches that rely heavily on Bayesian inference to handle mental state transitions,[…] our method maintains the ontology of these states while focusing on the direct modelling of agents’ attentional mechanisms.

• Do as We Do, Not as You Think: the Conformity of Large Language Models (Weng, Chen, and Wang 2024)

• Strong Preferences Affect the Robustness of Preference Models and Value Alignment (Xu and Kankanhalli 2024)

• Competing Large Language Models in Multi-Agent Gaming Environments (Huang et al. 2024)

• Towards Empowerment Gain through Causal Structure Learning in Model-Based Reinforcement Learning ECL site (Cao et al. 2024) (straddling causal and multi-agent)

  In Model-Based Reinforcement Learning (MBRL), incorporating causal structures into dynamics models provides agents with a structured understanding of the environments, enabling efficient decision. Empowerment as an intrinsic motivation enhances the ability of agents to actively control their environments by maximizing the mutual information between future states and actions. We posit that empowerment coupled with causal understanding can improve controllability, while enhanced empowerment gain can further facilitate causal reasoning in MBRL. To improve learning efficiency and controllability, we propose a novel framework, Empowerment through Causal Learning (ECL), where an agent with the awareness of causal dynamics models achieves empowerment-driven exploration and optimizes its causal structure for task learning. Specifically, ECL operates by first training a causal dynamics model of the environment based on collected data. We then maximize empowerment under the causal structure for exploration, simultaneously using data gathered through exploration to update causal dynamics model to be more controllable than dense dynamics model without causal structure.

  If this takes off, the machines will come for us all. I am faintly relieved that it doesn’t work that well yet.

6.4 Causal learning

• Efficient and Trustworthy Causal Discovery with Latent Variables and Complex Relations XiuchuanLi/ICLR2025-ETCD (Li and Liu 2024)
• Unifying Causal Representation Learning with the Invariance Principle (Yao et al. 2024) (boring example, but tidy method)
• Identifiable Exchangeable Mechanisms for Causal Structure and Representation Learning
• Counterfactual Generative Modelling with Variational Causal Inference (Y. Wu, McConnell, and Iriondo 2024)
• Differentiable Causal Discovery for Latent Hierarchical Causal Models (Prashant et al. 2024)
• Selective induction Heads: How Transformers Select Causal Structures in Context (D’Angelo, Croce, and Flammarion 2024)
• Towards Empowerment Gain through Causal Structure Learning in Model-Based Reinforcement Learning ECL site (Cao et al. 2024)
• Causal Reasoning and Large Language Models: Opening a New Frontier for Causality (Kiciman et al. 2024)
• Causal Order: The Key to Leveraging Imperfect Experts in Causal Inference (Vashishtha et al. 2024)
• Neural Causal Graph for Interpretable and Intervenable Classification (Wang et al. 2024)
• Causal Discovery via Bayesian Optimization (Duong, Gupta, and Nguyen 2024)
• A Meta-Learning Approach to Bayesian Causal Discovery (Dhir et al. 2024)
• From Probability to Counterfactuals: the Increasing Complexity of Satisfiability in Pearls Causal Hierarchy (Dörfler et al. 2024)
• Causal Graph Transformer for Treatment Effect Estimation Under Unknown Interference anpwu/CauGramer (A. Wu et al. 2024)
• Standardizing Structural Causal Models (Ormaniec et al. 2024) werkaaa/iscm
• Systems with Switching Causal Relations: A Meta-Causal Perspective (Willig et al. 2024)

6.5 Useful diffusions

• How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Framework
• Discrete Diffusion Schrödinger Bridge Matching for Graph Transformation
• Denoising Levy Probabilistic Models
• Probing the Latent Hierarchical Structure of Data via Diffusion Models
• Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction
• Diffusion Bridge Implicit Models
• Learning Distributions of Complex Fluid Simulations with Diffusion Graph Networks
• Sequential Controlled Langevin Diffusions
• Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis
• Diffusion Models are Evolutionary Algorithms

6.6 Serendipitous weirdness

• Superdiff Skreta et al. (2024) https://github.com/necludov/super-diffusion

• Rusch and Rus (2024)

  We propose Linear Oscillatory State-Space models (LinOSS) for efficiently learning on long sequences. Inspired by cortical dynamics of biological neural networks, we base our proposed LinOSS model on a system of forced harmonic oscillators. A stable discretization, integrated over time using fast associative parallel scans, yields the proposed state-space model. We prove that LinOSS produces stable dynamics only requiring nonnegative diagonal state matrix. This is in stark contrast to many previous state-space models relying heavily on restrictive parameterizations. Moreover, we rigorously show that LinOSS is universal, i.e., it can approximate any continuous and causal operator mapping between time-varying functions, to desired accuracy.…