Filippos Christianos

I am in the final stages of my PhD in the CDT for Robotics and Autonomous Agents, advised by Stefano Albrecht (University of Edinburgh) and a member of the Autonomous Agents Research Group.

My PhD research is in the area of Multi-Agent Deep Reinforcement Learning. In particular, I study how multiple agents can efficiently explore and learn in environments with sparse rewards.

I am the author and maintainer of the Multi-Robot Warehouse environment for multi-agent RL research. I also developed and maintain the Python version of Level-based Foraging. Our group has been using both environments to develop new and exciting algorithms for MARL. I am the first author of two such algorithms: Shared Experience Actor-Critic (SEAC), and Selective Parameter Sharing (SePS) that have been published in NeurIPS (2020) and ICML (2021) respectively.

Finally, I am a co-author of E-PyMARL, a library for MARL which has been widely used by the community.

Keywords: Machine Learning, Deep Reinforcement Learning (RL), Multi-agent Systems, Exploration in RL.

news

Jan 18, 2023	My NVIDIA internship resulted in “Planning with Occluded Traffic Agents using Bi-Level Variational Occlusion Models” which was just accepted in ICRA 2023!
Oct 29, 2022	Preprints of my two new papers are online! My NVIDIA internship paper on AV occlusions and Pareto Actor-Critic: a new algorithm for MARL.
Jun 23, 2022	I joined NVIDIA Research for a three month internship on autonomous vehicles!
Dec 20, 2021	Our paper titled “Decoupling Exploitation and Intrinsically-Motivated Exploration in Reinforcement Learning” has been accepted in AAMAS 2022!
Sep 27, 2021	Another paper accepted at NeurIPS 2021: Agent Modelling under Partial Observability for Deep Reinforcement Learning.
Jul 29, 2021	Our benchmarking paper for MARL has been accepted at NeurIPS 2021: Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks.
May 10, 2021	Two new papers accepted at ICML 2021: Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing and Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning.
Dec 20, 2020	My blog post on two new environments for MARL has just been posted in our groups webpage.
Dec 8, 2020	Our paper, Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning, has been accepted and published in Neural Information Processing Systems (NeurIPS 2020).

selected publications

NeurIPS

Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning

Christianos Filippos, Schäfer Lukas, and Albrecht Stefano

In Advances in Neural Information Processing Systems, 2020

Abs Paper Code Poster Slides

Exploration in multi-agent reinforcement learning is a challenging problem, especially in environments with sparse rewards. We propose a general method for efficient exploration by sharing experience amongst agents. Our proposed algorithm, called shared Experience Actor-Critic(SEAC), applies experience sharing in an actor-critic framework by combining the gradients of different agents. We evaluate SEAC in a collection of sparse-reward multi-agent environments and find that it consistently outperforms several baselines and state-of-the-art algorithms by learning in fewer steps and converging to higher returns. In some harder environments, experience sharing makes the difference between learning to solve the task and not learning at all.
ICML

Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing

Christianos Filippos, Papoudakis Georgios, Rahman Arrasy, and Albrecht Stefano

In Proceedings of the 38th International Conference on Machine Learning, 2021

Abs Paper Code Poster Slides

Sharing parameters in multi-agent deep reinforcement learning has played an essential role in allowing algorithms to scale to a large number of agents. Parameter sharing between agents significantly decreases the number of trainable parameters, shortening training times to tractable levels, and has been linked to more efficient learning. However, having all agents share the same parameters can also have a detrimental effect on learning. We demonstrate the impact of parameter sharing methods on training speed and converged returns, establishing that when applied indiscriminately, their effectiveness is highly dependent on the environment. We propose a novel method to automatically identify agents which may benefit from sharing parameters by partitioning them based on their abilities and goals. Our approach combines the increased sample efficiency of parameter sharing with the representational capacity of multiple independent networks to reduce training time and increase final returns.
NeurIPS

Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks

Papoudakis Georgios *, Christianos Filippos *, Schäfer Lukas, and Albrecht Stefano

In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021

Abs arXiv OpenReview Code

Multi-agent deep reinforcement learning (MARL) suffers from a lack of commonly-used evaluation tasks and criteria, making comparisons between approaches difficult. In this work, we evaluate and compare three different classes of MARL algorithms (independent learners, centralised training with decentralised execution, and value decomposition) in a diverse range of multi-agent learning tasks. Our results show that (1) algorithm performance depends strongly on environment properties and no algorithm learns efficiently across all learning tasks; (2) independent learners often achieve equal or better performance than more complex algorithms; (3) tested algorithms struggle to solve multi-agent tasks with sparse rewards. We report detailed empirical data, including a reliability analysis, and provide insights into the limitations of the tested algorithms.
ICRA

Planning with Occluded Traffic Agents using Bi-Level Variational Occlusion Models

Christianos Filippos, Karkus Peter, Ivanovic Boris, Albrecht Stefano, and Pavone Marco

In IEEE International Conference on Robotics and Automation, 2023

Abs arXiv

Reasoning with occluded traffic agents is a significant open challenge for planning for autonomous vehicles. Recent deep learning models have shown impressive results for predicting occluded agents based on the behaviour of nearby visible agents; however, as we show in experiments, these models are difficult to integrate into downstream planning. To this end, we propose Bi-level Variational Occlusion Models (BiVO), a two-step generative model that first predicts likely locations of occluded agents, and then generates likely trajectories for the occluded agents. In contrast to existing methods, BiVO outputs a trajectory distribution which can then be sampled from and integrated into standard downstream planning. We evaluate the method in closed-loop replay simulation using the real-world nuScenes dataset. Our results suggest that BiVO can successfully learn to predict occluded agent trajectories, and these predictions lead to better subsequent motion plans in critical scenarios.