publications

2024

1. AAAI

   Pure-Past Action Masking

   Giovanni Varricchione, Natasha Alechina , Mehdi Dastani , and 3 more authors

   In Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI 2024) , 2024

   Abs PDF Poster

   We present Pure-Past Action Masking (PPAM), a lightweight aproach to action masking for safe reinforcement learning. In PPAM, actions are disallowed (“masked”) according to specifications expressed in Pure-Past Linear Temporal Logic (PPLTL). PPAM can enforce non-Markovian constraints, i.e., constraints based on the history of the system, rather than just the current state of the (possibly hidden) MDP. The features used in the safety constraint need not be the same as those used by the learning agent, allowing a clear separation of concerns between the safety constraints and reward specifications of the (learning) agent. We prove formally that an agent trained with PPAM can learn any optimal policy that satisfies the safety constraints, and that they are as expressive as shields, another approach to enforce non-Markovian constraints in RL. Finally, we provide empirical results showing how PPAM can guarantee constraint satisfaction in practice.

2023

1. EUMAS

   Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning

   Giovanni Varricchione, Natasha Alechina , Mehdi Dastani , and 1 more author

   In Proceedings of the 20th European Conference on Multi-Agent Systems (EUMAS 2023) , 2023

   Abs PDF Poster

   Reward machines have recently been proposed as a means of encoding team tasks in cooperative multi-agent reinforcement learning. The resulting multi-agent reward machine is then decomposed into individual reward machines, one for each member of the team, allowing agents to learn in a decentralised manner while still achieving the team task. However, current work assumes the multi-agent reward machine to be given. In this paper, we show how reward machines for team tasks can be synthesised automatically from an Alternating-Time Temporal Logic specification of the desired team behaviour and a high-level abstraction of the agents’ environment. We present results suggesting that our automated approach has comparable, if not better, sample efficiency than reward machines generated by hand for multi-agent tasks.