IEEE PES Schenectady Chapter (Virtual) Lecture on Multiagent Reinforcement Learning under Nonstationarity

# #Multiagent #Reinforcement #Learning

In this presentation, we summarize our recent work on multi-agent reinforcement learning (MARL) under nonstationarity, motivated by the fact that practical multi-agent systems are intrinsically nonstationary because multiple agents learn and adapt concurrently. We frame three fundamental challenges that hinder reliable deployment: (i) learning effective policies under continually changing peer behaviors, (ii) reasoning about the long-run evolution of other agents’ learning dynamics rather than only short-horizon updates, and (iii) providing robustness guarantees under perturbations arising from both the environment and peer learning agents. We then present three technical thrusts to address the respective challenges. First, we introduce meta multi-agent reinforcement learning, in which MARL is modeled as a Markov chain over joint policies; this perspective enables a meta policy-gradient method that accounts not only for each agent’s self-learning gradient but also an explicit peer-learning gradient, thereby shaping how other agents adapt in the future. Second, we address long-term influence in MARL by moving beyond few-step opponent modeling and targeting limiting interaction outcomes as learning proceeds, which is essential when other agents continue to adapt indefinitely. Third, we present ROMAX, a framework for certifiably robust deep MARL via convex relaxation, designed to improve resilience against nonstationary adversaries and cyber-physical perturbations by incorporating formal structure into robustness-oriented objectives. Across representative matrix games and coupled multi-agent control benchmarks, the presented methods demonstrate improved performance, adaptation, and robustness relative to standard MARL baselines.