Dr Anjaly Parayil of Microsoft Research

Reinforcement learning is a framework for interactive decision-making with incentives sequentially revealed across time without a system model. Here I focus on policy search in continuous action space using stochastic policy gradient (PG) updates.  In contrast to discrete action space, the non-convexity in continuous space poses a pathological challenge due to its limited convergence guarantees.

Therefore, in this talk, I will focus on our recent research efforts at the US Army Laboratory to address this gap. Inspired by persistent exploration and its tendency to avoid spurious behavior, we step towards persistent exploration in continuous space through policy parameterizations defined by distributions of heavier tails. Doing so invalidates smoothness conditions of the score function common to PG. Thus, the research establishes convergence rate to stationarity with the policy’s tail index and an exploration tolerance parameter introduced here for the first time. Further, we characterize the dependence of the set of local maxima on the tail index through metastability analysis of a suitably defined Markov chain, which identifies the convergence of heavy-tailed policies to wider peaks. This phenomenon yields improved stability to perturbations in supervised learning, which manifests in the improved performance of policy search, particularly when myopic and farsighted incentives are misaligned. The last part of the talk discusses real-world applications and future research directions.

Biography:

Anjaly Parayil is a researcher at Microsoft Research India. Prior to joining Microsoft, she was a postdoctoral research fellow with the US Army Research Laboratory in the Computational and Information Sciences Directorate Adelphi from September 2019 to August 2021. Her research interests include Multi-agent Control and Coordination, Reinforcement Learning, Bifurcation Theory, Optimization, State Estimation, Machine Learning Applied to Large-scale Datasets, and Bayesian Approaches. She received her Ph.D. in Aerospace Engineering from the Indian Institute of Science (IISc), India, in 2020.

Email:

Address:Microsoft Research, , Bangalore, Karnataka, India