UniADS: Universal Architecture-Distiller Search for Distillation Gap

XIAOYU SEAN LU, Ph.D. & Lecturer

School of Cyber Science and Engineering, Nanjing University of Science and Technology, Nanjing, China

Time: 9:30pm, Dec. 19                                  

Place: ECE 202, NJIT, Newark, NJ

https://njit.webex.com/join/zhou

Abstract

In this talk, we present our proposed method called UniADS. It is the first Universal Architecture-Distiller Search framework for co-optimizing student architecture and distillation policies. Teacher-student distillation gap limits the distillation gains. Previous approaches seek to discover the ideal student architecture while ignoring distillation settings. In UniADS, we construct a comprehensive search space encompassing an architectural search for student models, knowledge transformations in distillation strategies, distance functions, loss weights, and other vital settings. To efficiently explore the search space, we utilize the NSGA-II genetic algorithm for better crossover and mutation configurations and employ the Successive Halving algorithm for search space pruning, resulting in improved search efficiency and promising results. Extensive experiments are performed on different teacher-student pairs using CIFAR-100 and ImageNet datasets. The experimental results consistently demonstrate the superiority of our method over existing approaches. Furthermore, we provide a detailed analysis of the search results, examining the impact of each variable and extracting valuable insights and practical guidance for distillation design and implementation.

Bio-Sketch

XIAOYU SEAN LU received the B.S. degree from the Nanjing University of Technology, Nanjing, China, in 2011, and the M.S. and Ph.D. degrees from the New Jersey Institute of Technology, Newark, NJ, USA, in 2015 and 2019, respectively. He was a Research Scholar with the Department of Electrical and Computer Engineering, Stevens Institute of Technology, Hoboken, NJ, USA. He joined the School of Cyber Science and Engineering, Nanjing University of Science and Technology, Nanjing, China in 2022. He has published more than 20 articles in journals and conference proceedings, including the IEEE TRANSACTIONS ON SYSTEM, MAN AND CYBERNETICS: SYSTEMS, the IEEE/CAA JOURNAL OF AUTOMATICA SINICA, and the IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS. His current research interests include social media data analysis, cyberbullying detection, knowledge distillation, data mining, deep learning, and their applications in industry.