IEEE CSS Control Science Frontier Academic Salon--A Multi-discipline Approach to Advancing Knowledge Distillation Technology
Existing studies on knowledge distillation focus on teacher-centered methods, in which the teacher network is trained according to its own standards before transferring the learned knowledge to a student one. However, due to differences in network structure between teacher and student ones, the knowledge learned by the former may not be truly desired by the latter. Inspired by human educational wisdom, automatic control theory, and fuzzy logic concept, we propose a Student-Centered Distillation (SCD) method that enables the teacher network to adjust its knowledge transfer according to the student’s true needs. We implement it based on various human educational wisdom, e.g., the teacher network identifies and learns the knowledge desired by the student network on the validation set, and then transfers it to the latter through the training set. To address the problems of current deficiency knowledge, hard sample learning and knowledge forgetting faced by a student network in the learning process, we introduce and improve Proportional-Integral-Derivative (PID) algorithms from the field of automation to make them effective in identifying the current knowledge required by the student network. Furthermore, we propose a curriculum learning-based fuzzy logic strategy and apply it to the proposed PID control algorithm, such that the student network can actively pay attention to the learning of challenging samples. Experimental results show that SCD outperforms existing teacher-centered on knowledge distillation method in multiple computer vision tasks.
Date and Time
Location
Hosts
Registration
-
Add Event to Calendar
- Guangzhou, Guangdong
- China
- Contact Event Host
- Co-sponsored by Feiqi Deng, Xueyan Zhao
- Starts 20 August 2025 12:00 PM UTC
- Ends 22 August 2025 03:30 AM UTC
- No Admission Charge
Speakers
MengChu Zhou
A Multi-discipline Approach to Advancing Knowledge Distillation Technology
Existing studies on knowledge distillation focus on teacher-centered methods, in which the teacher network is trained according to its own standards before transferring the learned knowledge to a student one. However, due to differences in network structure between teacher and student ones, the knowledge learned by the former may not be truly desired by the latter. Inspired by human educational wisdom, automatic control theory, and fuzzy logic concept, we propose a Student-Centered Distillation (SCD) method that enables the teacher network to adjust its knowledge transfer according to the student’s true needs. We implement it based on various human educational wisdom, e.g., the teacher network identifies and learns the knowledge desired by the student network on the validation set, and then transfers it to the latter through the training set. To address the problems of current deficiency knowledge, hard sample learning and knowledge forgetting faced by a student network in the learning process, we introduce and improve Proportional-Integral-Derivative (PID) algorithms from the field of automation to make them effective in identifying the current knowledge required by the student network. Furthermore, we propose a curriculum learning-based fuzzy logic strategy and apply it to the proposed PID control algorithm, such that the student network can actively pay attention to the learning of challenging samples. Experimental results show that SCD outperforms existing teacher-centered on knowledge distillation method in multiple computer vision tasks.
Biography:
MengChu Zhou received his B.S. degree in Control Engineering from Nanjing University of Science and Technology, Nanjing, China in 1983, M.S. degree in Automatic Control from Beijing Institute of Technology, Beijing, China in 1986, and Ph. D. degree in Computer and Systems Engineering from Rensselaer Polytechnic Institute, Troy, NY in 1990. He joined the Department of Electrical and Computer Engineering, New Jersey Institute of Technology in 1990, and is now a Distinguished Professor. His interests are in intelligent automation, robotics, Petri nets, Internet of Things, edge/cloud computing, AI, and big data analytics. He has over 1400 publications including 17 books, over 900 journal papers including over 700 IEEE Transactions papers, 31 patents and 32 book-chapters. He is a recipient of Excellence in Research Prize and Medal from NJIT, Humboldt Research Award for US Senior Scientists from Alexander von Humboldt Foundation, and Franklin V. Taylor Memorial Award and the Norbert Wiener Award from IEEE Systems, Man, and Cybernetics Society, Computer-Integrated Manufacturing UNIVERSITY-LEAD Award from Society of Manufacturing Engineers, and Distinguished Service Award from IEEE Robotics and Automation Society. He is a Highly-cited Scholar with over 81,800 GoogleScholar citations and h-index being 144. He is Fellow of IEEE, International Federation of Automatic Control (IFAC), American Association for the Advancement of Science (AAAS), Chinese Association of Automation (CAA) and National Academy of Inventors (NAI).