IEEE PES & WIE AG Schenectady Chapter & IEEE RPI Student Branch (Virtual) Lecture on “Memory and Energy Efficient Inference of Large Mixture-of-Experts Foundation Models”

#WIE # #MoE #foundation #models #Machine #learning

Title: Memory and Energy Efficient Inference of Large Mixture-of-Experts Foundation Models

Abstract: 

Large Mixture-of-Experts (MoE) foundation models improve scalability by activating only a sparse subset of parameters for each input, but inference remains bottlenecked by memory movement, hardware inefficiency, and energy consumption. This talk presents two complementary, theory-guided approaches for efficient MoE inference across digital and analog hardware platforms. First, I will introduce an expert-wise mixed-precision quantization framework that identifies critical experts through router training dynamics and allocates quantization bit-widths accordingly, achieving improved low-bit inference accuracy and lower memory cost on large models such as Switch Transformer and Mixtral. Second, I will discuss a theory-guided analog–digital mapping strategy for analog in-memory computing (AIMC), where a proposed metric identifies noise-sensitive experts and dense modules for selective deployment on digital hardware, while the remaining operations are executed on energy-efficient analog units. Experiments on models including DeepSeek MoE and OLMoE demonstrate near full-precision accuracy with substantially reduced memory movement and energy consumption.