"Path towards better deep learning models and implications for hardware"

#Deep #Learning

ABSTRACT: Advances in Deep Learning over the past several years have demonstrated two paths to better models: scale and algorithmic innovation. Brute-force scaling of model parameter count increases model capacity, and when presented with enough training data, has shown better performance in many domains. But it requires more compute than can be delivered by a single traditional processor – clusters of 10s to even 1000s of general purpose processors are commonly used today for neural network training. This approach to scaling is not sustainable. We need algorithmic innovations to find more efficient neural network architectures and training methods. This requires more flexible hardware to develop and test novel approaches. In this presentation, we will look at the trends of recent deep learning models, discuss implications for hardware, and share how the Cerebras CS-1 addresses these requirements for both scale and flexibility of compute.