Parallelizing “Inherently Sequential” Processes

Abstract: Stateful models like recurrent neural networks (RNNs) and Markov chain Monte Carlo (MCMC) were believed to be “inherently sequential,” and so could not use massively parallel hardware like GPUs to accelerate computation over the sequence length. Recent work showed stateful models can be parallelized by recasting sequential evaluation as solving a high-dimensional equation with Newton’s method, but it was unclear how to scale the method or where it could be used. In this talk, I will introduce a new method that scales parallel evaluation of “inherently sequential” models; provide provable guidance on when to use such methods; and discuss the impact of these contributions to natural and artificial intelligence.