Skip to yearly menu bar Skip to main content


Poster

State-Free Inference of State-Space Models: The *Transfer Function* Approach

Rom N. Parnichkun · Stefano Massaroli · Alessandro Moro · Michael Poli · Jimmy Smith · Ramin Hasani · Mathias Lechner · Qi An · Christopher Re · Hajime Asama · Stefano Ermon · Taiji Suzuki · Atsushi Yamashita


Abstract:

We approach designing a state-space model (SSM) through its dual representation, the transfer function, and uncover a highly efficient sequence parallel inference algorithm that is state-free, wherein unlike other proposed algorithms, an increase in state size does not incur any significant memory or computational cost.Our experimental results across multiple sequence lengths and state-sizes illustrates, on average, a 35% training speed improvement over the S4 algorithm on the Long Range Arena benchmark, while delivering state-of-the-art downstream performances over other attention-free approaches. Moreover, we report improved performances in WikiText103 perplexity over a long convolutional Hyena baseline, by simply replacing its filter with our SSM.

Live content is unavailable. Log in and register to view live content