35K views · Jul 5, 2023 · 39:43 · 3 HN recommenders · 3 mentions · discussed 2024
Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.
Why Hacker News points people here: HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.
“I highly recommend Sasha Rush's video on it”HN comment, 2024
“I highly recommend Sasha Rush's video on it”
Explains tradeoffs between transformer and state space model layers.
To those curious about the tradeoffs between transformer and state space model layers, I highly recommend Sasha Rush's video on it: this video
“Would recommend Sasha Rush's lecture”
Gives intuition for state space models via linear RNNs.
Would recommend Sasha Rush's lecture this video , it provides the intuition for state space models via linear RNNs
“Here is a video going through this topic as well”
Author points to a video walkthrough of the S4 topic.
Hi! Blog author. This was an attempt a couple years ago to understand and write about this paper in a detailed way. Here is a video going through this topic as well: this video Section 2 of the blog post is no longer very relevant. A lot of advances (DSS, S4D) simplified that part of the process. Arguably also this all should be updated for Mamba (same authors).
