Topic: Modeling self-attention in transformers as a multiagent dynamical system: equilibria, stability, and asymptotic behavior.
Abstract: Transformer architectures have rapidly become the standard algorithm for many problems in deep learning (e.g., large language models or image classification). At the core of a transformer lies a so-called self-attention mechanism. In this presentation Prof. Altafini will show how to model such self-attention mechanism as a continuous-time multiagent-like dynamical system model living on a sphere. Such dynamics is related to a multiagent version of the Oja flow, a dynamical system that computes the principal eigenvector of a matrix, corresponding for transformers to the value matrix. In the “single-head” time-invariant case, the equilibria of a self-attention dynamics can be classified into four classes: consensus, bipartite consensus, clustering and polygonal equilibria. For this simplified dynamics, multiple asymptotically stable equilibria from the first three classes often coexist. Interestingly, equilibria from the first two classes are always aligned with the eigenvectors of the value matrix, often but not exclusively with the principal eigenvector.