Transformer networks

1 Introductions

So many.

TODO: rank in terms of lay-person-friendliness.

• 3b1b: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning - YouTube
• [2207.09238] Formal Algorithms for Transformers
• Compact precise definition of a transformer function – foreXiv
• Understanding einsum for Deep learning: implement a transformer with multi-head self-attention from scratch | AI Summer
• Jay Alammar’s Illustrated Transformer
• Lilian Weng, Large Transformer Model Inference Optimization
• Lilian Weng, The Transformer Family Version 2.0
• The AlphaFold2 Method Paper: A Fount of Good Ideas
• Xavier Amatriain, Transformer models: an introduction and catalog — 2023 Edition
• nostalgebraist, An exciting new paper on neural language models
• This guide to pruning multihead attention NN should probably go somewhere useful if I actually end up doing NLP like all the recruiters seem to want.
• 1A - Scaled Dot Product Attention explained
• John Thickstun, The Transformer Model in Equations
• Large language models, explained with a minimum of math and jargon
• [1hr Talk] Intro to Large Language Models
• Jeremy Howard on X: Here, in full directly on Twitter, is “A Hackers’ Guide to Language Models”. This 90 minute tutorial is designed to be the one place I point coders at when they ask “hey, tell me everything I need to know about LLMs!” It covers both OpenAI models and open source ones in depth.
• Large language models, explained with a minimum of math and jargon
• A good paper read is Yannic Kilcher’s.

2 Power of

Transformers are pretty good at weird stuff, e.g. automata — see Unveiling Transformers with LEGO (Zhang et al. 2022).

How about Bayesian inference? (Müller et al. 2022)

Can they be an engine of intelligence? What do they do in society? etc. Controversial — see the Stochastic Parrots paper (Bender et al. 2021), and the entire internet commentariat from November 2022 onwards.

3 As set functions

Transformers are neural set functions (!).

4 RWKV

State-space, i.e. recurrent transformers, without (classic) attention. Suggestive connection to S4 models.

RWKV is inspired by Apple’s Attention Free Transformer.(@ Zhai et al. 2021). …

How to combine the best of transformers and RNNs? The main drawback of transformer-based models is that it can become challenging to run a model with a context window that is larger than a certain value, as the attention scores are computed simultaneously for the entire sequence.

RNNs natively support very long context lengths—only limited by the context length seen in training, but this can be extended to millions of tokens with careful coding. Currently, there are RWKV models trained on a context length of 8192 (ctx8192) and they are as fast as ctx1024 models and require the same amount of RAM.

The major drawbacks of traditional RNN models and how RWKV is different:

1. Traditional RNN models are unable to utilize very long contexts (LSTM can only manage ~100 tokens when used as a LM). However, RWKV can utilize thousands of tokens and beyond…
2. Traditional RNN models cannot be parallelized when training. RWKV is similar to a “linearized GPT” and it trains faster than GPT.

By combining both advantages into a single architecture, the hope is that RWKV can grow to become more than the sum of its parts.

• Introducing RWKV - An RNN with the advantages of a transformer
• How the RWKV language model works
• The RWKV language model: An RNN with the advantages of a transformer
• BlinkDL/ChatRWKV: ChatRWKV is like ChatGPT but powered by RWKV (100% RNN) language model, and open source.

5 Tokens as recurrent state

See also RecurrentGPT (Zhou et al. 2023)

GitHub - aiwaves-cn/RecurrentGPT

RecurrentGPT replaces the vectorized elements (i.e., cell state, hidden state, input, and output) in a Long-short Term Memory RNN (LSTM) with natural language (i.e., paragraphs of texts), and simulates the recurrence mechanism with prompt engineering.

At each timestep t, RecurrentGPT receives a paragraph of text and a brief plan of the next paragraph, which are both generated in step t − 1. It then attends to the long-term memory, which contains the summaries of all previously generated paragraphs and can be stored on hard drives, and relevant paragraphs can be retrieved with semantic search.

RecurrentGPT also maintains a short-term memory that summarizes key information within recent timesteps in natural language and is updated at each time step. RecurrentGPT combines all aforementioned inputs in a prompt and asks the backbone LLM to generate a new paragraph, a short plan for the next paragraph, and updates the long-short term memory by rewriting the short-term memory and appending the summary of the output paragraph to the long-term memory.

6 Practicalities

Democratizing the hardware side of large language models seems to be an advertisement for some new hardware, but there is interesting background in there.

HuggingFace distributes and documents and implements a lot of Transformer/attention NLP models and seem to be the most active neural NLP project. Certainly too active to explain what they are up to in between pumping out all the code.

7 Embedding vector databases

See embedding vector databases.

• LMQL: Programming Large Language Models: “LMQL is a programming language for language model interaction.” (Beurer-Kellner, Fischer, and Vechev 2022)

  LMQL generalizes natural language prompting, making it more expressive while remaining accessible. For this, LMQL builds on top of Python, allowing users to express natural language prompts that also contain code. The resulting queries can be directly executed on language models like OpenAI’s GPT models. Fixed answer templates and intermediate instructions allow the user to steer the LLM’s reasoning process.

8 References