Plate VI. The Loom of Attention, 2017–2022
The Loom of Attention
One architecture learns to weigh every word against every other.
The 2017 transformer replaced recurrence with attention, and it scaled. BERT read, GPT-2 and GPT-3 wrote, and scaling laws suggested that bigger would keep being better.
The same machinery folded proteins (AlphaFold 2) and turned text into images (DALL·E, Stable Diffusion). The loom was ready to weave conversation.
Events
- Attention Is All You Need (12 Jun 2017)
- BERT (11 Oct 2018)
- GPT-2 and staged release (14 Feb 2019)
- GPT-3: few-shot learners (28 May 2020)
- AlphaFold 2 at CASP14 (30 Nov 2020)
- Anthropic is founded (2021)
- DALL·E and CLIP (5 Jan 2021)
- “Stochastic parrots” (Mar 2021)
- An engineer says LaMDA is sentient (11 Jun 2022)
- Stable Diffusion released openly (22 Aug 2022)