The Transformer and Everything It Became

In June 2017, eight researchers at Google published a paper with a title so understated it bordered on arrogant: "Attention Is All You Need." The paper proposed a new neural network architecture called the Transformer. It had no recurrence. No convolution. Just attention: a mechanism that lets every element in a sequence look at every other element simultaneously and decide what matters. The paper was about machine translation. It beat everything that came before it. And then, over the next eight years, it ate the entire field of artificial intelligence.

Today, the Transformer is the architectural backbone of virtually every frontier AI system: GPT-4, Claude, Gemini, LLaMA, DALL-E, Stable Diffusion, Sora. It powers language models that write code, vision models that interpret medical scans, and multimodal systems that reason across text, images, and video simultaneously. As of 2025, the original paper has been cited over 173,000 times, making it one of the most referenced scientific papers of the twenty-first century.¹

But here's what I've noticed in conversation after conversation with engineers, executives, and researchers: most people use these systems daily without understanding the architectural genealogy that connects them. They treat "transformer," "large language model," and "vision-language model" as interchangeable buzzwords. They're not. They're layers in an evolutionary stack, each built on the one below, each introducing new capabilities and new failure modes. Understanding that stack isn't optional if you're building with these systems. It's the difference between using a tool and understanding what it can and cannot do.

The stack, from the bottom up.

The General-Purpose Sequence Machine

Before 2017, if you wanted a neural network to process sequences (sentences, time series, audio), you used recurrent neural networks. RNNs processed tokens one at a time, left to right, passing a hidden state forward like a baton in a relay race. This worked, but it was slow (inherently sequential, so you couldn't parallelise it on GPUs) and fragile (information decayed over long distances, no matter how clever your gating mechanisms were). LSTMs helped with the decay problem but couldn't fix the parallelisation problem.

The Transformer replaced all of this with a single mechanism: self-attention. In self-attention, every token in a sequence computes a relevance score against every other token, simultaneously. There's no sequential processing. There's no hidden state being passed forward. Instead, the model builds a complete map of relationships across the entire input in one pass. The mathematical machinery is elegant: each token is projected into three vectors (a query, a key, and a value), and the output is a weighted combination of all values, where the weights are determined by how well each query matches each key.

The original Transformer used an encoder-decoder structure: the encoder reads the input (a sentence in English), the decoder generates the output (a sentence in German). But the key insight was that neither the encoder nor the decoder needed recurrence. Self-attention plus feedforward layers, stacked deep, was enough. And because self-attention processes all tokens in parallel, training could scale across thousands of GPUs in a way that RNNs never could.

This is the property that made the Transformer not just a better architecture for translation, but a general-purpose sequence processing machine. Anything that can be represented as a sequence of tokens (words, image patches, audio frames, amino acids, chess moves) can be fed into a Transformer. The architecture doesn't care about the modality. It only cares about the relationships between tokens.

There's a cost, though. Self-attention is quadratic in the sequence length: every token attends to every other token, so the computational cost grows as O(n²) where n is the number of tokens. For a 512-token sequence, this is manageable. For a million-token context window, it becomes a serious engineering challenge. This quadratic bottleneck is the Transformer's original sin, and much of the research since 2017 (FlashAttention, sparse attention, linear attention, state-space models like Mamba) has been about trying to mitigate it without losing the expressiveness that makes self-attention so powerful.

The Decoder That Ate the Internet

The Transformer paper described an encoder-decoder architecture. But researchers quickly discovered that you didn't always need both halves. For language generation (predicting the next word in a sequence), you only need the decoder. Strip away the encoder, train a decoder-only Transformer on massive amounts of text, and you get what we now call a large language model.

The training objective is deceptively simple: given all the words so far, predict the next one. That's it. There's no explicit instruction to learn grammar, facts, reasoning, or code. The model simply learns to predict the next token, over and over, on trillions of tokens scraped from the internet. And yet, somewhere in the process of getting extraordinarily good at next-token prediction, LLMs develop emergent capabilities: abilities that were never explicitly trained, arriving unannounced at scale (the definitional story belongs to another essay). What matters architecturally is where they come from: the interaction of scale, data diversity, and the expressiveness of the Transformer's self-attention mechanism.

The Three Convergences

Three things converge. First, self-attention gives LLMs the ability to model long-range dependencies in text: understanding that a pronoun in sentence twelve refers to a character introduced in sentence two. Second, the sheer scale of training data means the model internalises an enormous statistical map of human knowledge, language patterns, and reasoning strategies. Third, the Transformer's parallelism makes it possible to train these massive models at all, something that was computationally infeasible with RNNs.

Where They Struggle

The failure modes are as fundamental as the strengths. LLMs hallucinate: they generate fluent, confident text that is factually wrong. Formal learning theory has shown that hallucination is not a bug to be patched but an inevitable consequence of using finite models trained on incomplete data to approximate an infinite world.² No model can learn all computable functions; therefore, every model will produce errors on some inputs. The question isn't whether LLMs hallucinate, but whether you can build systems that know when they're about to.

LLMs also struggle with genuine reasoning. They are superb at pattern completion (recognising structures they've seen in training data and extrapolating), but they stumble on problems that require novel logical inference, especially when the problem structure differs from anything in the training distribution.

And then there's the context window problem. LLMs advertise context windows of 128K or even a million tokens, but effective context utilisation is strictly sublinear in window size.³ The result is that a model with a 128K context window may effectively utilise only 64K of it, with performance degrading sharply for information placed in the middle of long documents: the "lost in the middle" phenomenon.⁴

Giving the Language Model Eyes

Language is powerful, but the world is visual. Most of the information humans use to navigate reality comes through our eyes, not our ears. If LLMs can only process text, they're operating with one hand tied behind their back. Vision-language models untie that hand.

The journey began with the Vision Transformer (ViT), introduced in 2020 by Dosovitskiy and colleagues at Google in a paper titled "An Image is Worth 16x16 Words." The insight was elegantly simple: take an image, split it into a grid of fixed-size patches (say, 16×16 pixels each), flatten each patch into a vector, add positional embeddings, and feed the resulting sequence into a standard Transformer encoder. No convolutional layers. No image-specific inductive biases. Just patches treated as tokens: images as sequences of visual words.

But ViT alone only processed images. The breakthrough into true vision-language modelling came with CLIP (Contrastive Language-Image Pre-training), released by OpenAI in 2021. CLIP trains two encoders simultaneously (one for images, one for text) and learns to align their representations by pulling matching image-text pairs close together and pushing non-matching pairs apart in a shared embedding space.

The current generation of VLMs (LLaVA, Qwen2.5-VL, PaliGemma, GPT-4V, Claude's vision capabilities) go further. They combine a visual encoder (usually a ViT variant like CLIP ViT-L) with a large language model through some form of cross-modal adapter: a projection layer, an MLP bridge, or a querying transformer (Q-Former) that translates visual representations into the token space the LLM expects.

Where VLMs Struggle

The fundamental tension is what researchers call textual dominance: the LLM backbone was pre-trained on orders of magnitude more text than the visual encoder has seen images, so gradients during multimodal training flow overwhelmingly toward text representations. Visual features get compressed, quantised, and sometimes effectively ignored. This produces a model that can describe an image eloquently but miscount the objects in it, or hallucinate details that "should" be there based on textual priors rather than what's actually in the pixels.

The Evolutionary Stack

Think of these three concepts as layers in an architectural genealogy, each building on the one below.

The Transformer is the foundational architecture: the general-purpose sequence processing engine built on self-attention. It is modality-agnostic: it doesn't know or care whether its tokens represent words, image patches, audio frames, or protein sequences. It's the computational substrate.

The Large Language Model is a Transformer (specifically, a decoder-only Transformer) that has been trained on massive text corpora to predict the next token. It inherits all the Transformer's strengths (parallelism, long-range dependency modelling, scalability) and adds emergent capabilities that arise from the interaction of scale and data diversity.

The Vision-Language Model is an LLM augmented with visual perception. It adds a visual encoder (itself a Transformer, typically a ViT) and a cross-modal adapter that bridges the gap between visual and linguistic representations. A VLM doesn't replace the LLM; it extends it, giving it eyes.

This layered relationship matters for practical system design. Every limitation of the Transformer propagates upward: the quadratic attention cost constrains LLM context windows, which in turn constrains how many visual tokens a VLM can process. Every LLM failure mode (hallucination, pattern fixation, context degradation) is inherited by VLMs and often amplified by the additional complexity of cross-modal reasoning.

The Deeper Tensions

Four tensions rarely surface in popular explanations, and each shapes what these systems can and cannot do.

The creativity-factuality trade-off is mathematically fundamental. The same mechanism that allows LLMs to produce creative, novel, engaging text (sampling from a probability distribution over possible next tokens) is the same mechanism that produces hallucinations. Higher temperature (more creative sampling) necessarily means higher hallucination rates.

The homogenisation risk is accelerating. Because Transformers, LLMs, and VLMs are so expensive to train from scratch, the entire ecosystem increasingly converges on a handful of base models: as of 2025, a small number of model families (GPT, Claude, LLaMA, Gemini, Qwen) underpin thousands of downstream applications across healthcare, finance, law, and education. Why that concentration is the central design tension of this era is an argument of its own.

Multimodal alignment remains shallow. Current VLMs don't truly "understand" images the way they process language. The visual encoder retains co-occurrence statistics from its contrastive pre-training, not genuine perceptual structure. The result is that VLMs can describe what's in an image but often fail at tasks requiring precise spatial reasoning, counting, or distinguishing fine-grained visual details.

The Transformer itself may not be the final architecture. State-space models like Mamba, hybrid architectures like Jamba, and linear attention variants like RWKV are achieving competitive performance with subquadratic complexity. The question isn't whether the Transformer will be replaced; it's whether the replacement will be a clean break or a gradual hybridisation.

What Comes Next

We stand at a peculiar moment in the history of artificial intelligence. A single architectural idea, self-attention over sequences of tokens, has proven so general that it now powers text generation, image understanding, video synthesis, protein folding, robotic planning, and scientific reasoning. The Transformer didn't just solve translation. It became the universal substrate for learned computation.

But universality comes with fragility. The same architecture that enables LLMs to write poetry also causes them to hallucinate case law. The same training paradigm that gives VLMs the ability to describe surgical scenes causes them to confuse left and right. The same scaling laws that produce emergent capabilities also produce emergent failure modes that we don't fully understand and can't reliably predict.

The questions that will define the next decade of AI are sharpening into focus. Can we build architectures that preserve the Transformer's expressiveness while breaking free of its quadratic cost? Can we design multimodal systems where visual understanding is native rather than bolted on? Can we move beyond next-token prediction as the dominant training paradigm? Can we build systems that know what they don't know?

And perhaps the most consequential question of all: as Vision-Language-Action models begin to bridge the gap from perception to physical interaction (from seeing the world to acting in it), will the architectural patterns that worked for text and images transfer to embodied intelligence? Or will the messiness of the physical world demand something fundamentally new?

The Transformer gave us a universal language for learned computation. What we build on that foundation, and whether the foundation itself needs to be rebuilt, is the open question that will shape the trajectory of artificial intelligence for the next generation.

Key references:

Vaswani et al. (2017). "Attention Is All You Need." NeurIPS. arXiv:1706.03762
Dosovitskiy et al. (2020). "An Image is Worth 16x16 Words." ICLR 2021. arXiv:2010.11929
Radford et al. (2021). "Learning Transferable Visual Models from Natural Language Supervision" (CLIP). OpenAI.
Liu et al. (2023). "Visual Instruction Tuning" (LLaVA). arXiv:2304.08485
Xu et al. (2024). "Hallucination is Inevitable: An Innate Limitation of Large Language Models." arXiv:2401.11817
Lin et al. (2025). "Vision Language Models: A Survey of 26K Papers." arXiv:2510.09586

Footnotes

Citation count as of 2025, per Google Scholar. ↩
Xu, Z. et al. (2024). Hallucination is Inevitable: An Innate Limitation of Large Language Models. arXiv:2401.11817. ↩
Measured directly by the RULER benchmark: Hsieh, C.-P. et al. (2024). RULER: What's the Real Context Size of Your Long-Context Language Models? Most models' effective context falls well short of the advertised window. ↩
Liu, N.F. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. TACL. ↩