The Most Misunderstood Word in AI

There's a phrase echoing through every AI conference, every boardroom pitch, every LinkedIn post from someone who just discovered machine learning last Tuesday: foundation model. It has become the industry's most prestigious label, and its most misunderstood one.

"We trained our model on a massive dataset, so it's a foundation model."

This sounds reasonable. It feels intuitively right. And it is, quite precisely, wrong.

The confusion isn't trivial. Misunderstanding what makes a model foundational leads to misallocated budgets, overpromised capabilities, and architectures that collapse the moment they're asked to do something they weren't explicitly trained for. If you're building AI systems, or betting your company on them, getting this definition right isn't academic. It's existential.

So let's get it right.

The Origin of a Name

The term "foundation model" was coined in August 2021 by a team of over 100 researchers at Stanford's Center for Research on Foundation Models (CRFM), led by Rishi Bommasani and colleagues. Their landmark report, "On the Opportunities and Risks of Foundation Models," gave the field its first rigorous definition:¹

A foundation model is any model that is trained on broad data (generally using self-supervision at scale) that can be adapted to a wide range of downstream tasks.

Read that definition carefully. There are three load-bearing words, and "large" isn't one of them. The words that matter are broad, self-supervision, and adapted. Each one does critical conceptual work, and each one is routinely ignored.

The Three Pillars

Breadth of data, not volume

The first requirement is that the training data be broad: covering diverse domains, modalities, or distributions. A model trained on ten billion radiology images is trained on a lot of data. It is not trained on broad data. It knows one thing very well. A foundation model, by contrast, has ingested enough variety to develop representations that transfer across domains it was never explicitly taught. The distinction is between depth and breadth, and foundation models need breadth.

Self-supervision as the training paradigm

Foundation models are typically trained through self-supervised learning: a paradigm where the model generates its own supervision signal from the structure of the data itself. In language, this might mean predicting the next word. In vision, it might mean predicting masked patches of an image. The key insight is that self-supervision on broad data forces the model to learn general-purpose representations rather than task-specific shortcuts. You're not teaching the model to classify cats; you're teaching it to understand the structure of visual information, after which cat classification falls out as a trivial downstream application.

Adaptability to downstream tasks

This is the criterion that most "we have a foundation model" claims fail. A foundation model is, by definition, incomplete. It's a foundation: a base upon which many different structures can be built. It must be adaptable, via fine-tuning, prompting, or other mechanisms, to tasks it was never explicitly trained to perform. If your model does one thing brilliantly and nothing else, it is an expert system. It might be a very good expert system. But it is not a foundation model.

The Emergence Problem

There's a fourth property that doesn't appear in the formal definition but is inseparable from what makes foundation models genuinely different: emergence.

Emergent capabilities are behaviours that appear in a model at scale despite never being explicitly trained. GPT-3 was trained to predict the next token. Nobody trained it to write code, translate between languages, or perform arithmetic. These capabilities emerged: they appeared as a consequence of scale applied to broad, self-supervised training on diverse data.² The smaller GPT-2, trained on essentially the same paradigm at a fraction of the scale, showed no comparable abilities.

This is the property that creates the most confusion, because it looks like scale alone is the explanation. "We just need a bigger model and more data!" But scale is a necessary condition, not a sufficient one. You can scale a model trained on a narrow dataset to a trillion parameters and you will get a very large narrow model. You will not get emergence. You will not get a foundation model. You will get an expensive disappointment.

Emergence requires the interaction of scale with breadth, self-supervision, and architectural choices that allow the model to form transferable internal representations. Remove any one of these ingredients and the magic doesn't happen, no matter how many GPUs you throw at the problem.

The Cost of the Confusion

When organisations mistake "large model trained on lots of data" for "foundation model," the consequences ripple outward in predictable ways.

They build monolithic systems that can't be adapted, and then wonder why they need to retrain from scratch for every new use case.

They invest in data quantity over data diversity, accumulating terabytes of homogeneous information that produces models with deep expertise and zero flexibility. They deploy models without adaptation layers, expecting a model trained on internet text to perform domain-specific reasoning in healthcare, legal, or financial contexts without any fine-tuning, and are shocked when the outputs are confidently, dangerously wrong.

Perhaps most importantly, they miss the real insight of the foundation model paradigm: that the value isn't in the model itself, but in its adaptability. The Stanford report was explicit about this: the word "foundation" was chosen deliberately over "foundational" because these models aren't foundational in the sense of providing fundamental principles. They're foundations in the architectural sense: incomplete structures that only become useful when you build something on top of them.

What Actually Makes a Model Foundational

If we strip away the hype, a foundation model is defined by a convergence of properties:

Training on broad, diverse data that spans domains and distributions
A self-supervised (or similar) training objective that forces the model to learn general-purpose representations rather than task-specific mappings
Sufficient scale (in parameters, data, and compute) that emergent capabilities can arise from the interaction of breadth and learning objective
Demonstrated adaptability to multiple downstream tasks through fine-tuning, prompting, in-context learning, or other adaptation mechanisms

No single one of these properties is sufficient. A model that satisfies only the first three but cannot be adapted isn't a foundation model; it's a large pretrained model. A model that satisfies only the last one through careful engineering but lacks broad pretraining isn't a foundation model; it's a well-designed modular system. The definition requires all four properties working in concert.

This is why BERT, GPT-3, CLIP, and DINO are foundation models.³ They were trained on broad data, using self-supervised objectives, at sufficient scale for emergent capabilities to appear, and they can be adapted to tasks their creators never imagined. It's also why a 100-billion-parameter model trained exclusively on dermatology images, no matter how accurate, is not a foundation model. It's a very large, very specialised model: impressive, valuable, but categorically different.

The Homogenisation Trade-off

There's a subtlety the Stanford report identified that most discussions of foundation models completely miss: homogenisation.

Because foundation models are so effective and so expensive to train, the entire industry increasingly builds on a handful of base models. This creates enormous leverage (thousands of applications built on the same foundation) but also creates systemic risk. Every bias in the foundation model, every failure mode, every vulnerability propagates to every downstream application. If the foundation cracks, everything built on it falls.

This is not a hypothetical concern. It's the central design tension of the foundation model era. And it's completely invisible if you think foundation models are just "big models trained on lots of data." The homogenisation risk only becomes apparent when you understand that foundation models are, by definition, shared bases for diverse applications: that the whole point is for many systems to depend on the same foundation.

The Bigger Picture

We are still in the early days of understanding what foundation models really are and what they can become. The definition coined in 2021 was deliberately broad because the phenomenon it describes is still evolving. But some questions are already sharpening into focus.

Can we build foundation models that are genuinely multi-modal from the ground up: not language models with vision bolted on, but systems that learn a unified representation of text, images, audio, video, and sensor data simultaneously?

What happens when foundation models are applied to domains where data is inherently scarce but stakes are high: drug discovery, surgical planning, climate modelling?

Can we develop foundation models for specialised scientific domains that are broad within that domain, even if they're narrow by general standards? And would such models satisfy the definition, or do we need a new term entirely?

And perhaps the most provocative question: if emergence is real and reliable, and if it depends on the interaction of scale, breadth, and self-supervision, then are we approaching a regime where the training paradigm matters more than the architecture? Where the decisive design choice isn't whether to use transformers or state-space models or mixture-of-experts, but rather how you structure the self-supervised objective and what distribution of data you feed into it?

The foundation model paradigm has already reshaped AI research, industry, and policy. But I suspect the deepest consequences of this shift (the ones that will matter most for how we build intelligent systems over the next decade) haven't arrived yet.

They're still emerging.

Footnotes

Bommasani et al., On the Opportunities and Risks of Foundation Models, Stanford Center for Research on Foundation Models, August 2021 (arXiv:2108.07258). ↩
Brown et al., Language Models are Few-Shot Learners (2020) documented these abilities in GPT-3; Wei et al., Emergent Abilities of Large Language Models (2022) named the phenomenon and catalogued it across model families. ↩
BERT: Devlin et al., 2018. GPT-3: Brown et al., 2020. CLIP: Radford et al., 2021. DINO: Caron et al., 2021. Two modalities, four architectures, one definition satisfied. ↩