The Data Paradox: Why 90% to 95% Is a Different Problem

The journey from 90% to 95% accuracy is fundamentally different from the journey from 80% to 90%. Not just harder: different in kind.

The Curve That Bends Against You

Getting to 90% accuracy on a computer vision task often feels like climbing a steep but well-marked trail. The path is visible. You collect data, you label it, you train, you iterate. The model learns the obvious patterns first: the common cases, the well-represented categories, the clearly-defined boundaries.

But then you hit the last 5%.

That final stretch is where the economics of machine learning become genuinely painful. Each additional percentage point of accuracy requires exponentially more data. What took 10,000 labelled examples to achieve the first 80% might require 100,000 more to gain another 10%, and then another million to squeeze out the final 5%. The relationship isn't linear. It's a curve that bends against you.

This is why the most important question in applied computer vision isn't "how do we get more data?" but rather "how do we get better results with less data?"

That question has defined much of my work over the past several years.

The Long Tail: Where Models Go to Struggle

Real-world data almost never looks like the tidy, balanced datasets we see in academic benchmarks. Instead, it follows a long-tail distribution: a small number of categories dominate the data, while a vast number of rare cases stretch out in an endless tail of infrequent occurrences.

Consider a medical imaging system trained to detect pathologies. Common conditions (pneumonia, fractures, basic tumours) might comprise 80% of the training data. But the rare conditions, the edge cases, the unusual presentations? They're scattered across that long tail, sometimes with only a handful of examples each. And it's precisely those rare cases where accurate classification matters most. A missed rare cancer is not equivalent to a misclassified common cold.

The same pattern appears everywhere: manufacturing defect detection, autonomous driving perception, wildlife species identification, satellite imagery analysis. The head of the distribution is easy. The tail is where products fail in the real world.

Traditional supervised learning struggles with this fundamental asymmetry. Models optimised on balanced loss functions naturally gravitate toward the majority classes. They learn to recognise the common cases well and essentially guess on the rare ones. You can weight losses, oversample minorities, or undersample majorities, but these techniques treat symptoms, not causes.

The deeper problem is that learning from labelled examples alone creates a dependency on having labelled examples, and for long-tail categories, those examples either don't exist or are prohibitively expensive to obtain.

From Supervised Learning to Zero-Shot: A Paradigm Shift

For decades, supervised learning has been the dominant paradigm in computer vision. You collect images, you label them, you train a model to map inputs to outputs. The model learns exactly what you show it: nothing more, nothing less. This approach works remarkably well when you have abundant labelled data for every class you care about. It breaks down when you don't.

Zero-shot learning inverts this dependency. Instead of requiring labelled examples of every category, zero-shot models leverage prior knowledge, typically encoded in language, to recognise categories they've never explicitly seen during training. You describe what you're looking for, and the model finds it.

This isn't magic. It's a fundamentally different way of encoding knowledge about the visual world.

Between these extremes lies a spectrum of approaches: few-shot learning (learning from a handful of examples), self-supervised learning (learning representations without explicit labels), and various forms of transfer learning that move knowledge from data-rich domains to data-scarce ones.

Understanding where your problem falls on this spectrum, and which approach fits your constraints, is now one of the most important decisions in applied computer vision.

The Current State of the Art: Classification

Image classification has been transformed by vision-language models, and the flagship example is CLIP (Contrastive Language-Image Pre-training). Trained on hundreds of millions of image-text pairs scraped from the web, CLIP learns a shared embedding space where images and text descriptions can be directly compared. Ask "is this a photo of a golden retriever?" and CLIP computes the similarity between the image embedding and the text embedding without ever having been explicitly trained on golden retriever classification.

The zero-shot classification performance is remarkable, often competitive with fully supervised models trained on the target dataset. CoCa (Contrastive Captioners) has pushed this further, achieving 91% top-1 accuracy on ImageNet when fine-tuned, while maintaining strong zero-shot transfer across domains.

But CLIP's real power isn't raw accuracy on benchmarks. It's the ability to classify images into categories defined at inference time, using nothing but natural language. This changes the deployment model entirely. Instead of retraining when requirements change, you update the text prompts.

For self-supervised approaches, DINOv2 represents the current frontier. Unlike CLIP, DINOv2 learns purely from images without any text supervision, using a teacher-student framework where the model learns to produce consistent representations across different augmented views of the same image. The resulting features are general-purpose: they work across classification, segmentation, depth estimation, and retrieval without task-specific fine-tuning.

DINOv2's features can be used with simple linear classifiers to achieve strong results, making it exceptionally practical for scenarios where you have limited labelled data but need robust visual understanding. DINOv3, released recently, scales this to 7 billion parameters with improved positional encoding for handling varied image resolutions.

The Current State of the Art: Segmentation

Segmentation, the task of delineating precise object boundaries at the pixel level, has historically required meticulous annotation. Labelling a single medical image for segmentation might take hours of expert time. This made training data the primary bottleneck.

The Segment Anything Model (SAM), released by Meta AI, fundamentally changed this calculus. Trained on over 1 billion masks across 11 million images, SAM demonstrates zero-shot segmentation: give it a point, a bounding box, or a text prompt, and it produces high-quality segmentation masks for objects it has never seen during training.

SAM 2 extended this to video, enabling tracking and segmentation across frames with temporal consistency. SAM 3, the latest iteration, introduces concept-based segmentation: using text descriptions like "red apple" or "person wearing a hat" to segment all matching instances in an image or video without any additional training.

For medical imaging, where SAM's zero-shot performance initially showed limitations on dense or unusual anatomical structures, specialised adaptations like MedSAM have emerged. These combine SAM's promptable architecture with domain-specific training to achieve strong results even on pathological structures the original model never encountered.

The practical implication is profound: you can now build segmentation systems for new domains with a fraction of the labelled data previously required, or in some cases, with no domain-specific labels at all.

The Current State of the Art: Identification and Detection

Object detection, finding and localising objects within images, has similarly been transformed. Foundation models like Grounding DINO combine the visual grounding capabilities of DINO with language understanding, enabling open-vocabulary detection: describe what you want to find in natural language, and the model locates it.

This approach is particularly powerful for the long tail. Instead of needing thousands of labelled bounding boxes for each rare category, you can detect rare objects by description. "Find the damaged bearing" or "locate the corroded valve" becomes possible without explicit training on those specific defect types.

For few-shot scenarios, embedding-based approaches have matured significantly. Models like DINOv2 produce visual embeddings that cluster semantically similar images together. You can build effective classifiers with as few as 5–10 examples per class by simply comparing embeddings to prototype representations: no gradient updates required.

The combination of SAM for segmentation, CLIP for classification, and DINOv2 for robust embeddings creates a flexible toolkit that can be assembled for specific use cases with minimal task-specific training.

Practical Approaches to the Long-Tail Problem

Understanding the theory is one thing. Deploying solutions that work in production is another. Here are the approaches I've found most effective for addressing long-tail distributions in real-world computer vision systems:

Leverage foundation models as feature extractors. Instead of training end-to-end on your limited data, use DINOv2, CLIP, or similar models as frozen feature extractors. Train only a lightweight classification head on your domain-specific data. This dramatically reduces the amount of labelled data needed because the hard work of learning visual representations has already been done.

Use embedding similarity for rare classes. For categories with very few examples (5–20), don't try to train a classifier. Instead, compute embeddings for your examples and classify new inputs by nearest-neighbour matching. This works surprisingly well with good embeddings and requires no training whatsoever.

Combine zero-shot and supervised approaches. Use zero-shot models like CLIP for long-tail categories where you lack training data, and supervised models for head categories where you have abundant data. This hybrid approach lets you leverage the strengths of both paradigms without their individual weaknesses.

Generate synthetic data strategically. Synthetic data generation (using 3D rendering, domain randomisation, or generative models) can augment rare categories. The key is targeting generation at the specific gaps in your distribution rather than generating data uniformly. Generating 10,000 variations of common defects doesn't help; generating 1,000 variations of the rare defect that appears three times in your dataset does.

Implement class-balanced sampling during training. When you must train supervised models, use progressively-balanced sampling: start with the natural distribution, then gradually shift toward uniform sampling as training progresses. This lets the model learn robust features from abundant data while also seeing sufficient examples of rare classes.

Use multi-expert ensembles. Train separate models optimised for different parts of the distribution (one expert for head classes, another for tail classes) and combine their predictions. This approach can adapt to arbitrary test distributions by adjusting the expert weighting at inference time.¹

Build continuous learning pipelines. The long tail isn't static. New rare cases emerge over time. Build infrastructure to identify novel patterns in production data, flag them for labelling, and incrementally update your models. The teams that succeed treat data curation as an ongoing process, not a one-time event.

Apply test-time adaptation. Some models can adapt their predictions based on the statistics of incoming test data without retraining. This is particularly useful when deployment distributions shift from training distributions, a common scenario in real-world applications.

Data Quality Over Data Quantity

One insight that emerges from working with foundation models is that data quality matters more than ever. When you're training a model from scratch, you can sometimes compensate for noisy labels with more data. When you're fine-tuning a pretrained model or using few-shot learning, every example counts.

Small, carefully curated datasets often outperform large, noisy ones. A few hundred precisely labelled examples of a rare category can be more valuable than thousands of approximately labelled common cases. This shifts the economics: instead of optimising for annotation throughput, optimise for annotation precision.

Up to 80% of an AI project's timeline is consumed by data preparation: collection, cleaning, curation, and annotation.² The organisations that excel are those that treat data curation as a first-class engineering discipline, not an afterthought.

Critical Considerations Often Overlooked

Several important factors are frequently underestimated when deploying these approaches:

Domain gap is real. Foundation models trained on web-scraped data may not transfer cleanly to specialised domains like medical imaging, satellite imagery, or industrial inspection. The visual statistics are different. The relevant features are different. Always validate zero-shot performance on your actual data before assuming it will work.

Prompt engineering matters. The text descriptions you use with CLIP or SAM significantly affect performance. "A photo of a dog" and "a photograph of a canine" can produce different results. Invest time in finding prompts that align with your model's training distribution.

Calibration degrades at the tail. Model confidence scores become unreliable for rare classes. A model might report 95% confidence for a tail-class prediction while actually being wrong 40% of the time. Build calibration into your evaluation; don't trust raw softmax outputs.

Evaluation on balanced test sets is misleading. If your test set is class-balanced but your deployment is long-tailed, your metrics will overstate real-world performance. Always evaluate on distributions that match deployment.

Computational cost varies dramatically. Foundation models are large. Running SAM on every frame of a video stream may not be feasible. Understand the latency and throughput requirements of your application before committing to an architecture.

Failure modes are different. Zero-shot models fail differently than supervised models. They might confidently classify something into a plausible-sounding but incorrect category. Build human-in-the-loop verification for high-stakes decisions.

Where This Is Heading

The trajectory is clear: we're moving from task-specific models trained on task-specific data toward general-purpose visual understanding systems that can be adapted to new tasks with minimal additional training.

This doesn't mean supervised learning is obsolete. It means the role of supervised learning is changing. Instead of teaching models to see, we're teaching already-capable models to see the specific things we care about. Instead of building visual understanding from scratch, we're adding domain-specific layers on top of robust foundations.

The practical implication is that smaller teams can now tackle problems that previously required massive datasets and compute budgets. The democratisation is real, but so is the expertise required to navigate this landscape effectively.

Foundation models have changed what's possible. They haven't eliminated the need to think carefully about your specific problem, your specific data, and your specific constraints.

The Engineer's Responsibility

Working between computer vision research and real-world deployment, I've come to believe that our responsibility extends beyond building accurate models. We need to build systems that degrade gracefully, that know what they don't know, and that handle the long tail of reality without catastrophic failures.

The journey from 90% to 95% accuracy is hard. But it's also where the interesting problems live, and where the solutions matter most.

The tools we have today (CLIP, SAM, DINOv2, and their successors) give us new ways to approach these problems. They don't solve them automatically. They require understanding, judgement, and careful application.

The models have changed. The engineering discipline remains.

Footnotes

1. Zhang et al., SADE: test-agnostic long-tailed recognition through self-supervised aggregation of diverse experts, 2022. ↩

2. Cognilytica's widely cited estimate. The exact share varies by survey; the dominance of data preparation does not. ↩