In News|38 Minutes

Multi-modal foundation models out of the lab: a reality check

Multi-modal foundation models out of the lab: a reality check

The field of computer vision is in the middle of a paradigm shift, driven by the revolutionary power of foundation models.

A new class of “zero-shot” and “few-shot” systems, exemplified by powerful multi-modal architectures like GPT-5, LLaVA, CLIP or Molmo, has captured the industry’s imagination. 

These models demonstrate an unprecedented ability to understand and interact with visual information in an “open-world” context, meaning they can classify and detect objects they have never been explicitly trained on. They can describe complex scenes in natural language, and “fill in the gaps” by leveraging vast, internet-scale knowledge. This promise of generalist, adaptable visual intelligence threatens to make the classical, data-intensive approach to computer vision obsolete, offering a view of a future where development costs drop and time-to-market accelerates.

Example of what a multi-modal foundation model, Gemini 3 Pro, can do today. The use case is borrowed from one of Deep Vision Consulting past project. Answers, on the right, have been parsed and shortened for clarity, but not altered in correctness.

Let’s start our journey with an example of what one such model can already do today. First we input the model with an image, in our case the one on the left. Now we can start a conversation with this model by asking either new or fixed questions and wait for the answers.

Question 1: Describe the image.
Answer: This image shows a heavy-duty automotive wheel hub assembly, likely from a truck or trailer, resting on a workbench.

Question 2: Knowing there should be 10 metal studs, are there any missing from the hub?
Answer: We can identify the following:

  • 6 studs present (3 on the left side and 3 on the right side);
  • 4 empty mounting holes (2 at the top located between the uppermost left and right studs and 2 at the bottom located between the lowermost left and right studs).

Question 3: Are there any of the following defects? For each category answer only yes (and where) or no: 1. rust, 2. copious lubricant leakage on the hub, 3. broken parts (not studs)
Answer: Based on the visual evidence:

  1. Yes, around the perimeter and inside the empty holes at the top and bottom.
  2. Yes, dark, wet sludge is heavily coating the central hub seal area.
  3. No.

While it is impressive that a model (Gemini 3 Pro, in the example above) has been able to answer to the questions without any sort of specialized training, a careful reader might have spotted an incorrect answer out of the many correct ones. In the last round of conversation the model stated that there has been lubricant leakage, while this is not the case, the black coating is just anti-rust paint and should definitely be present.

Indeed, a critical disconnect exists between this compelling “open-world” promise and the unforgiving reality of industrial production. As computer vision consultants specializing in real-world AI deployment, we have consistently found that the demands of enterprise-grade production, e.g. reliability, accuracy and accountability, are often fundamentally at odds with the generalist (and mostly generative) nature of these models.

So as a solution provider, some of the key questions that we are interested in are:

  • (obviously) How often is the model right?
  • When there is ambiguity, is the model detecting it or he just tries to answer anyway?
  • How can we change the threshold for what we consider a defect and what we want to accept as good?
  • If the model needs to improve, what knobs do I have to make it stronger on the specific use case?

As of today technology level, these questions don’t find reassuring answers and our stance is as follows: for any well-defined, high-stakes task, a data-centric approach of fine-tuning a model on sufficient, high-quality, task-specific data offers superior control, performance, and long-term robustness with respect to any generalist few-shot model.

This report provides a critical examination of this trade-off between prior knowledge and data, moving beyond the technological hype of zero-shot and Vision-Language Models (VLM).

TABLE OF CONTENTS:

1. The Zero-Shot to Adaptation Continuum
2. The Landscape of Multi-Modal Foundation Models
3. What Do These Models Actually Understand?
4. The Industrial Gap: Or Why Multi-Modal Foundation Models Fail in Practice
5. A Strategic Framework for Deploying Vision Models
6. Conclusions

1. The Zero-Shot to Adaptation Continuum

Before we start, it’s important to clarify the continuum of learning paradigms. In this context, a “shot” simply refers to a tagged example of the task the model needs to solve. This spectrum ranges from “zero-shot” to “few-shot” and, ultimately, to traditional “full-training.”

  • The Zero-Shot approach, by definition, requires the model to perform a task without any specific examples. To achieve this, the task must be described in some other way, which in practice means using natural language. For instance, a model is asked to choose between text prompts like “a photo of a perfect glass bottle” and “a photo of a broken glass bottle”, or to generate a detailed description of the image and then parse it according to one’s goal.
  • Conversely, One-Shot & Few-Shot methods provide the model with one or a few examples of the task to be solved. Peculiarly, these examples can be provided through vision alone. For example, you could simply show the model an image of a “broken glass bottle” and ask it to find similar items, perhaps by clustering them in a visual embedding space. This flexibility makes few-shot approaches highly relevant for industrial tasks where visual examples are easier to provide than text descriptions.
  • At the end of the spectrum, we find the Adaptation paradigm, where a model (or a subset of it) is fine-tuned on a dataset containing sufficient, high-quality, task-specific data. Based on the complexity of the task and its gap from the model previous knowledge, the amount of data required may go from a few dozens to hundreds of samples. But it’s not “zero”, “one” nor “few” anymore.

The allure of these “open-world” approaches is their revolutionary promise: to bypass the classical, data-intensive training process, enabling rapid prototyping and deployment with minimal setup time. It is easy to understand why everyone is talking about zero-shot and VLM these days.

2. The Landscape of Multi-Modal Foundation Models

Another interesting way to deconstruct these systems is based on their underlying architecture and the specific output they produce. This slicing reveals four different technical paradigms, sorted in Table 1 by their ability to produce more complex and semantic output.

ParadigmOutputExamplesCore & Enabling Mechanics
Vision-Vision Matching **distance / similarity scoreDINO, I-JEPASelf-Supervised Learning (SSL) on raw pixels. Learns visual correspondence and structure without any language.
Open world matchingdistance / similarity scoreCLIP, SigLip, Perception EncoderContrastive Learning on (image, text) pairs. Aligns image and text encoders in a shared feature space.
Text-Decoder VLMtextLLaVA, Molmo, Gemma3, Qwen-VLGenerative LLM. A vision encoder (e.g., CLIP) provides features to a text-decoder (LLM) to generate conversational text outputs.
Any-to-anyanything4M-21Omni-modal space. Uses modality-specific tokenizers to map all data types (images, text, masks, depth) to a shared “invertible” space.
Table 1: A Comparative Analysis of foundation models paradigms in Computer Vision.
** The careful reader will notice that these methods are not really multi-modal, but their few-shot performance is so strong that they deserve to be on the menu.

In the rest of this section we describe these paradigms in more detail and comment on the applicability of each one of them.

Vision-Vision Matching (e.g., DINO or I-JEPA)

The first paradigm, represented by models like the ones in the DINO or JEPA families, is fundamentally different from the others. These are self-supervised learning (SSL) models, which learn visual representations directly from raw pixel data without any human-provided text annotations or labels. The objective is not to associate an image with a word, but to learn the inherent structure of the visual world: how pixels in an image relate to each other, how objects move in a video, or what defines a coherent 3D shape.

For industrial applications, this is a critical distinction. DINOv3 features, as an example, are not “semantic” in an open-vocabulary sense; one cannot ask it to find “a broken gear” using text. But you can find “a broken gear” by showing another image of a broken gear. The semantic strength of these features are so impressive that many tasks can often be solved with just a single example – taking these approaches very close to zero-shot.

Notably, the independency from text has also a second advantage: it can be trained on any image dataset without need for any label attached. This enlarges the training data pool to many more orders of magnitude in size and domains w.r.t. other approaches described below.

The first paradigm, represented by models like the ones in the DINO or JEPA families, is fundamentally different from the others. These are self-supervised learning (SSL) models, which learn visual representations directly from raw pixel data without any human-provided text annotations or labels. The objective is not to associate an image with a word, but to learn the inherent structure of the visual world: how pixels in an image relate to each other, how objects move in a video, or what defines a coherent 3D shape.

For industrial applications, this is a critical distinction. DINOv3 features, as an example, are not “semantic” in an open-vocabulary sense; one cannot ask it to find “a broken gear” using text. But you can find “a broken gear” by showing another image of a broken gear. The semantic strength of these features are so impressive that many tasks can often be solved with just a single example – taking these approaches very close to zero-shot.

Notably, the independency from text has also a second advantage: it can be trained on any image dataset without need for any label attached. This enlarges the training data pool to many more orders of magnitude in size and domains w.r.t. other approaches described below.

The output of such models are features (a.k.a. embeddings). On top of these features it is common to add a layer of logic e.g. train a simple MLP (multi-layer perceptron; or some other classical machine learning technique) or compute PCA (principal component analysis) to specialize the semantics of these features to the use case. These are two ways in which few-shots can be used to improve over raw features. Notably, even in the case of just one-shot, one can still compute a distance of any input image from such shot and apply a threshold to decide whether you are looking at the same object (even at the same pixel).

Typical tasks that can be solved with one- or few-shots through vision-vision matching include classification, segmentation and semantic keypoint detection.

Open World Matching (e.g., CLIP, SigLip or Perception Encoder)

The second paradigm, and the one that truly ignited the zero-shot revolution, is vision-language matching, defined by OpenAI’s CLIP. In sharp contrast to the previous category, CLIP-like methods are trained via contrastive learning on millions of (image, text) pairs scraped from the public internet. Its architecture consists of two encoders, one for images and one for text. These encoders are trained jointly to map “correct” image-text pairs to a similar point in a shared embedding space, while pushing incorrect pairs apart.

The power of this approach is its emergent capability for zero-shot classification. A developer can create a classifier for any visual concept on-the-fly, without a single training example, simply by creating a set of text descriptions (a.k.a. prompts). For instance, to detect different objects that can end up on a conveyor belt, one can simply feed the model an image and ask it to choose between the text prompts “a photo of a nut”, “a photo of a screw” and so on for any other type of object that one wants to discriminate. The prompt with the highest similarity to the image embedding will be the system output.

Although the main task of CLIP is image-level classification, these approaches can be extended to make their patch level features comply to the text features themselves (e.g. MaskCLIP), effectively enabling segmentation abilities through text prompting.

Text-Decoder VLMs (e.g., LLaVA, Molmo, Gemma3, Qwen-VL)

The third paradigm, which includes popular models like LLaVA, Molmo, and GPT-4o onwards, represents a powerful architectural evolution. These models connect a pre-trained vision encoder (like CLIP’s) to a large language model (LLM) decoder using a simple projection layer. This simple connection fundamentally changes the model’s function. It transforms the system from a classifier / segmenter (like CLIP) into a text generator that outputs a stream of arbitrary text tokens. Very cleverly, the decoder is input with both image tokens and text tokens (the query), and handles them at the same manner, greatly simplifying the architecture design.

Through the generation of structured text, many of these models can not only classify images, but also describe them in detail (a.k.a. captioning) or even solve simple tasks like pointing or detection (i.e. by textually predicting x, y coordinates) if trained to do so. The key advantage with respect to CLIP, is that to apply such a system to a problem one doesn’t need to know in advance all the possible objects that can appear, because the text generation is free to craft any response.

Any-To-Any models (e.g., 4M-21)

The final paradigm is the “omni-model,” also referred to as “any-to-any” model, like the 4M-21 model. The genesis of these models was probably Grounding-DINO, i.e. a method that could detect (through bounding boxes) any category described through a text prompt, the first of what we now call “open-world detectors”, immediately followed by SAM that lets you segment a scene through pointing, clicking or text. After Groundin-DINO and SAM people started asking themselves: why don’t we try to predict “everything”, not just a caption nor just a detection, nor just a mask?

The answer was “any-to-any” models.

The 4M-21 model, for instance, was trained on 21 diverse modalities simultaneously. These include not only RGB images and text, but also geometric data (depth maps, surface normals), semantic data (segmentation masks, bounding boxes), 3D human poses, and even the feature maps from other specialist models like DINOv2 and SAM.

This means that by providing an image to such models, one can still get a text response like in VLM, but also gets a full range of additional information that can be used to design more complex vision pipelines. All of this, without a single training example.

The key innovation of 4M-21 is its use of modality-specific tokenizers to convert all of these disparate data types into a shared, discrete representation, which is then fed into a single transformer. This architecture allows it to be prompted with any combination of modalities to generate any other modality. As an example: it can be prompted with a segmentation mask to generate a realistic RGB image, or prompted with an RGB image to generate a corresponding depth map.

For industrial applications, this paradigm-shifting capability suggests a future where a single omni-model could be prompted with an engineering CAD file (a 3D modality), an image of the actual piece acquired from a simple RGB camera and a text prompt (“Inspect this real-world part for deviations”) to perform a complex inspection task. And then the same model could be used to generate a CAD of a piece just by providing an image and an optional text description. But we are not there yet.

There are obviously intermediate steps between VLMs and any-to-any models, but the key aspect of this category is that they can be trained to predict anything.

3. What Do Visual Language Models Actually Understand?

The claims around multi-modal foundation models and VLM are extensive, but to make strategic decisions, a rigorous validation is essential. Although these models work with image data and can solve computer vision tasks, it is not clear how much of the success is to be attributed to their language fluency (not related to the specific input image) and how much to vision understanding.

In the 2025 “How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks” paper link, a group of researchers led by Amir Zamir benchmark GPT-4o and other (at the time) state-of-the-art commercially available multi-modal foundation models, providing a rigorous, quantitative analysis on a battery of standard, academic computer vision tasks, as shown in the image to the right. All these models belong to the third category of “Text-Decoder VLM”.

The paper’s methodology is a clever solution to a difficult problem. Since proprietary models like GPT, Gemini and Claude are “closed-weight,” accessible only via API, and only output text, they cannot be directly tested on tasks that require pixel-level outputs (like segmentation or depth estimation). Researchers had to force text-only models to ‘see’ by breaking images into grids and asking the model to describe each grid cell sequentially. One could argue that this trick is severely limiting the context of these models, possibly degrading their performance. The authors also provide clear evidence of such impacts and show that the following three conclusions still hold.

Key Finding 1: Respectable Generalists, Poor Specialists

The paper’s primary conclusion is stated unequivocally: “The models are not close to the state-of-the-art specialist models at any task”, as shown in the graph on the right. This confirms that the generalist, open-world models are not a drop-in replacement for specialized, trained models. The performance gap is quite large, at the time of writing. For example, on the COCO semantic segmentation benchmark, the very best-performing VLM (GPT-4o) achieves a mean Intersection-over-Union (mIoU) of 44.89. A specialist model (OneFormer) on the same task scores 65.52. While the paper generously calls the multi-modal foundation models “respectable generalists”, for an industrial application requiring high precision, their performance is simply not competitive.

Key Finding 2: The Semantic vs. Geometric Divide

The most critical finding for industrial use cases is that multi-modal foundation models “perform semantic tasks notably better than geometric ones”.

The models are strong at semantic tasks like classification (e.g., performing well on ImageNet) or detection. This is their comfort zone, as it aligns well with their internet-text training. However, they “struggle” with geometric tasks that require an understanding of 3D space, shape, and physical layout. On tasks like surface normal prediction and depth estimation, their performance is weak, with some models showing a profound misunderstanding of the 3D world.

Key Finding 3: Hallucination and Misalignment

Finally, the paper notes that even when these models can be adapted to generate visual outputs (a new feature from GPT-4o onwards, that places in the middle of VLM and any-to-any models), they suffer from significant “failure modes, such as hallucinated objects or misalignment between input and output”.

4. The Industrial Gap: Or Why Multi-Modal Foundation Models Fail in Practice

Unfortunately, the key findings listed in the previous section have a non-negligible impact when transposed in the industry.

As this wasn’t enough, we now introduce what we think is the single greatest barrier to deploying (multi-modal) foundation models in practice: the “domain gap”. This is the disconnect between the clean, well-lit “natural scenes” found in internet-scale training data and the messy, specialized environment of a factory floor. The open-world knowledge is a literal reflection of its public internet training data. Specialized industrial data, such as X-rays, infrared scans, or high-magnification images of micro-defects, is inherently out-of-distribution. This is not a news in the field: CLIP creators documented that zero-shot usage of their model has “poor capability” on “specialized, complex, or abstract tasks” and fails on novel inputs unlikely to be in its pre-training dataset.

Examples from industrial scenario borrowed from the MVTec dataset. These images are difficult to be found in large scale training data composed of images and captions, so pretrained VLM understand very little about these images. Many industries also put lots of effort in avoiding images from their production site to go public.

Examples from industrial scenario borrowed from the MVTec dataset. These images are difficult to be found in large scale training data composed of images and captions, so pretrained VLM understand very little about these images. Many industries also put lots of effort in avoiding images from their production site to go public.

As a second point, the reader can consider the many industrial applications including robotics, assembly verification, and bin-picking that are fundamentally geometric tasks that rely on precise spatial and 3D understanding, not just semantic labeling. It is obvious that semantic understanding alone, something that VLM excels at, cannot solve these tasks.

So is this the end of multi-modal foundation models in industry?

No. Interestingly, models that are not trained mostly on text and language, such as self-supervised (e.g. DINOv3) or any-to-any models exhibit a higher degree of 3D understanding and out-of-domain generalization.

While general knowledge is usually a good thing to have, in industry is better to trade off some of it for specific knowledge.

Moreover, most of the starting knowledge these models have is so general that it is easy to specialize it to the use case at hand, through supervised or unsupervised few-shot techniques like PCA or a simple MLP. While general knowledge is usually a good thing to have, in industry is better to trade off some of it for specific knowledge.

This possibility to tune models also highlights a divide between open and private solutions. Models like GPT-5 or Gemini 3, two of the best performing VLM at the time of writing, simply cannot be specialized to custom tasks as one can access only its final output. So open models should be preferred in these cases.

5. A Strategic Framework for Deploying Vision Models

This analysis leads to a clear framework that we apply in our industrial AI and computer vision projects. We go beyond the binary choice between “zero-shot” and “fine-tuning” on a large dataset, to formalize a project-dependent progression: this framework validates the data-centric philosophy (that is now accepted in the industry) while leveraging new models to achieve it.

First, the open-world models still have a valid important role. When a project has no labeled data, or when time-to-market is more relevant than accuracy, a zero-shot or few-shot model is an invaluable tool as they are “better-than-nothing”.

These models are ideal for reducing data costs and establishing an initial, low-fidelity baseline. This is the correct, low-risk entry point for a project in the exploratory phase, allowing for rapid prototyping and hypothesis validation before committing a significant budget to data collection.

Use Zero-Shot Responsibly

The strategic error many companies make is seeing a zero-shot model as the final product. At Deep Vision Consulting, we view it differently: the zero-shot model is the tool used to build the solution.

We implement this idea through an active learning strategy:

  1. Deploy the Baseline: We deploy a “better-than-nothing” few-shot classifier (e.g., based on CLIP, DINOv3 or VLM). We accept that it will be imperfect.
  2. Hunt for Edge Cases: The model’s primary job is to fail productively. We configure it to flag every image where its confidence score is low or ambiguous.
  3. Human-in-the-Loop (HITL): These ambiguous images are funneled to a human expert. Because the model has filtered out the “easy” cases, the human only spends time labeling the high-value edge cases.
  4. Bootstrap the Specialist: We use this high-quality, hard-example dataset to fine-tune a smaller, faster model.

This transforms the zero-shot model from a weak production system into a powerful, automated data curation engine.

The Inevitability of Fine-Tuning: Data > Common Knowledge

If your task is clear and you have development budget, data is superior to common knowledge. The “out-of-the-box” performance of any foundation model on a specialized industrial task will be poor. Fine-tuning is the “standard approach” to close this domain gap.

The dataset gathered with a first model on the field becomes the high-quality fuel for this fine-tuning. This step represents the critical pivot from a general-purpose foundation model to a specialized, high-performance, and high-control production asset. This new, fine-tuned model, whether it’s a DINOv3 backbone with a custom segmentation head or a fine-tuned CLIP classifier, is now an asset owned by the client, trained on their proprietary data, and optimized for the one task it needs to do perfectly.

6. Conclusions

There is no doubt that the methods and approaches that fall under the umbrella of one-shot, few-shot, VLM, multi-modal foundation models can have a valuable impact in some part of the vast computer vision landscape, yet they are far from solving all industry problems. This analysis reveals several critical considerations for industrial deployment, moving beyond the hype to the practical realities of production.

The Limits of Language

In industrial applications, what often matters are the nuances and fine-grained details: the subtle visual difference of a hairline crack, a minor alignment shift, or a specific texture indicating contamination. These details are often extremely difficult, if not impossible, to describe accurately and unambiguously with a text prompt. This limitation is a significant barrier for text-reliant models like CLIP and VLMs. We have also seen how models fundamentally relying on language have poor 3D understanding of the world, something that can be critical in some vision and robotics tasks.

Domain Gap

Foundation models need to be trained on huge amounts of data. Inevitably such amount of data can only be “general” in context, scenes from our daily lives, depicting common knowledge and natural scenes. This discrepancy is the reason all pretrain models underperform when put to test in industrial setups, the images are just from two different worlds. The semantic of industrial images is alien to the understanding built within these foundation models, with or without language.

The "Out-of-the-Box" Control Problem

Zero-shot models are appealing because they “just work” out-of-the-box. But this becomes a critical liability when they fail. If a model in production provides a wrong or nonsensical answer, how do you improve it? One cannot simply interact with a live, high-throughput system to iteratively change text prompts when a result is unsatisfactory. This lack of control and lack of a clear path for improvement is a fundamental risk, especially when generative models are prone to “hallucination”, making them unsuitable for close to zero-tolerance tasks.

We have seen how these models can be used to kickstart a solution in the field, to collect proper data as soon as possible. The other side of the zero-shot coin offers a different, perhaps more intriguing, application. Instead of being used as the production system itself, generative any-to-any or text-to-image models can be used as powerful tools for synthetic data generation. This synthetic data can then be used to create the robust, high-quality datasets needed for more traditional, reliable, and controllable training procedures, bridging the gap between new paradigms and the proven, data-centric approaches required for industrial-grade AI. But this is a totally different story, worth a blog post on its own.

Zero-shot models are a powerful new tool in the kit, but they aren’t a universal solution. The real challenge lies in understanding the specific problem constraints, i.e. data availability, accuracy requirements, and production risks, to choose the right path forward. If you have a complex computer vision project, contact us: we can help you move beyond the modern and frenetic hype to build robust systems that actually deliver.

emerging-technologies foundation model visual language model vit zero-shot
Related Posts