CLIP Unsupervised Learning Domain Adaptation

The Challenge of Domain-Specific CLIP Classification

Large-scale vision-language pre-trained (VLP) models, such as CLIP, excel at diverse applications in zero-shot settings. However, their performance often falters in specialized domains due to domain gaps and underrepresentation in training data. While supervised fine-tuning with human-annotated labels can address this, it's expensive and time-consuming, especially for complex tasks.

LATTECLIP: Unsupervised Fine-Tuning for CLIP

LATTECLIP proposes an unsupervised approach to fine-tune CLIP models for classification in specific domains, even without human-annotated labels. It leverages Large Multimodal Models (LMMs) to generate expressive text descriptions for images and groups of images, providing valuable contextual information to guide the fine-tuning process.

The Power of LMM-Generated Text Descriptions

LATTECLIP utilizes LMMs to create different types of text descriptions, including:

• Image-Description: Detailed descriptions of individual images.
• Group-Description: Descriptions capturing common characteristics of images with the same pseudo-label.
• Class-Description: Descriptions for all images within a specific category, offering stable representations.

These multi-level descriptions provide richer supervision for training, surpassing the limitations of pseudo-labels alone.

Addressing Noise in LMM-Generated Text

Directly fine-tuning CLIP with LMM-generated text can lead to poor performance due to hallucinations and noise. LATTECLIP addresses this by employing a prototype learning framework that learns class-specific representations from the generated texts.

Prototype Learning for Robust CLIP Fine-Tuning

LATTECLIP's prototype learning approach combines three key elements:

• Dual Pseudo-Labels: Utilizing both zero-shot and fine-tuned CLIP models to generate pseudo-labels, ensuring robustness and improved accuracy.
• Dynamic Feature Mixer: Dynamically weighting the influence of different text descriptions based on their similarity to the class prototypes, minimizing the impact of noisy descriptions.
• Momentum Update: Updating prototypes with momentum, smoothing optimization and reducing the influence of outlier samples or incorrect synthetic texts.

Experimental Results

LATTECLIP was evaluated on 10 domain-specific classification datasets, demonstrating significant improvement in top-1 accuracy compared to baseline methods, including:

• Pre-trained CLIP: LATTECLIP outperforms pre-trained CLIP models by an average of 4.74 points in top-1 accuracy.
• Unsupervised Baselines: LATTECLIP surpasses other unsupervised fine-tuning baselines by 3.45 points.

Ablation Studies

Ablation studies were conducted to analyze the contributions of different components of LATTECLIP:

• Different Types of Synthetic Descriptions: Removing either image-description, group-description, or both led to a decrease in performance, highlighting the importance of multi-level descriptions.
• Dynamic Feature Mixer: Ablation studies revealed that the dynamic feature mixer effectively weights the influence of different text descriptions, leading to a significant improvement in accuracy.
• Dual Pseudo-Labels: Removing either zero-shot or fine-tuned pseudo-labels significantly reduced performance, showcasing the effectiveness of combining both sources of supervision.
• Momentum Update: Removing the momentum update resulted in a substantial drop in performance, emphasizing the importance of stable prototype learning.

Conclusion

LATTECLIP is a promising unsupervised method for fine-tuning CLIP models on specialized datasets where human annotations are costly. Its ability to leverage LMMs for expressive text generation and its robust prototype learning framework effectively address the challenges of noise and domain adaptation, achieving significant performance improvements without relying on human labels.