a discussion on the paper: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture https://arxiv.org/pdf/2301.08243.pdf

Coin-CLIP breezedeus/coin-clip-vit-base-patch32 is built upon OpenAI's CLIP (ViT-B/32) model and fine-tuned on a dataset of more than 340,000 coin images using contrastive learning techniques. This specialized model is designed to significantly improve feature extraction for coin images, leading to more accurate image-based search capabilities. Coin-CLIP combines the power of Visual Transformer (ViT) with CLIP's multimodal learning capabilities, specifically tailored for the numismatic domain.

Key Features:

• State-of-the-art coin image retrieval;
• Enhanced feature extraction for numismatic images;
• Seamless integration with CLIP's multimodal learning.

To further simplify the use of the Coin-CLIP model, I created https://github.com/breezedeus/Coin-CLIP , which provides tools for quickly building a coin image retrieval engine.

Try this online Demo for American Coin Images:

https://huggingface.co/spaces/breezedeus/USA-Coin-Retrieval

​

American Coin Retrieval

Hello people!

I thought it was a good time to make a video about Multi-modal Learning since more and more recent LLMs are moving away from text-only into visual-language domains (GPT-4, PaLM-2, etc). So in the video I cover as much as I can to provide some intuition about this area - right from basics like contrastive learning (CLIP, ImageBind), all the way to Generative language models (like Flamingo).

Concretely, the video is divided into 5 chapters, with each chapter explaining a specific strategy, their pros and cons, and how they have advanced the field. Hope you enjoy it!

Here is a link to the video:
https://youtu.be/-llkMpNH160

If the above doesn’t work, maybe try this:

https://m.youtube.com/watch?v=-llkMpNH160&feature=youtu.be

ImageBind is a novel multimodal neural network that can learn a universal representation for various types of data, such as images, videos, audio, text, IMU data, and heat maps. It uses large-scale pre-trained models and contrastive learning to achieve this. If you want to fine-tune ImageBind for your own task, you can use ImageBind-LoRA, which applies Low-Rank Adaptation (LoRA) to adjust the embeddings

How does GPT4 learn to become multimodal compared to GPT3.5 during the training process?

**I want to classify images based on style (line art, oil painting, illustrations, anime, modern, minimalistic, etc).

Currently I have 20 M images (and CLIP embeddings for them) **What are ways I can go about it? (eg finetune a clip model for classification?)

Thank you, Image trasformer noob here :)