PaliGemma – Google's Cutting-Edge Open Vision Language Model
•
81
Merve Noyan PRO
merve
AI & ML interests
VLMs, vision & co
Articles
Organizations
Posts
27
Post
867
So many of you have asked how to do segmentation and detection with PaliGemma, you've been served! 🫡
Here's the notebook to do so: https://colab.research.google.com/drive/16-Tq-iAMHNlSjDWgz43kYDMJERjU_KHW?usp=sharing 🤗
Here's the notebook to do so: https://colab.research.google.com/drive/16-Tq-iAMHNlSjDWgz43kYDMJERjU_KHW?usp=sharing 🤗
Post
1086
it's raining vision language models ☔️
CuMo is a new vision language model that has MoE in every step of the VLM (image encoder, MLP and text decoder) and uses Mistral-7B for the decoder part 🤓
You can try it yourself here: shi-labs/CuMo-7b-zero
the authors firstly did pre-training of MLP with the by freezing the image encoder and text decoder, then they warmup the whole network by unfreezing and finetuning which they state to stabilize the visual instruction tuning when bringing in the experts. 🤓
the mixture of experts MLP blocks above are simply the same MLP blocks initialized from the single MLP that was trained during pre-training and fine-tuned in pre-finetuning.
it works very well (also tested myself) that it outperforms the previous sota of it's size LLaVA NeXt and IDEFICS2-8B in several benchmarks! 😍
CuMo is a new vision language model that has MoE in every step of the VLM (image encoder, MLP and text decoder) and uses Mistral-7B for the decoder part 🤓
You can try it yourself here: shi-labs/CuMo-7b-zero
the authors firstly did pre-training of MLP with the by freezing the image encoder and text decoder, then they warmup the whole network by unfreezing and finetuning which they state to stabilize the visual instruction tuning when bringing in the experts. 🤓
the mixture of experts MLP blocks above are simply the same MLP blocks initialized from the single MLP that was trained during pre-training and fine-tuned in pre-finetuning.
it works very well (also tested myself) that it outperforms the previous sota of it's size LLaVA NeXt and IDEFICS2-8B in several benchmarks! 😍
Collections
21
spaces
96
Running
on
Zero
1
📚
Paligemma Doc
Try PaliGemma on document understanding tasks
Running
on
Zero
33
🌖
BLIP2 with transformers
BLIP2 (cutting edge image captioning) in 🤗transformers
Running
on
Zero
20
👀
Compare VLMs
Running
on
Zero
13
🦖🦉
GroundingDINO ⚔ OWL
Running
on
Zero
10
🦕🦖
GroundingSAM
Running
on
Zero
131
🔥
Llava Next
models
76
merve/checkpoint
Updated
merve/output8
Updated
merve/output4
Updated
merve/VeCap-DFN-h14
Zero-Shot Image Classification
•
Updated
•
5
merve/VeCap-DFN-l14
Updated
merve/VeCap-DFN-b16
Zero-Shot Image Classification
•
Updated
•
4
merve/VeCLIP-b16-100m
Zero-Shot Image Classification
•
Updated
•
4
merve/VeCLIP-b16-200m
Zero-Shot Image Classification
•
Updated
•
3
merve/VeCLIP-b16-12m
Zero-Shot Image Classification
•
Updated
•
3
merve/VeCLIP-b16-3m
Zero-Shot Image Classification
•
Updated
•
2
datasets
21
merve/faiss_embeddings
Updated
merve/pokemon-ds-embeddings
Viewer
•
Updated
•
3
merve/tr-h4-norobots
Updated
•
7
•
10
merve/lego_sets_latest
Viewer
•
Updated
•
12
•
1
merve/ai-tube-dummy
Updated
merve/my-blog-images
Viewer
•
Updated
•
1
merve/turkish_instructions
Viewer
•
Updated
•
849
•
31
merve/ner-flags
Updated
merve/xlm-roberta-large-df
Viewer
•
Updated
merve/parsed-dataset-xlm-roberta
Viewer
•
Updated