Improve model card: Add paper abstract, code link, project page link, usage example and bibtex citation

README.md

@@ -1,17 +1,24 @@
 ---
-license: apache-2.0
 language:
 - en
-library_name: transformers
+library_name: diffusers
+license: apache-2.0
 pipeline_tag: text-to-image
 ---
 
-<br>
+# DiffBlender: Composable and Versatile Multimodal Text-to-Image Diffusion Models
 
-# DiffBlender Model Card
+This repository contains the models from our paper [**DiffBlender: Scalable and Composable Multimodal Text-to-Image Diffusion Models**](https://huggingface.co/papers/2305.15194).
 
-This repo contains the models from our paper [**DiffBlender: Scalable and Composable Multimodal Text-to-Image Diffusion Models**](https://arxiv.org/abs/2305.15194).
+[Code](https://github.com/sungnyun/diffblender)
+[Project Page](https://sungnyun.github.io/diffblender/)
 
+<p align="center">
+<img width="1369" alt="teaser" src="https://github.com/sungnyun/diffblender/raw/main/assets/fig1.png">
+</p>
+
+## Abstract
+In this study, we aim to enhance the capabilities of diffusion-based text-to-image (T2I) generation models by integrating diverse modalities beyond textual descriptions within a unified framework. To this end, we categorize widely used conditional inputs into three modality types: structure, layout, and attribute. We propose a multimodal T2I diffusion model, which is capable of processing all three modalities within a single architecture without modifying the parameters of the pre-trained diffusion model, as only a small subset of components is updated. Our approach sets new benchmarks in multimodal generation through extensive quantitative and qualitative comparisons with existing conditional generation methods. We demonstrate that DiffBlender effectively integrates multiple sources of information and supports diverse applications in detailed image synthesis.
 
 ## Model details
 
@@ -28,11 +35,34 @@ Apache 2.0 License
 **Where to send questions or comments about the model:**
 https://github.com/sungnyun/diffblender/issues
 
+## Quick Start
+Install the necessary packages with:
+```sh
+$ pip install -r requirements.txt
+```
 
-## Training dataset
-[Microsoft COCO 2017 dataset](https://cocodataset.org/#home)
+Download DiffBlender model checkpoint from this [Huggingface model](https://huggingface.co/sungnyun/diffblender), and place it under `./diffblender_checkpoints/`.    
+Also, prepare the SD model from this [link](https://huggingface.co/CompVis/stable-diffusion-v-1-4-original) (we used CompVis/sd-v1-4.ckpt).
+
+### Try Multimodal T2I Generation with DiffBlender
+```sh
+$ python inference.py --ckpt_path=./diffblender_checkpoints/{CKPT_NAME}.pth \
+                      --official_ckpt_path=/path/to/sd-v1-4.ckpt \
+                      --save_name={SAVE_NAME} 
+```
 
+Results will be saved under `./inference/{SAVE_NAME}/`, in the format as {conditions + generated image}.
 
-<br>
+## Training dataset
+[Microsoft COCO 2017 dataset](https://cocodataset.org/#home)
 
-More detials are in our project page, https://sungnyun.github.io/diffblender/.
+## Citation
+If you find our work useful or helpful for your R&D works, please feel free to cite our paper as below.
+```bibtex
+@article{kim2023diffblender,
+  title={DiffBlender: Scalable and Composable Multimodal Text-to-Image Diffusion Models},
+  author={Kim, Sungnyun and Lee, Junsoo and Hong, Kibeom and Kim, Daesik and Ahn, Namhyuk},
+  journal={arXiv preprint arXiv:2305.15194},
+  year={2023}
+}
+```