Cartoonize Image Diffusion

Summary

• Introduction
• Parameters
• Examples
• Usage for developers
• Reference
• License
• Citation

Introduction

This pipeline is a customized version of Stable Diffusion (v1.5) that has been specifically tuned for instruction-based tasks. It has undergone fine-tuning using the existing InstructPix2Pix checkpoints to enhance its performance.

The motivation for this pipeline is derived from both FLAN and InstructPix2Pix. The primary objective is to enhance the capabilities of Stable Diffusion in effectively following specific instructions that involve image transformation operations. To achieve this, the pipeline involves creating an instruction-prompted dataset (as explained in our blog) and subsequently employing InstructPix2Pix-style training methodologies. The ultimate goal is to improve the performance of Stable Diffusion in accurately executing image transformation tasks based on explicit instructions.

Parameters

Inputs

• image - (image -.png|.jpg|.jpeg): The input of the model is an image, which can be an image of an object, or a landscape. It is provided by the user for the purpose of transforming it into a cartoonized version.

Output

• output - (image -.png): The output of the model is an image that has been processed by the model to resemble a cartoonized version.

Examples

image	output

Usage for developers

Please find below the details to track the information and access the code for processing the model on our platform.

Requirements

python~=3.10
torch~=2.0.0
diffusers

Code based on AIOZ structure

import torch, os
from diffusers import StableDiffusionInstructPix2PixPipeline
from diffusers.utils import load_image

...
def do_ai_task(
        image: Union[str, Path],
        model_storage_directory: Union[str, Path],
        device: Literal["cpu", "cuda", "gpu"] = "cpu",
        *args, **kwargs) -> Any:
        
        model_id = os.path.abspath(model_storage_directory + "...")
        pipeline = StableDiffusionInstructPix2PixPipeline.from_pretrained(model_id, 
                                                    torch_dtype=torch.float16, 
                                                    use_auth_token=True).to(device)

        image = load_image(image)
        image = pipeline(prompt="Cartoonize the following image", image=image).images[0]
        image.save("output.png")

        output = open("output.png", "rb")  # io.BufferedReader
    return output

Reference

This repository is based on and inspired by Instruction-tuned Diffusion Models's work. We sincerely appreciate their generosity in sharing the code.

License

We respect and comply with the terms of the author's license cited in the Reference section.

Citation

FLAN

@inproceedings{
    wei2022finetuned,
    title={Finetuned Language Models are Zero-Shot Learners},
    author={Jason Wei and Maarten Bosma and Vincent Zhao and Kelvin Guu and Adams Wei Yu and Brian Lester and Nan Du and Andrew M. Dai and Quoc V Le},
    booktitle={International Conference on Learning Representations},
    year={2022},
    url={https://openreview.net/forum?id=gEZrGCozdqR}
}

InstructPix2Pix

@InProceedings{
    brooks2022instructpix2pix,
    author     = {Brooks, Tim and Holynski, Aleksander and Efros, Alexei A.},
    title      = {InstructPix2Pix: Learning to Follow Image Editing Instructions},
    booktitle  = {CVPR},
    year       = {2023},
}

Instruction-tuning for Stable Diffusion blog

@article{
  Paul2023instruction-tuning-sd,
  author = {Paul, Sayak},
  title = {Instruction-tuning Stable Diffusion with InstructPix2Pix},
  journal = {Hugging Face Blog},
  year = {2023},
  note = {https://huggingface.co/blog/instruction-tuning-sd},
}