OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens

Summary

• Introduction
• Parameters
• Usage for developers
• Reference
• License
• Citation

Introduction

OmniLottie is the first family of end-to-end multimodal Lottie generators that leverage pre-trained Vision-Language Models (VLMs), capable of generating complex and detailed Lottie animations from multi-modal instructions including texts, images, and videos. We also introduce MMLottie-2M, a multimodal dataset with two million richly annotated Lottie animations, along with a standardized evaluation protocol for multi-modal vector animation generation tasks.

Benchmark Overview

MMLottieBench contains 900 samples split into:

• Real split: 450 real-world Lottie animations
• Synthetic split: 450 synthetically generated samples

Each split contains 3 task types (150 samples each):

• Text-to-Lottie: Generate from text descriptions
• Text-Image-to-Lottie: Generate from image + text guidance
• Video-to-Lottie: Convert video to Lottie animation

Parameters

Inputs

• task_type - (text): The type of generation task. Supported values: Text-to-Lottie, Image-to-Lottie, Video-to-Lottie
• image_file- (text - .png|.jpg|.jpeg, optional): Path or URL to the input image file. Required when task_type = "Image-to-Lottie".
• video_file- (text - .mp4|.mov, optional): Path or URL to the input video file. Required when task_type = "Video-to-Lottie".
• text_description- (text- None):Optional guidance text for Image-to-Lottie tasks. Not used in Text-to-Lottie or Video-to-Lottie.
• text_prompt- (text): Main animation prompt. Used for Text-to-Lottie tasks.
• max_tokens- (integer): Maximum number of tokens used during generation. Range: 512 – 5856, step 256.
• use_sampling- (bool): Whether to enable sampling. Default: true. When false, greedy decoding is used and temperature/top_p/top_k values are ignored.
• temperature- (float): Controls randomness of generation. Range: 0.1 – 2.0, step 0.1. Higher values produce more diverse outputs.
• top_p- (float): Nucleus sampling parameter controlling probability mass. Range: 0.1 – 1.0, step 0.1.
• top_k - (integer): Limits sampling to the top-k most probable tokens. Range: 1 – 100, step 1.

Output

• temp_path - (file -.json): The generated Lottie animation in JSON format.

Usage for developers

Please find below the details to track the information and access the code for processing the model on our platform.

Requirements

pip install -r requirements.txt

Code based on AIOZ structure

import torch
from PIL import Image as PILImage

...
def do_ai_task(
        task_type: Union[str, str],
        image_file: Union[str, None],
        video_file: Union[str, None],
        text_description: Union[str, None],
        text_prompt: Union[str, str],
        max_tokens: Union[int, int],
        use_sampling: Union[bool, bool],
        temperature: Union[float, float],
        top_p: Union[float, float],
        top_k: Union[int, int],
        model_storage_directory: Union[str, Path],
        device: Literal["cpu", "cuda", "gpu"] = "cpu",
        *args, **kwargs) -> Any:
    """Define AI task: load model, pre-process, post-process, etc ..."""
    # Define AI task workflow. Below is an example
    model, processor, device = load_model_once(model_storage_directory)
    
    if task_type == "Text-to-Lottie":
        messages = build_messages("text", text_prompt=text_prompt)
    elif task_type == "Image-to-Lottie":
        image = load_image_from_file(image_file)
        image = image.resize((448, 448), PILImage.LANCZOS)
        desc = text_description if text_description else "A simple animation"
        messages = build_messages("image", text_prompt=desc, image=image)
    else:
        frames = load_frames_from_video(video_file, num_frames=8)
        messages = build_messages("video", video_frames=frames)

    inputs = prepare_inference_input(processor, messages, device)

    generated_ids = generate_lottie(
        model, inputs, max_tokens, device,
        use_sampling, temperature, top_p, top_k
    )

    del inputs
    if torch.cuda.is_available():
        torch.cuda.empty_cache()

    lottie_json = tokens_to_lottie_json(generated_ids)
    temp_path = save_json_to_temp(lottie_json)

    output_file = open(temp_path, "rb")  # io.BufferedReader
    return output_file

Reference

This repository is based on and inspired by OpenVGLab's work. We sincerely appreciate their generosity in sharing the code.

License

We respect and comply with the terms of the author's license cited in the Reference section.

Citation

@article{yang2026omnilottie,
  title={OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens},
  author={Yiying Yang and Wei Cheng and Sijin Chen and Honghao Fu and Xianfang Zeng and Yujun Cai and Gang Yu and Xinjun Ma},
  journal={arXiv preprint arxiv:2603.02138},
  year={2026}
}