ToonComposer: AI Generation Tool to Simplify Cartoon Animation Production

ToonComposer is a tool developed by Tencent's ARC Lab that uses generative AI technology to streamline the traditional cartoon and animation production process. In traditional animation production, artists need to manually draw transitions between key frames (a process known as "intermediate frame" generation) and color each frame, which is both time-consuming and labor-intensive.ToonComposer's goal is to automate this repetitive manual labor. Users only need to provide a few key color images and some simple sketches as a guide, and the AI model will automatically generate all the transition animations between them, complete with coloring, resulting in a smooth cartoon video. This tool greatly shortens the animation production cycle, allowing creators to put more energy into the core creative and story design, rather than tedious post-production. The entire project is open source, with code and pre-trained models publicly released, and a web-based interactive interface for direct user experience and use.

Function List

• Automatic generation of intermediate frames: The user only needs to provide start and end keyframe images or sketches, and the model can automatically generate intermediate transition animations.
• automatic coloring: The model can be automatically colored for the entire animation sequence based on user-supplied colored keyframes.
• Text Tip Guides: Supports the use of natural language text (Prompt) to describe the desired animation content or style.
• Sketch Guidance Control: Allows the user to insert sketches at specific frame locations for more precise control of the character's movement and form.
• Dynamic area control: Users can specify which areas of the frame require more dynamic changes by drawing "motion masks" that allow the model to generate freer movements in those areas.
• Multiple resolution support: The model supports the generation of videos with different resolutions, such as480pcap (a poem)608pThe
• parameterization: A variety of parameter tuning options, such as CFG Scale (Classifier Free Guidance Scale) and Residual Scale, allow users to fine-tune the details of the generated video.
• Localized Deployment: A complete installation guide is provided to support getting up and running on your local computer.
• Web Interactive Interface: A user-friendly web interface was built with the Gradio library that allows users to upload clips and generate videos directly from their browsers.

Using Help

ToonComposer provides a locally deployed web application that allows users to run it on their own computers. The following is a detailed installation and usage procedure.

Installation process

1. Clone Code Repository
   First, you need to clone the ToonComposer source code locally from GitHub. Open a terminal (command line tool) and enter the following command:

   git clone https://github.com/TencentARC/ToonComposer
   

   Then go to the project directory:

   cd ToonComposer
   

2. Create and activate the Conda environment
   To avoid dependency conflicts with other Python projects on your computer, it is recommended to create a separate environment using Conda.

   conda create -n tooncomposer python=3.10 -y
   

   After the environment has been created successfully, activate this new environment:

   conda activate tooncomposer
   

3. Installation of dependent libraries
   Dependency libraries required by the project are documented in therequirements.txtfile. Use thepipcommand to perform the installation. In addition, a separate installation of theflash-attnCoop.

   pip install -r requirements.txt
   pip install flash-attn==2.8.2 --no-build-isolation
   

   take note of: According to the official documentation, this configuration is currently only tested on Linux.

4. Download model weights
   ToonComposer relies on two core models: one is the base modelWan2.1 I2VThe other isToonComposerOwn model weights. When you run the app for the first time, the program will automatically check the local cache and if the model does not exist, it will automatically download it from the Hugging Face Hub. This process may take some time because the model file is quite large.

Launching the Web Application

After completing the installation, run the following command in the terminal to start the web application:

python app.py

After a successful launch, you'll see output similar to the following in the terminal, signaling that the application is being run locally in the7860running on the port:

Running on local URL:  http://127.0.0.1:7860

Now, open your browser and visit this address (http://localhost:7860) to see the ToonComposer interface.

Functional operation flow

Once you enter the web interface, you will see several main input areas. Here are the exact steps to generate an animated video:

1. Setting Basic Parameters
   • Text Prompt: In the text box at the top, enter text that describes what you want to animate. For example, "A boy running in the wind".
   • Number of output frames: Set how many total frames you want the generated video to have. The more frames, the longer the video will be.
   • Resolution: Selects the resolution of the video, the default is480pThe
2. Upload keyframe color images
   This is the most critical step. You need to provide at least one color image as a starting point for the animation.
   • This chart is usually placed in the first0Frames.
   • Click on the upload area and select your color image file. This image sets the tone for the style, character and background of the entire animation.
3. Uploading keyframe sketches
   In order to control the movement during the middle of the animation, you can add sketches at different frame positions.
   • For example, if you want a character to do a specific action at frame 10, you can draw a sketch of that action and upload a designation for it to be used in the first10Frames.
   • You can upload multiple sketches to control different nodes of the animation.
4. (Optional) Draw Motion Mask
   When uploading a sketch, you can use the brush tool provided in the interface to paint black areas on the sketch. These black areas are called "Motion Masks" and tell the model that "these areas can move more freely and exaggeratedly". This is very useful for creating effects such as flowing hair, swinging clothes, etc.
5. Adjusting advanced parameters
   • CFG Scale: This parameter controls how well the generated content conforms to your textual cues. The higher the value, the more closely the AI will generate your textual descriptions, but possibly at the expense of some creativity.
   • Position-aware residual scale: This parameter is used to adjust the detail and stability of the generated video. You can try different values to see the effect.
6. Generate Video
   When all settings are complete, click the "Generate" button. The model will start processing your input and generate a video. The status of the process will be shown in the bottom right corner of the interface, and when finished, the generated video will be played and available for download here.

application scenario

1. animation
   For professional animation studios or individual animators, ToonComposer can be an efficient tool in the production pipeline. It can quickly generate intermediate frames for animation, freeing animators from heavy, repetitive labor so that they can devote more time to more creative tasks such as storyboarding, character design, and keyframing.
2. game development
   In 2D game development, a large number of animation frames are required for character movements such as walking, attacking, and casting spells. With ToonComposer, developers can quickly generate complete action sequences with a small number of keyframes, effectively reducing the cost and time of art resource production.
3. Education and training
   For students or beginners who are learning animation production, ToonComposer provides an intuitive way to understand the concept of keyframes and intermediate frames. They can quickly see the final animation effect by experimenting with different combinations of keyframes, speeding up the learning process.
4. content creation
   Social media content creators, VTuber or video bloggers can utilize ToonComposer to quickly create short animated clips or animated emoji packages to enrich the form of their content and increase the fun and appeal of their videos.

QA

1. Which operating systems does ToonComposer support?
   According to the official GitHub page, the current installation configuration has been tested primarily on Linux. While it may also work on Windows or macOS, it may encounter unknown dependency library compatibility issues.
2. How good of a graphics card do I need to run ToonComposer?
   ToonComposer is based on large generative AI models that require high computing resources. While there is no official minimum configuration given, running such models usually requires an NVIDIA graphics card with large video memory (e.g., RTX 30-series or 40-series) to get a good performance and experience.
3. What should I do if the generated video is not satisfactory?
   Since this is a generative model, the results may be different each time it is generated. If you are not satisfied with the results, you can try the following methods to make adjustments: changing the random seed (Seed), adjusting parameters such as CFG Scale, adding or modifying keyframe sketches to provide clearer instructions for movement, or optimizing your text prompts.
4. Is this tool free?
   Yes, ToonComposer is an open source project, and its source code and model weights are freely available and usable, subject to the stipulations in the LICENSE file provided with the project.