Modify video areas and insert characters with ComfyUI

What this is, in plain English

This workflow uses ComfyUI, a powerful open-source tool, to modify specific parts of a video and insert new characters or elements. It works by letting you “mask” (select) an area of a video, then using advanced AI models like Stable Diffusion to generate new content only within that masked area. This gives you very precise control over video edits, allowing you to change backgrounds, add objects, or place characters into existing footage.

This is an advanced workflow because it requires setting up ComfyUI on your own computer, which involves installing software, downloading large AI models, and often needs a powerful graphics card (GPU). It’s not a simple web tool you can use on a phone. The exact steps can also change as the tools update.

What you can use it for

• Add characters to videos: Insert new people or animated figures into existing video scenes.
• Change backgrounds: Select and replace the background of a video without affecting the main subject.
• Modify specific objects: Alter or remove objects within a video frame, like changing a car’s color or removing a sign.
• Create visual effects: Generate unique visual effects that blend seamlessly into your video content.

Tools you need

• ComfyUI (free): A powerful, open-source tool that lets you build custom AI image and video generation workflows using a visual node-based interface.
• Stable Diffusion models (free): These are the AI models that ComfyUI uses to generate images and video. You will need to download specific models for video editing.
• After Effects (paid): A professional video editing software that can be used to create very precise masks (shapes that define which parts of a video to change). Note: ComfyUI also has a built-in mask creation tool, so After Effects is optional.

How it actually works

To use this workflow, you will generally follow these steps:

1. Set up ComfyUI: Install ComfyUI on your computer. This typically involves installing Python, Git, and then downloading the ComfyUI software from its GitHub repository. You will also need to install any custom nodes required by the specific workflow.
2. Download the workflow: Get the specific ComfyUI workflow file (usually a .json file) from the author’s GitHub repository. This file contains the instructions for how the nodes are connected.
3. Download AI models: Obtain the necessary Stable Diffusion models for video generation. These are often large files (several gigabytes) and need to be placed in the correct folders within your ComfyUI installation.
4. Load the workflow: Open ComfyUI, load the downloaded workflow file, and ensure all required custom nodes and models are correctly installed and recognized.
5. Prepare your video and mask: Upload the video you want to edit. Then, create a mask either directly within ComfyUI using its mask creation node, or by uploading a mask you made in another program like After Effects. The mask tells the AI exactly which part of the video to change.
6. Configure and run: Adjust the settings and prompts within the ComfyUI workflow to describe what you want to insert or change in the masked area. Then, run the workflow to generate the modified video. The exact instructions for setting up and running this specific workflow, including links to the required files and detailed tutorials, are available in the author’s GitHub repository and linked Reddit posts.

Words you’ll see, explained

• ComfyUI: An open-source program that lets you build AI workflows by connecting different “nodes” (small functions) visually, like building with LEGO blocks.
• Stable Diffusion: A type of AI model that can create images and videos from text descriptions or by modifying existing ones.
• Node: A small, self-contained function within ComfyUI that performs a specific task, like loading an image, applying a mask, or generating AI content.
• Workflow: A complete sequence of connected nodes in ComfyUI that performs a specific task, like the video modification described here.
• Mask: A black and white image or shape that tells the AI which specific areas of your video or image to focus on or change, and which areas to leave alone.
• GPU (Graphics Processing Unit): A specialized computer chip that is very good at handling the complex calculations needed for AI tasks, making them run much faster.

Original source

This advanced video modification workflow was shared by Reddit user /u/roychodraws on the r/StableDiffusion subreddit. The author provides a specific ComfyUI workflow file and tutorials to help users achieve precise control over inserting characters into videos.

Notes & variations

• Do you even need this? For simpler video edits like basic cuts, adding text, or applying filters, free mobile video editing apps (e.g., CapCut, InShot) or online tools might be sufficient and much easier to use. This ComfyUI workflow is for advanced, AI-powered content generation within videos, requiring significant technical setup.
• Hardware and data limits: Running ComfyUI and Stable Diffusion models requires a powerful computer, ideally with a dedicated GPU (graphics card). The AI models themselves are very large downloads (multiple gigabytes), which can be costly on limited data plans. Processing videos also uses a lot of computing power and can take a long time.
• Common pitfall: The most common challenge is setting up ComfyUI correctly, installing all the necessary custom nodes, and downloading the right AI models. Errors often occur if files are in the wrong place or if your computer’s drivers are not up to date. Always follow the author’s specific setup instructions carefully.