Generate Controlled Video with ComfyUI

Basic idea

Text-only video prompts give limited control over composition, camera angle, and subject motion. This workflow helps you guide the output step by step on your hardware platform: generate 3D scene assets, lock composition with depth-guided image generation, turn first and last frames into video, then upscale to 4K.

You start by generating a 3D scene to guide composition, turn that layout into photorealistic keyframes, use those images as first and last frames for video generation, then upscale with the RTX Video Super Resolution node in ComfyUI. The result is a high-resolution clip that follows your composition, camera angle, and subject motion.

What you'll accomplish

You'll run a local storyboard-to-video workflow on your hardware platform that can:

• Generate 3D scene assets from text (standalone or in Blender)
• Produce composition-locked first and last frames with depth-guided image generation (FLUX.1)
• Generate video between those frames with LTX-2.3 in ComfyUI
• Upscale the result toward 4K with RTX Video Super Resolution

Creators can use any stage alone. For the full pipeline, complete each stage before starting the next so system resources stay available.

What to know before starting

Required:

• Comfort with Windows desktop apps (Blender, ComfyUI, PowerShell or Command Prompt)
• Familiarity with generative AI concepts (prompts, CFG / guidance, diffusion or video models)

Optional:

• Prior ComfyUI experience (How to Get Started With Visual Generative AI on NVIDIA RTX PCs)
• Basic Blender scene layout (cameras, assets, animation scrubbing)

Supported hardware platforms

Use the matrix below to confirm your hardware platform, recommended default local settings, and whether multi-node applies.

Prerequisites

Hardware requirements

• Supported hardware platform — see Supported hardware platforms matrix above
• At least 16 GB dedicated GPU memory (NVIDIA GeForce RTX 5070 Ti, NVIDIA RTX PRO 2000 Blackwell, or higher recommended)
• 64 GB system RAM recommended

Software requirements

• Windows 11
• Blender (source workflow uses Blender 4.2 for asset import and Blender 4.5 LTS with the ComfyUI Blender AI node)
• ComfyUI, including access to the template browser and ComfyUI Manager
• Network access to download blueprint dependencies and models from the linked GitHub repositories
• Browser access to local UIs (for example, http://127.0.0.1:7860 for the 3D Object Generation blueprint)

Workflow components (install and setup live in the linked blueprints — this playbook does not ship a local install bundle):

Time & risk

• Estimated time: 18 MIN (first run is longer when downloading models and blueprint dependencies)
• Risk level: Medium
  • Large model and blueprint downloads can fail if network connectivity is unstable
  • GPU memory pressure can appear if other GPU workloads are running during image or video generation
  • FLUX.1 in the guided blueprint is non-commercial unless you have a commercial license from Black Forest Labs
• Rollback: Close Blender and ComfyUI, deactivate any conda environments you created for the blueprints, and delete local blueprint clones or generated assets you no longer need
• Last Updated: 08/03/2026
  • Storyboard-to-4K video workflow with 3D-guided composition, LTX-2.3 first/last frame generation, and RTX Video Super Resolution upscaling