ComfyUI-3D-Pack

An extensive node suite that enables ComfyUI to process 3D inputs (Mesh & UV Texture, etc) using cutting edge algorithms (3DGS, NeRF, Differentiable Rendering, SDS/VSD Optimization, etc.)

Features — Roadmap — Install — Run — Tips — Supporters

Note: this project is still a WIP

Currently support:

• For use case please check Example Workflows. [Last update: 12/04/2024]

  • Note: you need to put Example Inputs Files & Folders under ComfyUI Root Directory\ComfyUI\input folder before you can run the example workflow
  • tripoSR-layered-diffusion workflow by @Consumption

• CRM: thu-ml/CRM

  • Three stages pipeline:

    1. Image to 6 multi-view images (Front, Back, Left, Right, Top & Down)
    2. Image & 6 multi-view images to 6 same views CCMs (Canonical Coordinate Maps)
    3. 6 multi-view images & CCMs to 3D mesh

  • Note: For low vram pc, if you can't fit all three models for each stages into your GPU memory, then you can divide those three stages into different comfy workflow and run them separately

    2024-03-12.22-05-10.mp4

• TripoSR: VAST-AI-Research/TripoSR | ComfyUI-Flowty-TripoSR

  • Generate NeRF representation and using marching cube to turn it into 3D mesh

    2024-03-05.22-38-36.mp4

• Wonder3D: xxlong0/Wonder3D

  • Generate spatial consistent 6 views images & normal maps from a single image

• Large Multiview Gaussian Model: 3DTopia/LGM

  • Enable single image to 3D Gaussian in less than 30 seconds on a RTX3080 GPU, later you can also convert 3D Gaussian to mesh

    2024-02-08.23-36-31.mp4

• Triplane Gaussian Transformers: VAST-AI-Research/TriplaneGaussian

  • Enable single image to 3D Gaussian in less than 10 seconds on a RTX3080 GPU, later you can also convert 3D Gaussian to mesh

    2024-02-08.23-57-37.mp4

• Preview 3DGS and 3D Mesh: 3D Visualization inside ComfyUI:

  • Using gsplat.js and three.js for 3DGS & 3D Mesh visualization respectively

  • Custumizable background base on JS library: mdbassit/Coloris

    2024-02-04.19-20-17.mp4

• Stack Orbit Camera Poses: Automatically generate all range of camera pose combinations

  • You can use it to conditioning the StableZero123 (You need to Download the checkpoint first), with full range of camera poses in one prompt pass

  • You can use it to generate the orbit camera poses and directly input to other 3D process node (e.g. GaussianSplatting and BakeTextureToMesh)

  • Example usage:

    
    

  • Coordinate system:

    • Azimuth: In top view, from angle 0 rotate 360 degree with step -90 you get (0, -90, -180/180, 90, 0), in this case camera rotates clock-wise, vice versa.
    • Elevation: 0 when camera points horizontally forward, pointing down to the ground is negitive angle, vice versa.

• FlexiCubes: nv-tlabs/FlexiCubes

  • Multi-View depth & mask (optional normal maps) as inputs

  • Export to 3D Mesh

  • Usage guide:

    • voxel_grids_resolution: determine mesh resolution/quality
    • depth_min_distance depth_max_distance : distance from object to camera, object parts in the render that is closer(futher) to camera than depth_min_distance(depth_max_distance) will be rendered with pure white(black) RGB value 1, 1, 1(0, 0, 0)
    • mask_loss_weight: Control the silhouette of reconstrocted 3D mesh
    • depth_loss_weight: Control the shape of reconstrocted 3D mesh, this loss will also affect the mesh deform detail on the surface, so results depends on quality of the depth map
    • normal_loss_weight: Optional. Use to refine the mesh deform detail on the surface
    • sdf_regularizer_weight: Helps to remove floaters in areas of the shape that are not supervised by the application objective, such as internal faces when using image supervision only
    • remove_floaters_weight: This can be increased if you observe artifacts in flat areas
    • cube_stabilizer_weight: This does not have a significant impact during the optimization of a single shape, however it helps to stabilizing training in somecases
    2024-04-12-16-21-24.mp4

• Instant NGP: nerfacc

  • Multi-View images as inputs
  • Export to 3D Mesh using marching cubes

• 3D Gaussian Splatting

  • Improved Differential Gaussian Rasterization
  • Better Compactness-based Densification method from Gsgen,
  • Support initialize gaussians from given 3D mesh (Optional)
  • Support mini-batch optimazation
  • Multi-View images as inputs
  • Export to standard 3DGS .ply format supported

• Gaussian Splatting Orbit Renderer

  • Render 3DGS to images sequences or video, given a 3DGS file and camera poses generated by Stack Orbit Camera Poses node

• Fitting_Mesh_With_Multiview_Images

  • Bake Multi-View images into UVTexture of given 3D mesh using Nvdiffrast, supports:
  • Export to .obj, .ply, .glb

• NeuS

  • Fit a coarse mesh from sparse multi-view images & normal maps, as little as 4 to 6 views, pretty good at reconstruct the shape from reference images but texture lacking details.

• Deep Marching Tetrahedrons

  • Allow convert 3DGS .ply file to 3D mesh
    Note: I didn't spent time to turn the hyperprameters yet, the result will be improved in the future!

• Save & Load 3D file

  • .obj, .ply, .glb for 3D Mesh
  • .ply for 3DGS

• Switch Axis for 3DGS & 3D Mesh

  • Since different algorithms likely use different coordinate system, so the ability to re-mapping the axis of coordinate is crucial for passing generated result between differnt nodes.

• Customizable system config file

  • Custom clients IP address

Roadmap:

• Add DMTet algorithm to allow conversion from points cloud(Gaussian/.ply) to mesh (.obj, .ply, .glb)

• Integrate Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers

• Add interactive 3D UI inside ComfuUI to visulaize training and generated results for 3D representations

• Add a new node to generate renderer image sequence given a 3D gaussians and orbit camera poses (So we can later feed it to the differentiable renderer to bake it onto a given mesh)

• Integrate LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation

• Add camera pose estimation from raw multi-views images

• Add & Improve a few best MVS algorithms (e.g instant-ngp, NeuS2, GaussianPro, etc.)

• Improve 3DGS/Nerf to Mesh conversion algorithms:

  • Support to training DMTet with images(RGB, Alpha, Normal Map)
  • Find better methods to converts 3DGS or Points Cloud to Mesh (Normal maps reconstruction maybe?)

• Add a general SDS/ISM Optimization algorithm to allow training 3D representations with diffusion model

  • Need to do some in-depth research on Interval Score Matching (ISM), since math behind it makes perfect sense and also there are so many ways we could improve upon the result obtained from LucidDreamer
  • On Hold since runtime cost to generate an is too big (3+hours for an average RTX GPU like 3080)

Install:

[IMPORTANT!!!]
Currently this package is only been tested in following setups:

• Windows 10/11 (Tested on my laptop)
• Ubuntu 23.10 (Tested by @watsieboi)
• ComfyUI python_embed/Miniconda/Conda Python 3.11.x
• Torch version >= 2.1.2+cu121

Assume you have already downloaded ComfyUI & Configed your CUDA environment.

Install Method 0: Directly inside ComfyUI Windows Python Embeded Environment

Currently support: (python3.10/3.11/3.12 cuda12.1)

First install Visual Studio Build Tools 2022/2019 with Workloads: Desktop development with C++ (There are a few JIT torch cpp extension that builds in runtime)

• Alternatively, according to @doctorpangloss, you can setup the c++/cuda build environments in windows by using chocolatey

Go to the Comfy3D root directory: ComfyUI Root Directory\ComfyUI\custom_nodes\ComfyUI-3D-Pack and run:

# Run .bat with python version corresponding to the version of your ComfyUI python environment

# install_windows_portable_win_py310_cu121.bat
install_windows_portable_win_py311_cu121.bat
# install_windows_portable_win_py312_cu121.bat

Install Method 1: Using Miniconda(Works on Windows & Linux & Mac)

Note: In some edge cases Miniconda fails Anaconda could fix the issue

Setup with Miniconda:

First download Miniconda (One of the best way to manage a clean and separated python envirments)

Then running following commands to setup the Miniconda environment for ComfyUI:

# Go to your Your ComfyUI root directory, for my example:
cd C:\Users\reall\Softwares\ComfyUI_windows_portable 

conda create -p ./python_miniconda_env/ComfyUI python=3.11

# conda will tell what command to use to activate the env
conda activate C:\Users\reall\Softwares\ComfyUI_windows_portable\python_miniconda_env\ComfyUI

# update pip
python -m pip install --upgrade pip

# You can using following command to installing CUDA only in the miniconda environment you just created if you don't want to donwload and install it manually & globally:
# conda install -c "nvidia/label/cuda-12.1.0" cuda-toolkit

# Install the main packahes
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121

pip install -r ./ComfyUI/requirements.txt

# Then go to ComfyUI-3D-Pack directory under the ComfyUI Root Directory\ComfyUI\custom_nodes for my example is:
cd C:\Users\reall\Softwares\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-3D-Pack

• Alternatively you can check this tutorial: Installing ComfyUI with Miniconda On Windows and Mac

Install with Miniconda:

Go to the Comfy3D root directory: ComfyUI Root Directory\ComfyUI\custom_nodes\ComfyUI-3D-Pack and run:

install_miniconda.bat

Just in case install_miniconda.bat may not working in your OS, you could also run the following commands under the same directory: (Works with Linux & macOS)

pip install -r requirements.txt

pip install -r requirements_post.txt

Plus:

• For those who want to run it inside Google Colab, you can check the install instruction from @lovisdotio

Install and run with docker:

Gpu support during Docker build time is required to install all requirenents. On Linux host you could setup nvidia-container-runtime. On Windows it is quite different and not checked at moment.

Linux setup:

1. Install nvidia-container-runtime:

   sudo apt-get install nvidia-container-runtime

2. Edit/create the /etc/docker/daemon.json with content:

   {
       "runtimes": {
           "nvidia": {
               "path": "/usr/bin/nvidia-container-runtime",
               "runtimeArgs": []
           } 
       },
       "default-runtime": "nvidia" 
   }

3. Restart docker daemon:

   sudo systemctl restart docker

Finally build and run docker container with:

docker build -t comfy3d . && docker run --rm -it -p 8188:8188 --gpus all comfy3d

Run:

Copy the files inside folder __New_ComfyUI_Bats to your ComfyUI root directory, and double click run_nvidia_gpu_miniconda.bat to start ComfyUI!

• Alternatively you can just activate the Conda env: python_miniconda_env\ComfyUI, and go to your ComfyUI root directory then run command python ./ComfyUI/main.py

Tips

• OpenGL world & camera coordinate system:

    World            Camera        
  
     +y              up  target                                              
     |               |  /                                            
     |               | /                                                
     |______+x       |/______right                                      
    /                /         
   /                /          
  /                /           
 +z               forward           

elevation: in (-90, 90), from +y to -y is (-90, 90)
azimuth: in (-180, 180), from +z to +x is (0, 90)

• Wonder3D world & camera coordinate system:

• Three.js coordinate system: (z-axis is pointing towards you and is coming out of the screen)

• If you encounter OpenGL errors (e.g., [F glutil.cpp:338] eglInitialize() failed), then set force_cuda_rasterize to true on corresponding node
• If after the installation, your ComfyUI get stucked at starting or running, you could following the instruction in following link to solve the problem: Code Hangs Indefinitely When Evaluating Neuron Models on GPU

Supporters

• MrNeRF