The article introduces Paint-it, a method for synthesizing high-quality texture maps for 3D meshes using text descriptions. It utilizes a synthesis-through-optimization approach, leveraging Score-Distillation Sampling (SDS). However, direct application of SDS can yield poor texture quality due to its noisy gradients. To address this, the authors propose Deep Convolutional Physically-Based Rendering (DC-PBR) parameterization, which re-parameterizes the physically-based rendering (PBR) texture maps with randomly initialized convolution-based neural kernels. This approach allows the system to filter out noisy signals from SDS and schedule the optimization curriculum according to texture frequency. The authors demonstrate the practicality of Paint-it by synthesizing high-quality texture maps for large-scale mesh datasets and showing its application in real-time relighting and material control using a popular graphics engine.
Publication date: 18 Dec 2023
Project Page: https://kim-youwang.github.io/paint-it
Paper: https://arxiv.org/pdf/2312.11360
