The paper presents a novel method for generating PBR images directly, avoiding the challenges and inaccuracies associated with RGB generation and extraction. Existing methods for fine-tuning the weights of a base RGB model are unsuitable for PBR image generation due to a lack of data and the high dimensionality of the output modalities. The proposed solution retains a frozen RGB model while training a new PBR model, using a novel cross-network communication paradigm. This method overcomes issues of data sparsity and remains compatible with techniques such as IPAdapter.
Publication date: 8 Feb 2024
Project Page: https://unity-research.github.io/holo-gen
Paper: https://arxiv.org/pdf/2402.05919
