DreamText: Harnessing text descriptions as an intermediate step for 3D reconstruction

Abstract

[en] The field of 3D generation, a rapidly emerging domain within generative AI, holds immense potential for various applications in fields such as architecture, product design, marketing, entertainment, and even in the novel realm of virtual reality. Enhancing 3D technologies bears significant utility in fostering society development and serves as a captivating and intellectually stimulating field of study, offering intriguing challenges and opportunities for innovative advancements. In this dissertation, we introduce DreamText, an innovative Image to 3D generative model that harnesses text descriptors as an intermediate step for 3D reconstruction. Our proposed method effectively learns to describe objects within images, capturing crucial object details while disregarding extraneous contextual information such as lighting, point of view, or specific arrangements. This learned information serves as the foundation for generating compelling and novel views of the object, subsequently facilitating the creation of a comprehensive and accurate 3D reconstruction. Remarkably, our approach achieves high quality results, surpassing current state-of-the-art methodologies like RealFusion (CVPR2023)[1] in several test cases. Concrete evidence of our results can be observed in the following link. Furthermore, we present FitFusion, which leverages the knowledge of a pretrained image generative model, Stable Diffusion, to train a Neural Radiance Field capable of generating 3D models when provided with image data during training. This concept stems from a comprehensive analysis and understanding of a previous model called Stable DreamFusion[2], combined with meticulous parameter tuning that culminates in improved outcomes. This project entails extensive mathematical and experimental analysis of cutting-edge models, encompassing a comprehensive understanding of their intricate details.