Meta 3D Gen: New Algorithm Converts Text Descriptions into 3D Models

Meta, the technology company, has recently released a new cutting-edge algorithm for converting text to 3D known as Meta 3D Gen (3DGen).

This new technology is designed to save time when using textual descriptions to produce character and prop models, and render scenes.

3D creation is one of the most complex and time-consuming activities involved in producing video games, augmented and virtual reality applications, and any other industry that requires special effects for its films. Meta takes social interactions a step further by offering AI avatars that also function as 3D artists to bring new realistic experiences focused on generating user-creatable 3D content.

How to use Meta AI in WhatsApp, Instagram, Facebook, and Messenger to Get Rapid Answers

2 Stage Process

3DGen is a two-stage process consisting of a first component for text-to-3D generation and a second component for text-to-texture generation to produce much better 3D for immersive content.

Stage I: 3D asset generation. Given a text prompt provided by the user, Stage I creates an initial 3D asset using our Meta 3D AssetGen (Siddiqui et al., 2024) model (AssetGen for short). This step produces a 3D mesh with texture and PBR material maps. The inference time is approximately 30 seconds.

Stage II, use case I: generative 3D texture refinement. Given a 3D asset generated in Stage I and the initial text prompt used for generation, Stage II produces a higher-quality texture and PBR maps for this asset and the prompt. It utilizes our text-to-texture generator, Meta 3D TextureGen (Bensadoun et al., 2024) (TextureGen for short). The inference time is approximately 20 seconds.

Stage II, use case 2: generative 3D (re)texturing. Given an untextured 3D mesh and a prompt describing its desired appearance, Stage II can also be used to generate a texture for this 3D asset from scratch (the mesh can be previously generated or artist-created). The inference time is approximately 20 seconds.

Meta has explored uses from its investors extensively to assess the utility and textual reproduction of 3DGen’s generations. They stand against existing public source industrial solutions for the task of text-to-3D asset conversion.

Thus, it can be seen that Meta’s 3DGen integrates Meta’s foundational generative models for text-to-3D generation with texture editing and material generation capabilities. By combining the strengths of AssetGen and TextureGen, 3DGen achieves high-quality 3D object synthesis from textual prompts in less than a minute. While the current integration of AssetGen and TextureGen is straightforward, it sets a promising research direction that builds on two thrusts: generation in view space and UV space, and end-to-end iteration over texture and shape generation

Meta Changes AI Label to ‘AI Info’ After Backlash from Photographers Over Misleading Tags