As we enter the new era of spatial internet, 3D assets are becoming key building elements for virtual worlds, product presentations, games and entertainment. The need for content creation tools that can scale in terms of the quantity, quality, and diversity of 3D content is becoming more evident.
In 2022 we saw an explosion of generative AI tools, from creating articles and product descriptions to generating design concepts and art.
While there are so many applications for this technology, and it will undoubtedly grow exponentially in 2023, I want to focus on a topic closer to me as a professional – the 3D industry.
Let’s look what’s out there!
NVIDIA's 'Get3D'
NVIDIA is well-known in the 3D industry for their fantastic GPUs. Since I built my first rendering machine with legendary 1080ti’s, it was a game changer. The innovation that NVIDIA’s team brings to the industry is fascinating!
Trained using only 2D images, NVIDIA GET3D generates 3D meshes with complex topology, rich geometric details, and high-fidelity textures. The model can be easily scaled up to train with image resolution as high as 1024 × 1024, allowing it to learn high-quality geometric and texture details.
Source: NVIDIA
GET3D generates objects in the same format used by popular graphics software applications like Blender, allowing users to easily import the shapes into 3D rendering and game engines for further modification.
GET3D is not trained using raw meshes or pictures from the real world but rather using synthetic data.
Source: NVIDIA
The NVIDIA Research team believes future versions of GET3D could be trained on real-world data instead of synthetic datasets. The model can also be trained on various 3D shapes at once rather than focusing on one object category at a given time.
Like the majority of AI tools, ‘Get3D’ requires developer knowledge to be able to use and experiment with it. If you do want to try it, follow this Git link here: https://github.com/nv-tlabs/GET3D
'Point-E' by Open AI
Open AI was founded in 2015 by a group of entrepreneurs, researchers, and philanthropists, including Elon Musk and Sam Altman. I first discovered Open AI through their text-to-image generator Dall-E in late 2021, which was very impressive. Remember to mention probably the hottest AI tool today, ChatGPT, but this is a topic for another article.
Point-E is an extension of Dall-E, as it converts your worded prompt into an image using a text-to-image AI and then uses a second function to turn that image into a point cloud representing a 3D shape.
Source: Open AI
While this might look less exciting, the model is very light to use, and according to the paper, a computer with a single graphics card will only take 1-2 minutes to generate the result.
Source: Open AI
You can try Point-E below, just put your text prompt and check the result. This demo uses a small, worse-quality text-to-3D model to produce 3D point clouds directly from text descriptions.
Point-E has yet to launch officially through Open AI, but it is available via GitHub for those who are more technically minded.
Google's 'Dream-Fusion'
Dream Fusion is another generative AI tool capable of turning text prompts into digital 3D representations and is an expanded version of its predecessor Dream Fields released in 2021.
This AI tool requires no prior training – meaning DreamFusion can generate 3D representations of objects without hardly available 3D data.
Source: Google
After you put your text prompt the system will first use 2D images of an object generated by the Imagen text-to-image diffusion model to then generate a 3D model.
Source: Google
DreamFusion adds additional regularizers and optimization strategies to improve geometry. The 3D models generated by Dreamfusion are high quality and can be further edited/processed in common 3D software.
What's next?
Well, good question. As I mentioned before AI technology will grow exponentially in the near future, that’s a fact. I believe that generative AI tools will compliment creative industry rather than completely disrupting it. This technology is currently in it’s infancy and it will take time to see what it is actually capable of.
In my article I’ve written about the tools that are currently available for developers mainly, but generative AI tools for 3D creatives are also becoming available. I’m currently testing the latter ones and will share my findings soon. Stay tuned!