Generate 3D Models from AI Images with Advanced Agents

What this is, in plain English

This workflow describes an advanced method for creating 3D models using artificial intelligence. It starts by generating an image from a text description, then uses a specialized AI program, called an AI agent, to interpret that image and build a 3D model from it. The agent uses a “skill” that teaches it how to work with 3D modeling software like Blender.

This process is highly technical and relies on tools that are either hypothetical (like GPT-6 Astra and ChatGPT Images 2.5) or require significant setup and coding knowledge (like Codex with custom skill files). Because of this, there isn’t a simple, copy-paste recipe for a beginner to follow.

The core idea is to automate complex creative tasks, moving from a simple text idea to a visual image, and then to a fully realized 3D object, all with the help of AI. This pushes the boundaries of what AI can do in design and digital content creation.

What you can use it for

• Create custom game assets: Generate unique 3D models for video games based on specific themes or concepts.
• Design unique objects for virtual reality: Populate virtual worlds with custom-made 3D items that match a particular aesthetic.
• Generate prototypes for physical products: Quickly visualize and create basic 3D models of new product ideas for early-stage design reviews.
• Explore creative concepts: Experiment with different visual styles and themes by rapidly generating corresponding 3D models.

Tools you need

• GPT-6 Astra (paid): A hypothetical advanced AI model, likely from OpenAI, used here for complex reasoning and task execution within an AI agent.
• Blender (free): A powerful, open-source 3D creation suite used for modeling, animation, rendering, and more. The AI agent uses its capabilities to build the 3D model.
• ChatGPT Images 2.5 (freemium): A hypothetical version of ChatGPT’s image generation feature, used to create the initial image from a text prompt.
• Codex (paid): A hypothetical AI coding agent, likely from OpenAI, that can understand and generate code, and in this case, execute tasks using specific “skills” like 3D modeling.
• .blend URL Viewer (free): A custom web-based tool developed by the author to view .blend 3D model files directly in a web browser. Its homepage is the author’s main blog.

How it actually works

The process begins by using an AI image generator, like the hypothetical ChatGPT Images 2.5, to create a visual concept from a text description. For example, the author prompted it to generate a Fabergé egg themed after a TV show.

Once the image is generated, it is fed into an advanced AI agent, in this case, Codex running GPT-6 Astra. This agent is equipped with a special “skill file” that teaches it how to interact with Blender, the 3D modeling software. The author then gives the agent a prompt like, “Use your blender local skill to create a blender model of this faverge egg.”

The AI agent then processes the image and the prompt, using its Blender skill to generate one or more .blend files, which are the native files for Blender 3D models. This process can take a significant amount of time, as noted by the author (17 minutes and 51 seconds).

Finally, to view the generated 3D model, the author uses a custom tool called the “.blend URL Viewer.” This allows the .blend file to be displayed directly in a web browser. The exact steps for setting up Codex with a custom skill file are not detailed in the source and would require advanced technical knowledge.

Words you’ll see, explained

• AI agent: A computer program that uses artificial intelligence to perform tasks, often by interacting with other tools or systems automatically.
• Blender: A free and open-source software application used for 3D modeling, animation, visual effects, rendering, and more.
• .blend file: The standard file format used by Blender to save all the data related to a 3D project, including models, textures, and animations.
• Skill file: A custom set of instructions or a program that extends the capabilities of an AI agent, allowing it to perform specific, complex tasks, such as using a particular software like Blender.

Original source

This concept was shared by Simon Willison on his blog. He detailed his experiment with using advanced AI models and agents to generate 3D models from AI-created images, inspired by Imperial Fabergé Easter eggs.

Notes & variations

• Do you even need this?: For most users, generating 3D models from scratch using advanced AI agents is not yet a practical or accessible solution. Simpler alternatives include using existing 3D model libraries (many are free), hiring a 3D artist, or using more user-friendly 3D design tools that don’t require coding or advanced AI setup.
• Free-tier limits: Advanced AI models like GPT-6 Astra and specialized AI agents like Codex are typically paid services, often requiring API keys and incurring costs based on usage. While Blender is free, the AI components are not.
• Common pitfall: Expecting perfect, ready-to-use 3D models directly from AI. AI-generated 3D models often require significant human refinement, editing, and optimization in Blender or other 3D software to meet specific quality or functional requirements.