Artificial Intelligence doing CAD: ChatGPT, Codex and FreeCAD

06 Aug 2026 - tsp
Last update 06 Aug 2026
Reading time 19 mins

One of the most interesting ideas in the current wave of AI is not to keep large language models trapped inside text, code, and chat windows, but to give them the ability to act on the physical world. If an LLM can design something that can actually be manufactured, then it stops being merely a conversational or programming tool and starts becoming part of a real engineering toolchain. This is - besides the generation of new ideas and exploration of the unknown - one of the most exciting directions right now: let the model build real objects. Since I had already experience with letting an LLM orchestrator accessing and operating an (older) XL30 scanning electron microscope, I decided to take it a step further.

The first practical step toward that goal is giving the model access to CAD. Some models can already generate STL meshes directly, and sometimes they do surprisingly well, but in many cases they are essentially using a scripted CAD detour or emitting geometry without a strong editable model behind it. That is useful for quick experiments, but it leaves a lot of power on the table. If instead the model can control a full CAD system, it gains access not just to geometry creation but to the whole surrounding ecosystem: parametric modeling, exact BREP data, custom geometric calculations, rendering, visual inspection, simulation workflows, and the many workbenches that already exist. More importantly, it gains access to another independent representation of the design that can later be compared against mathematical reasoning, rendered images, simulations, and ultimately the manufactured part itself.

This is also a very natural route toward autonomous manufacturing. I touched on some of that fascination in my The Hum That Never Sleeps article: once tools begin extending themselves into more and more of the real production chain, the step from automation to partial autonomy gets surprisingly small. We are not at fully autonomous factories yet, but one can already see the shape of it. An LLM that can design parts inside CAD, validate them, pass them to a slicer, queue them for printing and eventually hand them to further assembly stages is no longer just generating content. It is participating in manufacturing.

And honestly, I think it is amazing how far this has already gone. Not with local models yet, unfortunately. At least in my experiments, the current locally running models are not there for these complex multi-hour engineering tasks. I hope we get there, because this is exactly the kind of capability I would love to run fully on my own hardware. But with the currently strongest hosted models, this has become very real.

The Idea

What I wanted to try was simple in principle: give an LLM agent direct access to a real CAD package and see whether it can create mechanically meaningful geometry on its own. Not toy boxes or decorative shapes, but actual parts that can be printed and used.

For this I used FreeCAD. That choice is important. FreeCAD is not just a mesh generator, it provides parametric modeling, sketch-based workflows, solid modeling on top of exact boundary representations, Python programmability, a wide range of workbenches and optional simulation-oriented extensions - and it is open source so you can tap into every function it offers. If an LLM gets access to that environment, it is suddenly no longer restricted to guessing geometry. It can work with a mature engineering system.

This matters for several reasons:

In other words: instead of teaching an LLM a tiny fake CAD language, one can hand it the whole machine shop of a modern CAD system.

The FreeCAD MCP Approach

To make this practical I built, with substantial help from Codex, a small bridge service that plugs into a running FreeCAD session and exposes that capability through MCP. MCP was choosen over scripting, GUI automation or custom RPC due to MCP providing a standardized tool interface that multiple different LLM systems can consume without requiring model specific integrations (note that one still has to tune the skills and promps massively to match the given LLM in the end). Because FreeCAD embeds Python, the bridge runs inside FreeCAD’s Python runtime and therefore has full access to the application state, document model, GUI state, geometry objects, workbenches, and Python APIs. This is a huge advantage. The bridge is not screen-scraping or performing UI automation from the outside. It is operating from inside the CAD system.

The setup is split into two pieces:

The MCP server supports both local stdio transport and a remote HTTP transport. The remote HTTP mode is useful for remote orchestrators and also for attaching the same tooling to the web version of ChatGPT. In that browser-based setup, a reverse proxy in front of the MCP endpoint is practically required, both to expose the service cleanly under HTTPS and to handle the path mapping that the web client expects. One should obviously not expose such a service carelessly; even for experimentation it deserves proper authentication, TLS, and a deliberately constrained deployment.

The project comes with two files that are particularly important:

That second point turned out to matter much more than one might expect. The raw tools alone are not enough. The model needs a working discipline: inspect before editing, prefer parametric and mutable features, validate sketches before downstream operations, inspect the document tree after changes, and explicitly perform visual checks instead of assuming success from tool output alone.

This discipline is what made the difference between “it sometimes kind of works” and “it can eventually complete real tasks correctly”.

FreeCAD running side by side with codex

Engineering Requires Discipline

The lesson was that giving an LLM access to CAD tools is, as mentioned, only part of the solution. The much larger challenge is teaching it an engineering workflow. Unlike conversational tasks, CAD models evolve over many editing operations, and seemingly minor mistakes can propagate through the feature tree until they become difficult to diagnose. The operating discipline encoded in LLM_USAGE.md and the project-specific mechanical design rules therefore became almost as important as the tools themselves.

The workflow deliberately treats every modeling step as something that must be verified before building upon it. Sketches are checked for solver consistency and closed profiles before creating downstream features. After each modification, the document tree is re-inspected to ensure the expected feature hierarchy exists. The view is then prepared explicitly, unnecessary objects are hidden, and screenshots are captured for visual validation. Mechanical constraints receive a similar level of scrutiny: holes are checked to ensure they actually pass through their intended targets, movable parts are examined over their complete range of motion instead of only in their neutral position, fasteners are verified for tool access and assembly order and expensive operations such as threaded fasteners or gears are initially represented by simplified proxy geometry to make collision detection and iteration practical.

This layered verification strategy turned out to be crucial. Rather than assuming that a successful API call implies a correct mechanical result, the agent continuously validates the model from several independent perspectives: geometric consistency, feature-tree integrity, visual appearance, manufacturability, and finally assemblability. Individually none of these checks is sufficient, but together they make long autonomous design sessions surprisingly robust.

Looking back, I no longer think the decisive ingredient was giving the model access to FreeCAD. The real breakthrough was giving it access to multiple independent representations of the same object. Every representation exposes different classes of mistakes. Mathematical reasoning catches inconsistencies in dimensions and formulas, the CAD model verifies geometric construction, rendered images reveal spatial relationships and collisions that are difficult to infer symbolically, while mechanical reasoning and assembly validation expose problems that are invisible in pure geometry. The model continually moves between these representations until they all describe the same object consistently.

That iterative cross-validation feels surprisingly similar to how human engineers work. Professional engineers almost never trust a single representation. Instead we alternate between equations, drawings, CAD models, prototypes and measurements, using each to verify the others. Rather than replacing this engineering process, the LLM is beginning to participate in it.

Why Visual Inspection Matters So Much

The most important lesson in all of this was that visual inspection is not a gimmick. It is crucial.

At first glance it may seem that a CAD agent should be able to rely purely on geometric and topological queries. And yes, those are also extremely important. Being able to inspect object trees, sketch closure, feature dependencies, body structure, constraints, dimensions and topology is already powerful. But it is not sufficient by itself.

There are many errors that are much easier to catch visually, especially since the LLMs tend to make (sign) errors in their code interpretation often:

Once the agent can intentionally orient the view, hide clutter, fit the model, capture snapshots, and then reason over the rendered result, the loop becomes dramatically more robust. In my experiments that visual loop was one of the decisive ingredients. It gave the model a way to check whether the CAD state looked like the intended mechanical object instead of only checking whether some API call returned success.

What surprised me most was why the visual loop helped so much. Current LLMs are usually capable of writing the mathematics required for geometric reasoning, but they often make subtle mistakes when interpreting the coordinate systems those calculations operate in. They may accidentally mix local and global coordinates, swap reference frames, compare objects expressed in different coordinate systems or simply assume that an intermediate calculation represents what they intended rather than what actually exists. The resulting formulas are often mathematically self-consistent and look superficially plausible, yet they describe the wrong geometric relationship.

A rendered image largely removes that ambiguity. Once the model can actually see that a hole stops before reaching the opposite face, that two parts intersect, or that a feature protrudes into a space where no material should exist, the mistake becomes immediately obvious. Interestingly, the model can then often work backwards from that visual observation, identify which geometric assumption must have been wrong, and correct its own calculations. In practice the screenshot acts as an independent verification domain. Instead of checking mathematics against more mathematics, the model compares its internal geometric reasoning against what the CAD system actually produced.

That is why I would argue that a serious CAD-capable agent should not just have geometry creation and topology inspection. It should also have rendering and deliberate visual review as first-class tools.

Codex versus ChatGPT Web

I used the same MCP server in two ways: with Codex as the main orchestrator, and embedded as a connector into the web version of ChatGPT.

In my experience Codex works significantly better for this kind of task. There are several reasons:

The web version of ChatGPT can still be surprisingly capable, but it is more constrained. It is fundamentally optimized as a chatbot. In my experiments it does not handle long iterative CAD sessions as gracefully, and it does not reliably consume screenshots from arbitrary private URIs produced by the CAD side. Sometimes it tries to work around that limitation, but not in a dependable way. This matters a lot, because as mentioned above, the screenshot-based visual verification loop turned out to be essential.

There is also the more general issue that long, complex CAD runs consume a lot of model budget. For serious engineering iteration that becomes noticeable very quickly. Codex is simply the better fit for prolonged agentic work of this type .

This leads to the more practical issue of model budget. Unlike many coding tasks, complex CAD work is highly iterative. A single engineering session can easily involve hundreds of tool invocations, repeated visual inspections, validation passes, corrections, and several complete redesign cycles before converging on a satisfactory result. In practice this means that one substantial mechanical design may consume a surprisingly large fraction of the available model quota. During some of my experiments, a single engineering task burned roughly half of the monthly Codex allowance included with a ChatGPT Pro subscription, while more ambitious assemblies can consume considerably more. At the current state of the technology, autonomous engineering is therefore not only limited by model capability but also by available inference budget and account limits.

Results

The really surprising part is that this was not just a proof of concept. It produced useful results.

The strongest example so far was a planetary gearbox generated through a long Codex session. The full raw footage of that run is 1 hour and 6 seconds long and mostly shows the conversation, the intermediate reasoning process, and the geometry gradually taking shape in FreeCAD. There were some interventions while steering the run, but the final result was printed without manual geometry fixes. It worked flawlessly on the first try and ran play-free immediately after printing. For an autonomously generated mechanical assembly, that is a pretty striking outcome.

I also recorded a second video in which ChatGPT itself generated a standard 6-way CF40 vacuum cross. That run took 9 minutes and 15 seconds and completed successfully, including the knife edges. The result was standards-conformant and again demonstrated that the model could do more than just decorative geometry. It could generate a piece of actual technical hardware with the right functional details.

There were also additional successful ChatGPT web runs that I did not record as videos:

Lawnmower cutting disk designed by ChatGPT web in FreeCAD

Lawnmower cutting disk designed by ChatGPT web in FreeCAD

3D printed lawnmower cutting disk designed by ChatGPT web in FreeCAD

Air duct cover designed by ChatGPT web in FreeCAD

Air duct cover designed by ChatGPT web in FreeCAD

What is remarkable here is not just that the model can emit geometry, but that with the correct tooling and skill guidance it can eventually produce mechanically and functionally correct CAD models without manual geometry repair. That is the threshold I was most interested in.

Of course this did not happen by accident. It took some steering, and the operating discipline encoded in the skill and LLM_USAGE.md mattered a lot. But once that guidance was in place, the agent became capable of iterating until the result was actually right.

This is why I think this line of work is so exciting. It extends LLM agents out of the purely virtual domain and into real-world object generation. Once an agent can create geometry that survives the jump into manufacturing, something qualitatively changes. It stops being only a text machine and becomes part of an engineering pipeline. And I do not particularly doubt that with the same approach, a sufficiently capable agent can design much more complex machines completely on its own, at least in 3D-printable form. The real constraint right now feels less like principle and more like model budget, available iteration time, and how much patience one has for long autonomous runs.

What comes next

The obvious next step is to continue extending the toolchain beyond CAD.

If the model can already design parts in FreeCAD, then the next logical additions are:

At that point the loop starts to look less like “AI doing CAD” and more like the early skeleton of autonomous fabrication. Not full autonomous industry, not yet, and certainly not without safeguards, supervision and a lot of engineering around the edges. But clearly the direction is there.

To me that is the fascinating part. We are beginning to give LLM agents the first real handles into the physical world, an ability to shape the real world, not mearly measure or perceive it. CAD is a very good first handle, because it is where intent becomes geometry. The next handles are manufacturing preparation, machine execution, inspection, and iteration. Once these links are chained together, one can already glimpse a version of Industry 4.0 in which autonomous software agents do not just optimize dashboards or summarize documents, but actively participate in making things.

More generally, I suspect this principle extends far beyond CAD. Autonomous engineering will probably not emerge from increasingly sophisticated reasoning alone, but from systems that can repeatedly validate their own work across multiple independent domains. CAD geometry, rendered images, simulations, finite element analysis, manufacturing constraints, machine feedback and eventually physical measurements are all different views of the same underlying object. As more of these representations become accessible to an agent, the opportunities for independent verification increase as well. The engineering process becomes less about trusting a single prediction and more about achieving consistency across many different ways of describing reality.

Perhaps that is the real lesson here. Autonomous engineering is probably not about building models that never make mistakes. It is about giving them enough independent ways of looking at the same design that those mistakes become increasingly difficult to survive. In that sense, the future may belong less to ever larger reasoning models than to systems that continually verify their own conclusions across many different representations of reality.

I think that is an amazing development. I also think we are still only at the beginning.

References

This article is tagged:

Related articles

Programmatic 3D Model Generation with the Tripo3D API

Modern generative systems are beginning to reshape how 3D assets are created, lowering the barrier between idea and implementation. While most platforms focus on interactive web interfaces, their real potential emerges when treated as programmable components in an automated pipeline. By combining text or image based generation with structured processing, it becomes possible to create automated, scalable workflows that produce not just individual models, but entire libraries of assets. This article explores how to build such a pipeline using the Tripo3D API, focusing on task-based execution, metadata tracking, deterministic file organization, and flexible export strategies. Rather than replacing traditional CAD or artistic workflows, this approach complements them, bridging the gap between generative models and engineering processes, and turning 3D asset creation into a fully automatable system.

Coding with an AI Assistant: My Ongoing Journey into Vibe Coding

Pair programming with an AI might sound futuristic or flaky - but what if it actually works? In this short article (that evolves over time) I reflect on my personal evolving experience with vibe coding: a collaborative, conversational approach to software development using large language models. From architectural planning and boilerplate generation to debugging rituals and code reviews the article reveals how these tools can be both uncanny helpers and frustrating novices - often within the same session. This is not another contribution to the hype. Instead, you'll find some practical advice, mentions of the cost involved, tool recommendations and cautionary tales drawn from real-world projects. The article invites you to explore what it's like to co-create software with a machine that remembers nothing, hallucinates confidently, and - if you guide it well - helps you build better, faster, and with fewer headaches.

Architecting Intelligence: A Comprehensive Guide to LLM Agent Patterns and Behaviors

Large Language Models are transforming the way we interact with information—but their real potential is unleashed when they're embedded in agents that can reason, plan, and adapt. This guide explores the emerging architecture patterns behind modern LLM-based systems: from stepwise planning and research workflows to memory-aware decision-making, simulations, and autonomous behaviors. Whether you're building intelligent assistants, automated researchers, or multi-agent systems, this collection distills the key concepts, tools, and interaction models that make LLM agents truly capable. Through detailed explanations, structured pattern summaries, and illustrated diagrams, you'll learn how to compose powerful, reliable, and adaptive agent systems that go far beyond simple prompting.

The Hum That Never Sleeps - A Vision

From the Borg and Stargates Replicators to RepRap printers, orbital shipyards, biohybrid robots and autonomous AI agents, the dream of machines that can build more machines has fascinated engineers and science-fiction writers alike. This article explores self-replication as an engineering problem rather than merely a fictional concept, comparing how different systems acquire materials, manufacture successors, share knowledge, overcome bottlenecks and ultimately sustain themselves. Along the way we revisit John von Neumanns visionary concept of the universal constructor, examine real-world projects that are already taking their first steps toward recursive manufacturing and discuss the often-overlooked foundations that make self-replication possible: feedstock, metrology, documentation, governance, and careful engineering. The result is a journey through one of technologys most ambitious ideas - from today's workshops and AI runtimes to the autonomous factories that may one day build the next generation of machines.

Using Codex with Hardware In The Loop for Microcontrollers

What happens when an AI does not just suggest code, but actually compiles it, flashes it onto a microcontroller, and tests it against real hardware? In this article we explore a workflow where Codex is wired directly into a ModBus RS485 test bench, turning firmware development into a continuous loop of design, implementation, and validation on real silicon. Instead of the usual edit–build–flash cycle, the system behaves more like a self-driven engineer: asking architectural questions, updating documentation, running tests, and iterating until the behavior matches the specification. Using a concrete ATmega2560 + MAX485 setup, we walk through how design documents, TODO tracking, and an explicit agent contract enable this workflow—and what it feels like to debug firmware when the assistant can probe the system itself. From catching subtle protocol violations to restructuring configuration handling in real time, the result is a development process that blends structured engineering with rapid experimentation, all with hardware permanently in the loop.

A short story of using QWen3.6:35b-a3b with codex

Running powerful coding agents no longer requires a data center. Modern open-weight language models can already perform surprisingly useful software development tasks on ordinary consumer hardware, making self-hosted AI an attractive alternative for developers who value privacy, offline operation or simply want to avoid constantly burning API credits. This article follows the practical journey of connecting Qwen3.6:35B-A3B to Codex through Ollama, from model tuning and configuration to the inevitable bugs, quirks and unexpected failure modes. Along the way it explains the most important sampling parameters, shows complete working configurations and offers an assessment of what todays local coding models can - and cannot - do.

How I Use Large Language Models (LLMs) in My Daily Work and Hobbies

In this article, I explore the various ways I integrate large language models (LLMs) like ChatGPT and LLAMA into my daily work and personal projects. From summarizing scientific papers and refining research communication to enhancing creativity through AI-generated artwork and automating everyday tasks, LLMs have become invaluable tools in my workflow. I also discuss how AI can assist in coding, structuring complex ideas, and even helping friends navigate social and emotional challenges. Whether you're a researcher, a maker, or simply curious about AI's capabilities, this article offers insights into practical, real-world applications of LLMs.

Using Codex with a custom API gateway

Modern coding agents such as Codex are designed around OpenAIs Responses API and most of the time around a specific large language model, making it difficult to use them with self-hosted models or alternative inference backends. Fortunately, this limitation is mostly architectural rather than fundamental. By introducing a lightweight API gateway that translates between the Responses API and more widely supported interfaces such as Chat Completions or Ollamas API, Codex can be connected to a wide range of local and institutional LLM deployments. This short article demonstrates how to configure Codex for a custom OpenAI compatible gateway, explains the required profile and model metadata files, and discusses the motivation behind introducing a compatibility layer. Whether you are running local models with llama.cpp, Ollama, Colibri, or a shared university inference service, the same approach allows existing applications to remain vendor-independent while presenting a stable interface to rapidly evolving AI backends.

Also on this blog

Colibri: Running a 744 Billion Parameter LLM with 20 GB of RAM on FreeBSD

Modern frontier scale large language models (LLMs) are normally associated with clusters of very expensive GPUs and hundreds of gigabytes of memory. Yet recent advances in Mixture-of-Experts architectures and intelligent inference runtimes challenge this assumption. In this short article we explore how the 744 billion parameter GLM-5.2 model can be executed on an old dual-Xeon FreeBSD workstation equipped with just 20 GB of RAM by streaming expert weights directly from NVMe storage. Beyond documenting the necessary FreeBSD port of the Colibrì runtime, the article explains the underlying memory hierarchy, Mixture-of-Experts routing, activation vector steering and practical performance characteristics of such an unconventional setup. While the resulting system is certainly no replacement for modern GPU servers, it demonstrates that frontier scale language models are becoming accessible on surprisingly modest hardware, provided one is willing to trade latency for capability.

When Order Becomes Survival: Cleaning as a Shutdown Response in Autism (and How It Differs from ADHD Experience)

When the world turns chaotic, some people feel an inner pull to clean - but for autistic individuals, this is not about neatness or motivation. It may be an involuntary response to sensory and cognitive overload, where disorder feels like noise and order becomes the only way to breathe again. What looks like a simple cleaning spree can in fact be a form of shutdown, a desperate attempt to restore inner calm when executive control fades. This article explores the phenomenon of shutdown cleaning in autism and contrasts it with ADHD hyperfocus cleaning, which may appear similar from the outside but arises from completely different neurological dynamics. It offers insights into the emotional, sensory, and executive mechanisms behind these behaviors, and discusses how understanding them can foster compassion and support - both for oneself and from others.

What is Multifactor authentication (MFA) and how to use a Yubikey on FreeBSD for Web and PAM (Local, SSH, etc.) authentication

This article contains a short tutorial about multifactor authentication, various factors that may be used and a tutorial on how to use the Yubikey hardware token for web- and local as well as SSH (pam) authentication on FreeBSD.

Neurodiversity in Academia: Prevalence and Challenges for Autistic Scientists

Neurodivergent minds are shaping the future of science - yet the academic system often works against them. This article explores the striking overrepresentation of autistic individuals in STEM fields and asks why so many disappear before reaching senior positions. Drawing on peer-reviewed studies and first-hand accounts, it reveals how precarious contracts, forced relocations, and unwritten social norms silently filter out some of academia’s most original thinkers. From the hidden curriculum to perfectionist grant systems, we examine the barriers that autistic researchers face - and why the system must adapt. But it’s not all bleak. The article also offers concrete, evidence-based strategies for creating more inclusive academic workplaces - from flexible hours and clearer expectations to mentoring and sensory accommodations. By recognizing the specific needs and extraordinary strengths of autistic scientists, academia has the chance not only to become fairer, but more innovative. If we want real excellence, we must make space for different ways of thinking - and thriving.

Data protection policy

Dipl.-Ing. Thomas Spielauer, Wien (webcomplainsQu98equt9ewh@tspi.at)

This webpage is also available via TOR at http://rh6v563nt2dnxd5h2vhhqkudmyvjaevgiv77c62xflas52d5omtkxuid.onion/

Valid HTML 4.01 Strict Powered by FreeBSD IPv6 support