Building Verifiable AI Systems: A case study on Engineering Design using AI Agents

2. The Problem and the Numbers

The site is one irregular ten-vertex polygon - a 6,420 sq ft lot with a corner notch and a south-frontage tab carrying a protected tree that cannot be built over. The goal: generate valid layouts that are checked against hard rules, so hallucinated geometry never makes it through.

Figure 1 — The plot specification.

Reported footprints ranged from under 1,000 to over 4,600 sq ft - and that spread is not a geometry dispute, it's a consequence of which zoning rule each team applied. Teams that sourced their coverage limit from web lookups inherited a 35% cap that compressed footprints to 989–2,170 sq ft. Teams that read the actual building constraints directly, or applied Multifamily Lowrise (LR) offsets (smaller than the 20–25 ft single-family defaults, because the LR zone permits denser construction), legally unlocked far larger envelopes - up to 4,606 sq ft. The real signal is convergence: once a rule set is fixed, independent models land on the same verified footprint.

Figure 2 — The parcel with setback boundaries and the maximum buildable footprint (shaded), clear of the protected tree.

Figure 3 — A final verified layout: a fully dimensioned site plan holding the footprint at the 35% cap, with the driveway exempt as access.

3. Two Ways to Check the Work

Every pipeline is a closed loop: design, check, feed the failures back, repeat until it passes. The decisive question is how the checker reasons.

Figure 4 — The generator–checker loop.

One option is to have another language model judge compliance in plain language. It's fast and needs no extra tooling - but it's sensitive to phrasing, inconsistent from run to run, and prone to waving through violations as the constraints pile up. The other option is a deterministic check that runs outside the model: outcomes become stable, auditable, and repeatable, and confirmation bias disappears. The trade-off is more moving parts in the pipeline. Where correctness actually has to be guaranteed, the independent check isn't an add-on to the agent's self-checking - it's a replacement for it.

4. How the Models Behaved

Across the runs, a few consistent personalities emerged.

As designers (A1):

• 🔹

  Gemini was fast and clean but lazy - it liked to treat any numbers in the prompt as anchors to copy rather than constraints to reason about, and stayed conservative on stepped, asymmetric setbacks. Turning off its default "thinking" mode cut latency from minutes to seconds, and elevating "maximize the legal footprint" to the primary goal added real coverage with no change to the model itself.

• 🔹

  Claude produced well-documented, high-fidelity geometry but tended to over-engineer - adding unrequested doors, partitions, and annotations out of professional habit, which ate into buildable areas and occasionally introduced its own violations that had to be stripped back out.

• 🔹

  DeepSeek generated quickly but tended to ignore constraints, and read specs hyper-literally - robust on tight, fully specified prompts, brittle on loose ones.

As checkers (A2):

• 🔸

  ChatGPT was a strong reviewer under rigid pass/fail checks, but under broad, open-ended review it missed global flaws - and if its first calculation was wrong, it would follow that error, even bending its judgement to fit faulty feedback, until a deterministic check broke the loop.

• 🔸

  Mistral gave accurate, reliable corrections when it wasn't asked to do the heavy arithmetic itself - a dependable checker within a clearly defined scope.

• 🔸

  Kimi depended heavily on structure; it could audit individual points but had blind spots in between, and needed the checking frame carefully to catch violations that fell between the obvious spots.

Prompt precision is usually the real bottleneck. Across versions, the difference between a wrong answer and a correct one was rarely the model's raw capability - it was how precisely the task was specified. Given only a rough image, models misread the lot and collapsed it to a rectangle; given explicit vertices, named reference lines, and a strict output format, the same models produced an accurate, non-rectangular footprint immediately, and token bloat vanished. The bottleneck is rarely capability; it's prompt precision.

A cost comparison made the point concrete. On a well-specified task, all the successful models reached the same verified footprint - they differed only in runtime and cost.

Same answer, very different cost. Fastest: Llama 3.1 8B. Deepest reasoning: DeepSeek-R1. Accuracy: a tie.

The starkest contrast: one large reasoning model burned more than 6,000 reasoning tokens to reach the answer a small 8B model reached in under 1,000. For a well-specified task with an independent check in place, the capability gap between small and large models was effectively zero. The cost gap was not.

5. The Takeaway

One challenge ran beneath everything else: geometric complexity itself. As the building shape moved from a simple rectangle to an L-shape to a multi-notch polygon, the agents kept falling back on rectangular thinking - reasoning about area as width × height, which only holds for rectangles. A room placed neatly against a bounding box could still spill outside the true lot shape. The checker handled these cases correctly, but the designing agent had no equivalent built-in tool - a sign that agents need genuine geometric computation aids, not mental approximations, for non-trivial shapes.

The project standardized on a single rulebook across four zones (Redmond, Bellevue, Bothell, Seattle Downtown) so that limits came from the code, not a model's memory. One elegant result fell out of this: under one zone's strict 35% coverage cap, the area-maximizing footprint turns out to be an L-shape, not a rectangle - capturing the most legal square footage without breaching coverage or the tree zone. And the project's core critique was finally closed by a pipeline where both agents read the raw code, worked out the buildable area, and designed the full interior themselves - converging in two iterations to a verified 2,016 sq ft layout, with no answer baked into the prompt.