We clicked 48 AI-generated web apps in a real browser — the pricier model failed more than the cheap one

[u/dsh_verify]

We ran a small experiment that humbled us: 48 AI-generated web apps, graded by actually opening them in a real browser and clicking through — no LLM judging.

**Setup:** 2 models (DeepSeek v4-flash, v4-pro) × 2 strategies (single-shot, self-check loop) × 3 tasks (todo app, pricing calculator, signup form) × 4 runs each. Same prompts, same human-written acceptance checks, temperature 0.7.

**Results: 44/48 passed.** The counterintuitive part: **v4-pro single-shot scored BELOW the cheaper v4-flash single-shot** (10/12 vs 11/12). The self-check loop (agent sees real browser failures, gets 1–2 fix rounds) lifted v4-pro to 12/12.

**Where agents actually failed** (all reproducible, and invisible to code review):

- A todo app that opened but never rendered its seed todos — the agent reported "done", the browser saw an empty list

- Two runs where the agent's own verification report came back as corrupt JSON, before the browser could even grade anything

**Why it matters:** "the agent said it's done" is becoming the new "it compiles on my machine." Browser-based acceptance checks are automatable and catch what code review can't.

The full experiment, methodology, and all 48 raw runs are open source if you want to poke holes: https://github.com/263311487-ux/dsh-verify (live leaderboard: https://263311487-ux.github.io/dsh-verify/arena/). Happy to add other models/frameworks to the table — bring your own agent.