A new benchmark covered by a single recent preprint, MobileForge (arXiv 2607.28645), asks the question screenshot review cannot: not whether one generated screen looks right, but whether a whole generated app works. Its headline result, demonstrated only in the paper’s tested environments, is that six frontier multimodal LLMs produce projects that compile and reach the correct pages while interactive navigation stays unreliable and visual fidelity and maintainability lag. If you ship AI-generated frontends, your acceptance tests are probably checking the wrong layer.
What does MobileForge measure that screenshot review misses?
MobileForge evaluates app generation at the project level, meaning the buildable, multi-screen artifact, rather than at the single-screen level where existing design-to-code benchmarks live. The abstract states their three limits plainly: they “focus on single-page generation rather than complete codebases, cannot evaluate cross-page navigation, and do not measure project-wide maintainability.” The authors describe it as the first benchmark for project-level multi-screen mobile app generation, per the arXiv abstract page, which now carries an “Accepted by EMNLP 2026 Main” notice.
That distinction sounds academic until you map it onto how teams actually review generated UIs today. The standard workflow: prompt a model, get a screen, look at the screen, iterate on the prompt until the screen looks right, ship. Vendor demos reinforce this loop because they showcase polished individual screens, not navigable projects. The implicit acceptance criterion is visual: does this pixel output resemble the thing I asked for.
A mobile app fails that criterion’s blind spot almost by definition. An app is a state machine with screens as its visible surface. The thing that makes it an app rather than a pile of mockups is the edges: what happens when a user taps a button, whether the cart on screen three reflects the item added on screen one, whether the back stack unwinds the way the platform says it should. None of that is visible in any screenshot, no matter how many you take.
The benchmark’s five evaluation axes make the gap concrete: build, navigation, visual fidelity, code maintainability, and efficiency, all per the abstract. Screenshot review covers exactly one of the five, and it is arguably the least load-bearing one. A screen can be pixel-perfect against a reference design and belong to an app that dead-ends the user on the second tap.
This is why the paper matters to practitioners even before you read a single score. It formalizes a mismatch that teams shipping generated frontends have been discovering anecdotally: the artifact the model is good at producing (a convincing screen) is not the artifact acceptance testing needs to verify (a working project).
Do AI-generated apps actually work?
According to the abstract, current frontier models can already build mobile-app projects that compile and reach the correct pages; interactive navigation remains unreliable, and visual fidelity and maintainability still lag. That is a more specific and more useful result than either “AI can build apps now” or “AI apps are broken.”
Read it carefully, because both hype readings are wrong. The optimistic reading says the hard part is solved. It is not: the part that is solved is the part that was already cheap to verify. Compilation is a binary signal your toolchain gives you for free. Reaching the correct page on an initial render is one step past that. Neither requires the model to have modeled user behavior, state transitions, or the app’s own information architecture across time.
The pessimistic reading says generated apps do not work. Also wrong, and the paper’s own result cuts against it: the projects build and the right pages resolve. The failures sit above that layer: in interactive navigation (the taps and flows that connect screens into an application) and, per the abstract, in visual fidelity and maintainability, which still lag.
That localization is the actionable part. A failure set this contained is a failure set you can target with testing. If models failed everywhere, the only honest advice would be “do not generate apps.” When the failures are navigation and the shared state it exercises, plus visual fidelity and code maintainability, the advice becomes: keep the generation speedup, move the review budget.
One caveat belongs here rather than in the footnotes. The abstract names no models and reports no scores. “Six frontier multimodal LLMs” ran end-to-end app generation in the study, but which six, and how large each gap is, lives in the paper’s tables. Treat any per-model ranking as unknown until you have read those tables yourself. The direction of the result (build and page resolution pass; navigation, fidelity, and maintainability lag) comes from the abstract; the magnitude does not.
Which five axes should gate an AI-generated app ship?
The benchmark’s evaluation framework doubles, almost verbatim, as an acceptance checklist for generated frontends: build, navigation, visual fidelity, code maintainability, and efficiency. The contribution is less the list itself than the evidence about which axes current models already pass and which they do not.
| Axis | Acceptance question | Where frontier models stand (per the abstract) | Review artifact |
|---|---|---|---|
| Build | Does the generated project compile and run? | Within reach | CI build on the generated repo |
| Navigation | Do interactions route across screens correctly? | Unreliable | Scripted multi-screen flow tests |
| Visual fidelity | Do screens match reference designs? | Lags | Anchor-referenced comparison, not vibes |
| Code maintainability | Can a human own this codebase next quarter? | Lags | Lint, structure review, diff-size audit |
| Efficiency | Does the app waste runtime resources? | Evaluated; the abstract reports no scores | Profiler pass on key flows |
Two things stand out in that table. First, the axes are ordered by how badly screenshot review serves them. Build and navigation are invisible to screenshots entirely. Visual fidelity is the only axis screenshots address, and it is one of the three the abstract reports as lagging; the next section explains why even it needs a better judge than an eyeball. Second, the two axes where the abstract reports success (build, page resolution) are precisely the ones cheapest to verify mechanically, while the lagging axes (navigation, fidelity, maintainability) are the ones that need actual test infrastructure, anchored comparison, or code review. The review-cost reallocation is not optional; it is the shape of the problem.
Maintainability and efficiency deserve a sentence each because they are the axes teams most often skip when adopting generated code. A generated project that passes navigation tests can still be a liability if the code is structured in a way no engineer wants to modify, and mobile apps have hard platform budgets (startup time, memory, battery) that generated code has no particular reason to respect. The paper evaluates both. The abstract reports a clear direction for maintainability, which lags alongside navigation and visual fidelity, and no direction for efficiency, which remains an open question your own review must answer.
Why did the authors have to fix their own judges?
The authors propose two evaluation techniques, state-isolated navigation testing and an anchor-referenced list-wise visual evaluation protocol, because the default ways of judging generated apps produce wrong answers. The existence of these protocols is itself a finding: naive evaluation of generated apps is broken in two specific, instructive ways.
Start with navigation. The obvious way to test navigation is end-to-end: launch the app, execute a flow, check where you land. The problem is cascading failure. If the first tap in a flow misroutes, every subsequent step is evaluated in the wrong screen context, and the measurement collapses into noise. One early failure corrupts every downstream observation, so the score tells you about the first bug, not about the app’s navigation quality. State-isolated navigation testing addresses this by letting each navigation step be evaluated without inheriting the wreckage of previous steps, per the abstract.
Now visual evaluation. The default approach to scoring generated screens is an LLM judge: show a model the generated screen and a reference, ask how close they are. The authors’ need to build an anchor-referenced list-wise protocol implies that this default is unreliable for generated screens. Their protocol has the judge compare candidates against anchors rather than assign absolute scores in a vacuum, and the abstract states its purpose plainly: “to improve visual-judge reliability.” If your team’s current quality bar is “a person or a model looked at the screenshot and it seemed fine,” the paper is telling you that bar mis-ranks outputs often enough that researchers had to engineer around it.
The deeper point connects the two repairs. Both failure modes are instances of evaluation that assumes independence where none exists. End-to-end navigation assumes each step is measurable in isolation; it is not, because state carries. Absolute visual scoring assumes each screen can be judged without reference points; it cannot, because judges drift. Generated-app evaluation is hard for the same reason generated apps are hard: state and context are the whole problem, and evaluation methods that ignore them inherit the same bugs as the models being evaluated.
How should review change: from per-page eyeballing to cross-screen state auditing
The practical consequence of the headline result is a reallocation of review time, away from per-screen visual polish and toward project-level integration tests of navigation flows and shared state. The vibe-coding speedup only survives if the second screen works, and the second screen is exactly what current review habits do not check.
Concretely, a review checklist for AI-generated UIs, derived from the benchmark’s axes and its reported failure mode:
1. Gate the build mechanically. Compilation and initial page resolution are within reach of frontier models, the abstract reports, so treat them as a floor, not a goal. A generated project that does not build in CI on the first try is a signal about your prompt and scaffolding, not about model capability. Spend zero human minutes here; the toolchain answers it.
2. Write navigation flow tests before you polish pixels. The abstract reports interactive navigation as unreliable, so this is where human attention goes. Script the critical flows: the sign-up path, the add-to-cart path, the settings round trip. Each flow should assert landing screens per step, not just final state, because the benchmark’s state-isolation insight applies to your test suite too. A test that only checks the final screen will report a cascade of failures as one failure and send you debugging in the wrong place.
3. Audit shared state at screen boundaries. Navigation unreliability and shared-state inconsistency are the same bug viewed from two sides: a tap misroutes, or it routes correctly and the destination screen renders stale or missing state. For every flow in step 2, assert that state mutated on screen N is visible and correct on screen N+1. This is the check screenshot review structurally cannot perform.
4. Compare screens against anchors, not against memory. Borrow the paper’s visual protocol at whatever fidelity you can afford. Pin reference designs as anchors and judge generated screens relative to them in a list-wise pass (rank this batch against the anchors) rather than asking “does this look right” in isolation. The latter is the judging mode the authors found unreliable enough to replace.
5. Read the code once, quickly, for ownership. Maintainability is one of the five axes, and one of the three the abstract reports as lagging. The question is narrow: if this project needs a change in three months, does the generated structure permit it, or is the diff easier to regenerate from scratch? Both are legitimate answers, but they lead to different maintenance policies, and you should know which one you have before the code is in production.
6. Profile the flows you scripted. Efficiency is the fifth axis and the easiest to defer indefinitely. Mobile platforms will eventually defer it for you, via app-store review, ANR rates, or user deletion. A single profiler pass over the critical flows costs little and closes the last axis.
The economics here are worth stating plainly. Per-screen eyeballing scales with the number of screens and produces a review artifact (a looked-at screenshot) that expires the moment the next regeneration happens. Cross-screen state auditing scales with the number of flows, which is smaller and more stable than the number of screens, and produces test files that survive regeneration. The teams that get durable value from app generation will be the ones whose review output is a test suite rather than a Slack thread of screenshots.
What can this benchmark not tell you?
Almost everything about magnitudes, and less than it appears about generality. The honest evidentiary position is narrow: a single preprint, moderated by arXiv but not peer-reviewed by it, with an “Accepted by EMNLP 2026 Main” notice on its abstract page. arXiv’s own documentation is explicit that moderation checks topicality and scholarly value, not correctness, and the repository hosts more than three million articles under that policy. Venue acceptance raises confidence that the work is serious; it does not substitute for reading the tables.
The missing specifics matter. The abstract names no models and reports no scores, so nothing here tells you whether the navigation gap is a two-point nuisance or a fifty-point cliff, or whether one model in the cohort has already closed it. The cohort itself, six frontier multimodal LLMs as of the study, will churn within months. The five-axis framework and the judging protocols are the durable part of the paper; the leaderboard is not.
There is also a methodological caveat from adjacent work. A separate 2026 preprint, arXiv 2608.20622, attributes to arXiv 2605.23950 the finding that harness choice accounts for most of the variance in agent benchmark results, more than model choice does. That claim is one level removed from primary measurement and should be weighted accordingly, but the direction is plausible and the implication is uncomfortable: some portion of MobileForge’s model-level gaps may be a property of its evaluation harness rather than of the models. The authors’ custom judging protocols reduce one class of measurement error while necessarily introducing their own design choices.
Finally, scope. The results are demonstrated only in the paper’s tested environments: its app cohort, its platforms, its prompts, its harness. “Interactive navigation remains unreliable” is a statement about those conditions. Your generated internal tool with four screens and a tab bar is a different distribution from whatever the benchmark generates, in both directions: simpler apps may route fine, and design-heavy apps may fail earlier and worse. The benchmark tells you where to look. It does not tell you what you will find.
What should you gate AI-generated app ships on?
Gate on project-level integration tests: multi-screen navigation flows and shared-state consistency, not screenshot review. That is the practical verdict, and it follows directly from the abstract’s reported result direction: compiling and reaching the right page is already within reach of frontier models, so verifying it buys you nothing, while interactive navigation is unreliable and visual fidelity and maintainability lag, so verifying those is the entire game.
Budget your review time accordingly. Cross-screen state auditing is where the screenshot-invisible failures live and where the review artifacts (flow tests, state assertions, anchor-referenced visual comparisons) survive the next regeneration. Per-page polish evaporates on every re-prompt, and the one lagging axis it touches, visual fidelity, is measurable only through the anchored comparison described above. The five axes from the benchmark give you the checklist; the state-isolation and anchor-referenced protocols tell you how to run the checks without your evaluation inheriting the apps’ own bugs.
Hold the counterweight in view as you adopt this. The evidence base is one preprint whose abstract reports no scores, its per-model rankings are unknown until you read the tables, and related 2026 work on evaluation harnesses warns that benchmark rankings can reflect the harness more than the model. What survives those caveats is not any leaderboard position. It is the structural finding that screen-level evaluation passes screens while apps fail at the seams, and the matching design insight that testing generated software requires isolating state and anchoring judgment. If your acceptance process for generated frontends is still “someone looked at the screens,” the benchmark has told you, with venue-accepted evidence and in enough detail to act on, which seam to test first.
Frequently Asked Questions
Does MobileForge’s navigation failure apply to web apps or only mobile?
The benchmark is explicitly scoped to mobile app generation, evaluating platform-specific constraints like back-stack unwinding and mobile runtime budgets. While the state-isolation principle applies to any stateful UI, the specific failure modes cited (e.g., platform navigation stacks) are mobile-specific, so web app teams cannot directly transfer the benchmark’s mobile-specific navigation scores to their own acceptance criteria.
How does state-isolated testing differ from standard end-to-end integration tests?
Standard end-to-end tests execute a linear flow where an early failure corrupts all subsequent assertions, masking the true location of the bug. State-isolated testing resets the application state before each navigation step, allowing each transition to be evaluated independently. This prevents a single misroute from generating a cascade of false negatives that would otherwise obscure the root cause in the test report.
What is the operational cost of implementing anchor-referenced visual scoring?
Teams must curate and maintain a fixed set of reference designs (anchors) for every screen type in the application. Unlike absolute scoring, which requires no baseline, this method adds a maintenance burden: whenever the reference design changes, the anchor set must be updated, and the list-wise comparison logic must be re-validated to ensure the judge is ranking against the current standard rather than stale visuals.
Why might the reported navigation gaps be artifacts of the evaluation harness?
A separate 2026 preprint (arXiv 2608.20622) indicates that harness choice often accounts for more variance in agent benchmark results than model choice does. This suggests that MobileForge’s specific implementation of state-isolation and visual judging may introduce its own biases, meaning the observed navigation failures could partly reflect the rig’s design choices rather than a universal limitation of the underlying multimodal LLMs.
