Are Vibe-Coded Apps Secure? What I Found in 53 AI-Built Repositories

The short answer: not by default. Of 53 public repositories built with Lovable, Bolt, v0, Cursor, Replit or Claude Code, at least 72% had no automated security checks at all (95% interval 58-82%): nothing like Dependabot, CodeQL or Snyk to flag a vulnerable package or a dangerous pattern. At least 45% carried a finding rated critical (33-59%). Yet up to 91% showed some sign of testing. The tools are not careless about everything. They reliably skip the safety net: security checks, input validation and secrets hygiene.

Scanned 1 October 2026. The checklist, the method, the anonymised per-repository dataset and the scanner’s accuracy are all public in the study’s repository, archived on Zenodo as doi:10.5281/zenodo.23094508. No repository is named.

How to cite: Chin, N. (2026). Production readiness checklist for AI-built apps, with measured research on 53 public repositories. Zenodo. https://doi.org/10.5281/zenodo.23094508

Correction, 2 October 2026. The first version of this article said 73% of repositories with server code had “no security middleware” such as rate limiting or CORS. The scanner column behind that figure detects security tooling(dependency and code scanning), not request middleware, so it is now described that way. “91% had a test file” was also too strong: the rule counts any sign of testing. No count changed. Details are in the study’s correction note.

What Was Measured

A repository counted only if it declares the tool itself, in a GitHub topic or its README, so the sample can be rebuilt by anyone. Forks, templates, tutorials, deliberately vulnerable projects and anything under 1,000 lines were excluded. The sample is stratified by tool: nine each from Lovable, Bolt, v0, Cursor and Claude Code, and eight from Replit. The median repository is 27,573 lines.

Each was read by the deterministic half of the SystemAudit scanner, which I built. No language model produced any number here. The scanner reads a capped subset of files (a median of 16% of them), so anything it found is a floor, not an estimate: finding nothing in the files it didn’t read proves nothing. Things it reports as missing are read from the full file tree, so those are true rates.

What It Found

Found in the files that were read. These are floors.

Missing from the repository. These are true rates, except the two marked “at least”, whose rules accept loose matches that can only flatter a repository.

A “critical” finding is the scanner’s own severity, not a CVSS score: a committed credential or env file, an injection sink reachable from request input, or an authorisation check satisfied by data the caller controls. The intervals are wide because 53 is a small sample, and they matter more than the single number: “72%” means somewhere between 58% and 82%.

The Safety Net Is What’s Missing

Nearly three in four repositories had no automated security checks configured: no Dependabot or Renovate to flag a vulnerable package, no CodeQL, Semgrep or Snyk to catch a dangerous code pattern, no secret scanner. These tools cost nothing to switch on for a GitHub repository, and without them a known vulnerability can sit in an app until someone exploits it.

The rate barely moves with the kind of app: 29 of the 40 repositories that serve requests (73%) and 9 of the 13 frontend-only ones (69%). Whether these apps configure rate limiting or CORS was not measured, and nothing here supports a claim either way.

“AI-Built Apps Have No Tests” Looks Like the Wrong Story

Only 5 of 53 showed no sign of testing at all: no test file, no test configuration and no test runner. That makes 91% an upper bound on repositories with tests, and it says nothing about whether the tests are meaningful or run; 28% have no CI to run them. Even so, the common complaint that these tools produce untested code does not match what is in the repositories. The gap is narrower and more dangerous: the code is checked for whether it works, not for whether it can be abused.

The Worst Failures Don’t Live in the Repository

The most serious problems in an AI-built app are row-level security policies that let every signed-in user read every row, admin checks that only hide a button, and tenant isolation nobody tested. In a Supabase app those live in the database project, not the code. This study cannot see them, and nothing in it should be read as evidence they are fine.

An independent study that audited deployed apps rather than repositories points the same way. Deng, Fan and Meng collected 9,041 open-source apps built with Claude Code and Lovable and audited 200 publicly deployed ones: 91.0% had at least one vulnerability, 65.77% of the vulnerabilities were rated critical or high, and the main categories were broken access control, injection and authentication failures. That is precisely the class a repository scan cannot see, which is why it needs a person. The mechanism is in why AI-built apps leak data.

At the code level, Carnegie Mellon’s SusVibes benchmark gave coding agents 186 real tasks that had previously led human programmers to write vulnerable code. In the paper’s September 2026 revision, 57% of SWE-Agent with Claude 4 Sonnet’s solutions worked, and 11.8% were secure. Working and secure are different tests, and the tools are mostly being judged on the first.

How Accurate Is the Scanner?

A rate is only as good as the instrument that measured it, so this is published too, including where it went wrong.

The first run reported that at least 15% of repositories had committed a credential. Before anything was published, all eight repositories behind that number were opened by hand. Six were false: test fixtures inside a secret-redaction module, a PII sanitiser’s own sample data, a UUID used as a share link, a hyphenated constant, shell variables in a deploy script, and a CI database on localhost. Precision was 25%.

The scanner was fixed. The corrected figure is at least 6% (three repositories), and the automated result now agrees with the hand check. The scanner is now scored on every change against a labelled set of 42 cases, 16 that must be reported and 26 that must not, and currently scores 100% precision and 100% recall. That means no known regression, not that it is correct: the cases are our own, and a larger adversarial set is the next step.

Other errors are recorded in the repository for anyone repeating this. A scanner that runs out of API budget skips files silently and reports a clean repository, so the run now stops instead. And a column name is not a definition: two rows were first described from their names rather than the rules behind them, which is the correction noted at the top.

What This Does Not Say

• Public repositories are not deployed products. A team that publishes its code may be earlier in its life or less careful than one that doesn’t, and the direction of that bias is unknown.
• A declared tool did not necessarily write everything. Many of these repositories may have been edited by hand afterwards.
• Per-tool rates are not published. At eight or nine repositories per tool the intervals are too wide to rank the tools against each other.
• Rates are not severity. A missing linter and a committed database password each count once.

What to Do With This

If you built an app with one of these tools, start with what can be automated. Then spend your attention on what can’t. The pre-launch checklist that comes with this study marks each of its 36 items as blocking launch or able to wait, and as checkable by a scan or needing a person. 16 of the 20 launch blockers need a person: whether each table’s policy names a user, whether admin routes check a role on the server, whether a webhook verifies its signature.

Frequently Asked Questions

The Data Argues for a Person

Every number here comes from a scanner, and the most important conclusion is about what the scanner can’t see. If your app is about to take payments, hold personal data or serve more than one customer from one database, a clean scan is not the all-clear it looks like. A review of an AI-built app before launch checks the access rules, roles and webhooks that live outside the code, and for a multi-tenant platform the Supabase architecture review goes through the data-access model in full.