SystemAudit: Shipping a Codebase Intelligence SaaS That Doesn't Hallucinate

Most AI code review tools have the same fundamental flaw: they ask a language model to read your code and tell you what's wrong. That produces confident-sounding prose with invented line numbers, fabricated CVEs, and architectural claims the model pattern-matched from training data rather than observed in your repository. It looks impressive until a reviewer actually opens the file and the "finding" isn't there. For a product that investors, engineering directors, and founding teams rely on for real diligence decisions, that failure mode is unacceptable.

SystemAudit is my answer: systemaudit.dev, a production SaaS that turns any GitHub repository into a full system health report in under three minutes, starting at $49 rather than weeks of a consultant's time. The core innovation is not the AI summary on top - it's the two-layer architecture underneath that makes the AI layer structurally incapable of hallucinating. This case study explains how that works and why it matters for production AI products in general.

The Problem: AI Code Review is a Trust Crisis

The category SystemAudit competes in has two broken halves. Traditional technical due diligence is expensive, slow, and unavailable to anyone without a budget and a network. A proper engineering audit takes one to three weeks of senior consultant time - fine for Series A diligence, useless for a founder on Thursday afternoon trying to understand whether they've accumulated enough technical debt to rewrite. The other half - automated AI-based code reviewers - solves the speed and price problem but introduces a worse one: outputs you can't actually trust.

When an AI code reviewer tells me "there's a SQL injection on line 42 of auth.ts," I need that to be true. If I open auth.ts and there's no such code, the whole tool is dead to me - not just for that report, but forever. This is why technical buyers bounce off most AI code review products: one hallucination and the trust is gone.

The design problem is therefore not "build a better prompt". It's structural. You have to make hallucination architecturally impossible for the claims that matter.

The Core Innovation: Two-Layer Architecture

SystemAudit splits analysis into two layers with a hard separation of responsibilities.

Layer 1 - Deterministic Analysis (No AI)

Before any language model sees the code, a comprehensive static analysis pass runs over the repository. This layer produces hard evidence, not opinions:

• Exact vulnerability patterns with file and line references (hardcoded API keys, SQL injection vectors, innerHTML sinks, sensitive data exposure).
• Full import dependency graph with proven circular dependencies, single points of failure, and dead code.
• Dependency health flags (outdated, unmaintained, known-CVE packages).
• Structural quality metrics (modularity, nesting depth, file-size distribution, type coverage).
• Verified configuration state - what's actually in package.json, CI files, Docker, lint configs, test configs.

Layer 1 runs at zero AI cost. These are facts extracted by deterministic code, and they're what downstream claims must anchor to.

Layer 2 - AI Interpretation, Enforced Against Layer 1

The AI layer receives the source code alongside every Layer 1 finding. Its job is not to discover vulnerabilities - Layer 1 has already done that deterministically. Its job is to synthesize patterns, identify architectural concerns, and translate technical findings into business language for non-engineer readers (investors, boards, CEOs).

But constraining a prompt is not enough - models drift, and "grounded" is not the same as "governed." So the model is not trusted; it is checked on every claim. After it produces its interpretation, a separate deterministic enforcement pass re-validates the output against the measured facts and overrides the model wherever the two disagree:

• Scores are capped or floored to reality. If the model rates type safety highly while the compiler config proves strict mode is off, the score is capped and the claim rewritten to what's actually true; if it reports "no tests found" while test files demonstrably exist, the coverage score is floored and the sentence corrected to the real count.
• Unprovable findings are downgraded. When a finding's cited code can't be located in the actual source, its confidence drops from confirmed to likely, and known false-positive shapes have their severity cut automatically.
• Disproven findings are deleted outright. A "hardcoded credentials" flag whose evidence actually references environment variables (process.env, os.environ) is removed entirely - the system would rather drop a finding than ship one it can't stand behind.

This is the difference between grounding a model and governing one. It is why SystemAudit reports cite specific files and line numbers for every security finding, why cost estimates calibrate to actual complexity metrics rather than guesses, and why a hallucinated claim cannot survive to the PDF: a separate, deterministic layer has the final say over the language model on every number and every finding.

What the Report Delivers

Every SystemAudit report includes:

• System Architecture Map - a visual diagram of how every component connects, generated from the actual import graph.
• Security & Vulnerability Scan - hardcoded secrets, SQL injection, XSS vectors, sensitive data exposure, with exact file and line evidence.
• Dependency Graph Analysis - circular dependencies, single points of failure, and dead code identified deterministically rather than estimated by an LLM.
• Risk Assessment - each risk ranked by severity with "cost to fix" and "cost if ignored" framed in business language a non-engineer can act on.
• Feature Verification - cross-references feature claims against test files, so "we have auth" only passes if there's actually auth code with tests.
• AI Readiness Score - a 5-dimension assessment (Code Clarity, Test Coverage, Modularity, Documentation, Type Safety) with an A-F grade per dimension.
• Health Score - a single number across 20+ checks (tests, security, CI/CD, documentation, dependency health, structural metrics).
• Prioritized Fix Plan - a week-by-week remediation roadmap with investment tiers, ROI projection, and hiring guidance.
• Exportable PDF - the whole report as a single document suitable for investors, boards, or dev teams.

Language and Ecosystem Support

SystemAudit supports 50+ languages across 9 major ecosystems: JavaScript/TypeScript (Next.js, React, Express), Python (Django, FastAPI, Flask), Java/Kotlin (Spring Boot), Go (Gin, Echo, Chi), Rust (Actix, Axum, Rocket), C#/.NET (ASP.NET Core), PHP (Laravel, Symfony), Ruby (Rails), plus Docker, CI/CD, serverless, and monorepo configurations. The deterministic analyzers are the per-language work; the AI layer and report format are the same across all of them.

Quality Assurance

A tool that claims "near-zero hallucination" has to prove it, continuously. SystemAudit is tested against a standing corpus of 10 real public codebases ranging from 800 lines to 600K+ lines. Each run is graded across four dimensions: factual accuracy, free-tier reliability, full-analysis quality, and business-translation clarity. The current benchmark is 10/10 across all four - though that rubric is one I designed and that corpus is one I chose, so it is a regression signal rather than an independent grade. Every AI finding is cross-checked against Layer 1 evidence; any finding the AI layer produces that can't be anchored to deterministic evidence is dropped rather than shown to the user.

What Shipped in September 2026, and What It Measured

Four changes went out in September. Each one exists because something was measured and found wanting, so they are worth stating with the numbers attached rather than as a feature list.

False positives, cut by three guards. The pattern rules were matching things that were not findings. Three guards went in: a declaration-line guard, so a rule table naming its own patterns no longer matches itself; a context window, so a JSON-LD block spanning several lines is no longer read as an injection sink; and a validator check, so request input is only flagged where nothing validates it. Measured against a real 41,000-line TypeScript codebase, raw pattern output went from 13 matches to 2 - and both survivors were genuine. The denominator matters more than the headline: the interesting claim is not "fewer false positives" but that eleven of thirteen were noise before anyone looked.

Coverage is now stated, not implied. The scanner reads a prioritised subset rather than cloning the whole repository, which means an empty finding list has always meant nothing found in what we read - never nothing wrong. Reports now say so explicitly: "Analysed 80 of 214 files." A security tool that lets the reader infer full coverage from a clean result is lying by omission, and the fix costs one sentence per report.

Evidence reaches the report. The deterministic layer always produced file-and-line evidence; it was stopping at the server. Each finding now carries the file, the line number and a redacted snippet, so the reader can open the code and judge it rather than taking a verdict on trust. The architecture had been right and the delivery had not - the two are separate problems, and only the second was visible to users.

Urgent findings are never paywalled. If a scan finds a live credential or a critical flaw, the exact location is shown - file, line, masked snippet - with no signup and no payment. Withholding the location of an exposed secret until someone pays is a conversion mechanism that works, which is precisely why it needed a standing rule rather than a case-by-case judgement. Lower-severity locations and the full analysis remain paid.

The Economic Argument

Traditional technical due diligence: a consultant engagement of one to three weeks. SystemAudit: under three minutes, starting at $49. That's not a marginal improvement - it's a different category of product. It changes who can afford technical diligence. A founder pitching investors on Friday can run their own repo through SystemAudit on Thursday night and walk into the meeting already knowing what the tech-literate board member will find. A VC screening 40 startups for a thesis round can afford to audit all of them instead of picking three.

Part of the deterministic layer is open source under MIT, and it is worth being exact about which part. Public at github.com/nicuk/systemaudit are the repository scanner and its supporting code: language and framework detection, size and structure metrics, dependency extraction, and the health indicators that feed the score. Not public are the security pattern rules and the Layer 2 enforcement pass described above. So what a sceptical reader can verify for themselves is how the evidence layer measures a codebase - not the rules it is measured against. For a product whose value proposition is trust, stating that boundary is worth more than claiming a transparency the repository does not deliver.

Why This Pattern Matters Beyond SystemAudit

The two-layer architecture is the generalizable lesson. Whenever you're shipping an AI product whose output needs to be trustworthy - not just plausible-sounding - the pattern is the same: do the deterministic work first, hand the LLM both the raw input and the deterministic findings, and constrain it to synthesizing and explaining rather than discovering. This is how I build RAG systems (see SureCiteAI - retrieval is deterministic, the LLM only generates on top), how I build agentic systems, and how I evaluate vendors for clients who are wondering whether to build or buy an AI layer on top of their own product.

The specific shape changes per domain. The underlying principle doesn't: if you let the LLM do the work of both finding and interpreting, you're going to ship something that looks impressive in demos and falls apart with real users. Move the finding into deterministic code; let the AI do the interpretation. That bright line is what turns a cool demo into a product people trust.

Related reading: For how the same "deterministic first, AI second" discipline applies to retrieval, see SureCiteAI. For the strategic version of this conversation - "should we build an AI layer in our product at all?" - see Build vs Buy AI Systems and Why AI Projects Fail: Architecture. I work with a small number of founders and operators each quarter as a fractional AI CTO, including clients in Malaysia and Singapore.