AI NeuroSignal: A 20-Agent LLM Ensemble Scored Against Market Outcomes

The short version: AI NeuroSignal ran a pool of 20 LLM agents, with three to eight voting on each trading call. Between 1 May and 13 August 2026 it produced 6,187 signals and scored 38,319 individual agent predictions against what the market did next, across five instruments. Its own publishing gate concluded it had not shown skill beyond market drift, and said so publicly. This page claims no returns. What it shows is the architecture: how to orchestrate probabilistic agents and measure them honestly.

I was the architect and principal engineer (494 of 513 commits). Every figure below can be checked against the project’s public edge endpoint, calibration endpoint and methodology page.

The Problem: Two Hard Parts, Not One

Ask an LLM for a trading call and you get a fluent, confident answer with no way to tell whether it is any good. The work had two parts.

1. Orchestration. Run several LLM analysts with different methods, combine their structured calls (direction, confidence, a stop and three targets) into one signal, and stop any single analyst’s bad output reaching users.
2. Evaluation. Each call is only right or wrong hours later, when price touches a target or the stop, or the call expires. The system has to score every call against the market, attribute the result to the exact agent that made it, and never claim more skill than the data supports.

The second part took most of the architecture work, and it is the part that transfers to any system where several LLM components make calls that are checked later.

Architecture: Confine the Probabilistic Part

The LLM calls sit in one box. Everything before them (market data, prompt assembly) and after them (validation, aggregation, resolution, scoring, publishing) is deterministic and mostly pure functions, which is what made it testable: 235 unit tests cover the scoring, statistics, resolution and calibration code.

• Parallel fan-out with a straggler cap. Agents run in parallel; a 45-second cap drops slow ones. One agent failing never fails the signal; only all of them failing does.
• Structured output or nothing. Every response is validated against a Zod schema with one retry, and falls back to NEUTRAL when it still fails. Confidence is clamped and impossible prices are nulled.
• Specialisation by method, not by information. Nine strategy templates, each with its own prompt and its own indicator set, plus market specialists for gold and GBP/JPY. All agents see the same candles, which limits how independent their errors can be.
• Models as a controlled variable. 79% of predictions came from DeepSeek-V3.1 and 21% from GPT-4o-mini. The house portfolio was deliberately pinned to one model per cohort, so the record measures strategies rather than models. There is no runtime model routing.

Deterministic Controls Around Every Model Call

Each of these rules was added after a failure seen in production.

• Abstention instead of coin flips. If the weighted margin between long and short is under 20 points the system publishes NEUTRAL, and long calls need a 40-point margin.
• Price levels are sanity-checked. Stops must sit within 6 ATR and targets within 10 ATR, on the correct side of price. This was added after the models proposed gold targets at prices from their training era: 6 of the first 602 signals, one of them 122R away.
• Levels are blended, then re-ordered. At least two valid proposals are blended 60/40 with an ATR baseline, and the three targets are re-sorted by distance, because averaging had inverted them in 5 of 602 signals.
• Simulated data is refused in production, loudly.

Scoring Agents Against Reality

• Agents are scored against the market, not the ensemble. A contrarian agent gets credit only when the market actually moved its way.
• An agent’s identity is a hash of its prompt and model. Editing either starts a new version with a fresh record, like hiring a new analyst. Old predictions keep their analytics but stop moving the current rating.
• Records cannot be inflated by cloning. The public pool counts each persona once per signal, so copying an agent across many accounts adds nothing to its record.

The Gate That Refused to Claim Skill

One statistics module serves both the internal and the public numbers, so they cannot drift apart. Before anything is published it must clear three tests: at least 30 decided calls, a two-sided z-test against 50%, and a margin of at least 5 points over the best naive baseline (always long or always short over the same window).

Over the live window the system was significant against a coin flip: 53.0% on 5,125 decided calls, z = 4.26. But the market fell on 51.8% of those calls, so “always short” would have scored 51.8%, and the system beat it by only 1.2 points. The public endpoint therefore reports skillBacked: false. The system declined to claim skill it could not separate from market drift.

Adaptive Weighting: Built, Measured, Found Inert

Each agent carries an Elo-style rating per market, and its vote is weighted by it. Because every prediction stores its decision, confidence and weight, all 5,125 decided signals could be re-voted offline:

Weighting only broke ties: a weight range of about 0.5 to 1.6 across three to eight correlated agents cannot overturn a majority. The same analysis found that stated confidence carries no usable information. Over 37,731 scored predictions, calls stated at 70-80% confidence were right 30.6% of the time, and the Brier score was 0.315 against a 0.240 base-rate baseline (calibration endpoint). The interface now says “agent agreement” instead of “confidence”.

Postmortem: One Write That Erased a Month of Scoring

A best-effort write to the public persona pool had been bundled into the core scoring transaction. One failing write silently rolled back all per-agent scoring for over a month, because the resolver’s per-signal error handler swallowed it. The fix: the rating, its history and the outcome commit in one transaction, and the pool write runs separately, idempotently, and logs loudly when it fails. Every control in the system now has a deliberately chosen failure direction: market data fails closed, the structure filter fails open, and both are written down.

Cost Architecture

Five-minute polling crons kept the serverless Postgres database awake around the clock. A key-value wake-gate lets each cron ask “is anything due?” without touching the database, and new signals invalidate the gate so real work is never delayed. Leases stop concurrent resolver runs racing on the same ratings.

Evidence

Limitations

• No demonstrated predictive skill beyond market drift. That is the honest result.
• Nearly single-vendor. Model diversity was controlled out, not exploited.
• Little visibility into the calls themselves. Latency, token and cost data are not persisted per call.
• The resolver last ran on 13 August 2026, so the figures cover May to August only.
• No external proof the history was not edited. Signals are not hashed before their outcome is known; a commit-before-outcome ledger is specified but not built.
• The code is private.

What Transfers Beyond Trading

1. Contain the probabilistic part. Everything around the model call should be deterministic and testable.
2. Score agents against reality, not against each other or the ensemble.
3. An agent’s identity is a hash of its prompt and model, and its record belongs to that hash.
4. Before tuning an orchestration mechanism, check that it changes decisions. This one changed one in 5,125.
5. A model’s stated confidence is not a probability. Calibrate it against outcomes or don’t use it.
6. Gate claims against the best naive baseline, not a coin flip.
7. Never put a best-effort write in the same transaction as the source of truth.

The same design fits support triage checked against resolution codes, fraud review checked against chargebacks, and code-review bots checked against reverted commits. How I build multi-agent systems more generally is in the production guide to multi-agent AI systems, and the measurement layer is covered in building an LLM evaluation set.

This is an architecture case study, not investment advice, and it claims no trading returns. The live record ran across seven accounts, mostly an operator-run house portfolio, and its MetaTrader bridge traded a demo account.

Related reading: for a different application of multi-agent coordination, see the SculptAI case study. I provide AI architecture, consulting and custom AI development to businesses in the UK, the US and Europe. Building a multi-agent system? See AI agent development, or the AI security and architecture review for one you already run. View all services.