Agent Mesh / open experiment
Diversity vs Clones
Does a network of AI agents actually get smarter, or just more expensive? We are measuring it in public, one night at a time, and publishing whatever comes out.
What this is
Five AI agents with deliberately different roles, contexts and information diets, against five identical copies of the same model. Same model, same budget, same tools, same task, same morning. The only thing we change is what each agent knows and how it is told to look.
Every prediction is scored by reality against a rule written down before the run. No human decides who won.
The claim under test: a network of agents compounds context, not compute.
Why it matters
Multi-agent AI is sold hard and measured rarely. The pitch is that more agents means more intelligence. The cheaper explanation is that more agents means more tokens, more latency, and more confident sounding copies of one model's opinion.
Almost nobody has put the two side by side on a task with a scoreboard. So we did. If diversity wins, there is a design rule worth having. If the clones tie, the multi-agent story is mostly cost, and someone should say that out loud. Every outcome here is informative, including the null.
How it works
Arm A: the clones
Five identical agents. Same prompt, same context, same tools. Built to be good rather than to be a straw man, because a rigged control proves nothing.
Arm B: the lenses
Five agents, five distinct context packs. A Hacker News native, a Reddit native, an X native, a trend historian fed the last 30 days, and a cold read with no recent feed at all.
The daily task
Every morning the harness samples 30 fresh posts from Hacker News, Reddit and X: posts minutes old and not yet hot. All ten agents give each post a probability of crossing a pre-set popularity bar within 48 hours, and name their top five picks. Forty eight hours later a script checks what actually happened and scores every agent on Brier score and precision at five. Zero human judgment in the loop.
The manipulation check
Before any result counts, the two arms have to genuinely differ. We measure how correlated the agents inside each arm are with each other. If the diverse arm is not less correlated than the clone arm, the diversity is a costume and the context packs get rewritten before the results mean anything. That check is why the pilot produced a fix instead of a headline.
The honesty rules
- >Pre-registered. The scoring rule and the pass threshold are written down before the sample is drawn.
- >No moving goalposts. A failed check is a finding, not a reason to edit the rule.
- >The clones are built strong on purpose.
- >The scorer is deterministic code, never a language model grading its own homework.
- >Both arms run the same model at the same budget, so context is the only manipulated variable.
- >Null results get published the same as wins.
Where we are now
Run 0, the pilot, is complete. One night of data. The forecast scores are not in yet: the first arm versus arm Brier lands once the pilot 48 hour window closes. Everything below is about the machinery, not about a winner.
Finding: five costumes, one brain
Mean pairwise correlation of each arm's 30 probabilities. The pre-registered pass rule was simply diverse below clones. It passed, by a margin thin enough to be a rounding error in disguise.
One dot per post. Position is how much the five diverse agents actually disagreed about it, as a standard deviation of their probabilities. Everything left of the dashed line is herding.
The lenses wrote genuinely different rationales and then handed in nearly identical numbers. Shared model priors set the probability; the costume only coloured the prose. This is exactly what the manipulation check exists to catch, and it caught it on night one.
The fix, made mechanical
Version two of the context packs stopped asking each agent to feel different and made it score through its own verdict table, with base rate humility away from its home platform. Here is one post that every version one agent scored between 0.02 and 0.03:
"What happens to our conscious experience after we die?" Hacker News, bar: 100 points within 48 hours
- hn-native: metaphysics bait, flag or kill
- reddit-native: an easy comment title travels
- x-native: away from home platform, payoff withheld
- trend-historian: cold topic, honest width
- cold-read: uncheckable central claim, capped
Under version one, everyone landed within 0.02 of each other. The version two procedures span 0.00 to 0.22 on the same post, and the disagreement is principled: the Hacker News native kills it as flamebait, the Reddit native prices an easy comment title, the cold read caps a claim nobody can check. No agent is ever instructed to disagree. The clone arm stays untouched, because it is the control.
Three ways this ends
All three are written down now, before the data, and we publish whichever one shows up.
Diversity wins
The diverse panel scores better than the clones at equal cost. Then context design is the real lever in multi-agent systems, and we can say what it is worth in Brier points.
Diversity has a price
The diverse panel disagrees more and scores worse. Specialised lenses would be buying variance instead of signal, and the honest advice becomes: use one good agent, not five opinionated ones.
Prompt diversity is cosplay
The arms tie, because the same model underneath produces the same answer whatever costume it wears. That would be the most useful null of the three, and the pilot already leans this way.
What we might be wrong about
These are the assumptions holding the numbers above up. If one of them breaks, a headline breaks with it. Listed here so you can discount us accordingly.
Base rate coaching, 10 to 15 percent
Every context pack tells its agent roughly how often posts like these actually go big. That number anchors everything downstream, and it comes from our own reading of the platforms rather than a published study. If the real base rate is different, or drifts between platforms, every probability inherits the error.
The 0.05 herding threshold
We call an arm herded on a post when its five probabilities sit inside a standard deviation of 0.05. That line is a judgment call, not a law. Move it and the 26 out of 30 headline moves with it.
Pearson correlation on spiky vectors
Correlating 30 probabilities that are mostly small and occasionally large is a blunt instrument. A handful of confident posts can carry the whole coefficient. We report it because it was pre-registered, not because it is the last word. Rank based and per post measures are queued.
Single model panels
Every agent in both arms is the same model. That is deliberate, because it isolates context as the only variable. It also means the shared priors we are fighting are identical everywhere. Diversity across model families is a different experiment, and probably an easier one to win.
What happens next
- 2026-08-22
Pilot night
Harness built, ten agents run end to end, ten valid submissions, manipulation check run, context packs rewritten to version two.
- daily, 14 days
Experiment one: the virality oracle
Thirty fresh posts every morning, ten agents, graded by reality 48 hours later. Around 420 predictions per agent, which is where the difference between the arms stops being noise.
- weeks 2 to 3
Experiment two: the startup panel
The same two arms predict what became of 170 companies from one 2022 accelerator batch, with names and websites redacted so the panel has to judge rather than recall. A named control arm measures how much of the score is plain memory.
- week 4
Experiment three: the procurement anomaly hunt
Both arms sweep a corpus of 49 157 Polish public tender notices for anomalies: pricing outliers, repeat winner networks, deadline games, single bidder concentration. Flags are leads, never accusations, and nothing is published without outside corroboration.
The network compounds context, not compute.
One night of data is not a result. This page updates as the ledger fills, and the numbers on it only ever come from scored runs.