Microsoft turned one AI into a panel of five doctors, and it beat real doctors on the hardest medical cases while spending less.
This week: exactly how they did it, and how you can use the same structure to get better answers from your own AI.
Microsoft built a tool called the Diagnostic Orchestrator (MAI-DxO). An orchestrator coordinates other pieces, which is exactly what this does. Then they tested it on 304 of the hardest real cases published in the New England Journal of Medicine. Doctors study these because they stumped other doctors.
The start: a 29-year-old woman with a sore throat that has lasted weeks, and swelling near her tonsil.
What it sounds like: a common infection.
What it really was: a rare cancer (embryonal rhabdomyosarcoma). You only get there by asking the right questions and ordering the right tests.
Doctors and the AI played the exact same game.
Not real money. A pretend bill to keep the test fair.
| Move | What it added to the bill |
|---|---|
| A round of questions | $300, like paying for a doctor visit |
| A test | Its real price from a 2023 U.S. hospital price list. A blood test is cheap. An MRI is not. |
Same game, different strategy. Before every move, the AI stops and thinks as five different doctors. It's not five different AIs. It's one AI with five jobs. The real doctors played as themselves.
Keeps a ranked list of the 3 most likely diagnoses and updates it after every new clue.
Chooses up to 3 tests per round, the ones that best tell the top suspects apart.
Plays devil's advocate: "Are we stuck on our first guess? What doesn't fit?"
Pushes for cheaper tests that answer the same question. Can veto expensive, low-value ones.
Makes sure test names are real and the panel's reasoning holds together.
Microsoft calls the debate a "Chain of Debate." In the budget version, the panel also saw each test's price before ordering it and could cancel tests that wouldn't change the answer.
| Who played | Right, out of 100 cases | Average pretend bill |
|---|---|---|
| Doctors, no references allowed | 22 | $3,049 |
| Doctors, with references like UpToDate | 36 | $4,725 |
| GPT-5 on its own, no roles | 79 | $8,409 |
| GPT-5 with the five roles | 83 | $5,430 |
| Five roles plus a budget | 81 | $2,241 |
Doctors struggled. The AI alone was smart but wasteful. The AI with roles was smart and careful. Look at GPT-5 on its own. It was accurate too, but it ordered everything and ran up about $8,400 per case. The panel structure is what brought the cost down.
Microsoft's tool isn't an AI. It's a set of instructions, the five roles, placed on top of an AI. The main test used GPT-5. Then they swapped the AI underneath: Claude played all five roles by itself, then Gemini, then others. Each AI was tested twice, once on its own and once with the five roles.
Like a student going from 50 to 58 out of 100, just by using the five roles.
Google's budget model, Gemini 2.5 Flash, went up about 24. Like a student going from 40 to 64.
Why did the cheaper AIs improve the most? They made more mistakes, so the roles had more to fix. Give a study method to a straight-A student and they improve a little. Give it to a struggling student and they improve a lot.
Sources: Microsoft AI · Science paper · Methods manuscript · SDBench
Andrej Karpathy is one of the best-known people in AI. He was a founding member of OpenAI, later ran AI at Tesla, and joined Anthropic in May 2026.
He put a free project on GitHub called LLM Council. He called it a fun Saturday hack and said he won't support or update it. So the idea is what matters, not his code.
| Karpathy's LLM Council | Microsoft's Diagnostic Orchestrator | |
|---|---|---|
| Who's in the room | Several different AIs | One AI playing five roles |
| How they work together | Answer, then rank each other | Debate in rounds, each role has a job |
| How many tries | One round | Many rounds, gathering more information each time |
| Cost control | None | Built in |
These are two separate projects, built on the same idea.
Source: Karpathy's LLM Council on GitHub
| Situation | What the roles do |
|---|---|
| Fixing a broken web app | "My login works on my laptop but not live." It lists the top 3 suspects and checks the cheapest first: did you push your code? Are your Supabase keys in Vercel? |
| Before building a feature | "Should I add AI chat?" The Challenger asks if users want it. The Budget Keeper estimates the monthly cost. |
| Pricing | $29 a month or $97 one time? It asks about your audience in rounds before answering. |
| Catching made-up answers | The Checker asks: is this setting real? Is this law real? |
| Outside of tech | Choosing a vendor, planning a course, writing a sales email. Any decision where a wrong first guess costs time or money. |
Works in Claude, ChatGPT, or Gemini. Replace the last line with your problem.
Act as a panel of five experts solving my problem in rounds:
1. The Hypothesis: keeps a ranked list of the 3 most likely answers and updates it each round.
2. The Next Step: picks the one question or check that best tells those answers apart.
3. The Challenger: argues against the top answer. What doesn't fit?
4. The Budget Keeper: pushes for the cheapest, fastest step first and vetoes anything wasteful.
5. The Checker: makes sure everything mentioned is real and the reasoning holds up.
Each round, show a short debate, then pick ONE move: ask me a question, ask me to run a check, or give your final answer. Only give the final answer when you're confident, and tell me how confident you are.
My problem: [describe it]
Open three fresh chats in the same AI and give each one a different role, like "You're a skeptic" or "You're focused on cost." Fresh chats can't see each other, so their opinions are more independent. Then paste their answers into one final chat to judge, like Way 2.
Below are three answers to the same question from different AIs, labeled A, B, and C.
Rank them for accuracy and insight, and explain why. Show me where they disagree. Then act as the chairman and write one best combined answer. Flag anything that only one answer claimed, so I know to double-check it.
My question was: [your question]
Answer A: [paste]
Answer B: [paste]
Answer C: [paste]
Different companies trained these AIs differently, so they disagree in useful ways.
One question box. Your question goes to Claude, GPT, and Gemini automatically, they review each other with names hidden, and a chairman gives you one final answer.
| Tool | Why |
|---|---|
| OpenRouter | One key that reaches Claude, GPT, Gemini, and more. Pay per use, usually pennies per question. |
| Claude Code | Builds the app from one prompt. |
| GitHub + Vercel | Stores your code and puts it online. |
| Supabase (later) | Not needed to start. Add it for logins, saved history, or to sell it as a subscription. |
Build me a simple "AI Council" web app with Next.js, deployed on Vercel.
1. One page with a question box.
2. Always send my question to the same 3 models through OpenRouter: one from Anthropic (Claude), one from OpenAI (GPT), and one from Google (Gemini). Use the latest version of each. Set this in one place in the code so I can easily swap a model later.
3. Round 1: show each model's answer side by side.
4. Round 2: send all the answers back to each model, labeled Answer A, B, C with no model names, and ask each one to rank them for accuracy and insight and explain why.
5. Round 3: Claude acts as the chairman. It reads everything, writes one final combined answer, and flags anything only one model claimed.
6. Keep my OpenRouter API key on the server only. Never expose it in the browser. Read it from an environment variable.
7. Add rate limiting so no one can run up my bill.
8. Show roughly what each question cost.
Explain each step in plain language before you build it, and walk me through testing it locally, pushing to GitHub, and adding my key to Vercel.
Claude Code can create subagents: separate helpers that each start fresh, like separate chats. You don't build anything. Just ask for them in your prompt, inside the project you're already working on.
Solve this problem using 5 subagents, one for each role. Run them separately so they don't influence each other:
1. The Hypothesis: list the 3 most likely causes or answers, ranked.
2. The Next Step: find the one check that best tells those apart.
3. The Challenger: argue against the top answer. What doesn't fit?
4. The Budget Keeper: find the cheapest, fastest fix. Don't rewrite anything big.
5. The Checker: make sure everything mentioned is real and actually exists in my project.
Then bring their findings together, show me where they disagreed, and recommend ONE next step. Don't change any code until I say yes.
My problem: [describe it]
Missed a week? See all past live sessions →