← Previous ☰ All Live Sessions Oct 23 · Friday Live
Dr. Erin Jacques · Friday Live
Friday Live · October 23

One AI. Five doctors.

Microsoft turned one AI into a panel of five doctors, and it beat real doctors on the hardest medical cases while spending less.

This week: exactly how they did it, and how you can use the same structure to get better answers from your own AI.

Part 1
The study

A medical mystery game.

Microsoft built a tool called the Diagnostic Orchestrator (MAI-DxO). An orchestrator coordinates other pieces, which is exactly what this does. Then they tested it on 304 of the hardest real cases published in the New England Journal of Medicine. Doctors study these because they stumped other doctors.

Example of a hard case

The start: a 29-year-old woman with a sore throat that has lasted weeks, and swelling near her tonsil.

What it sounds like: a common infection.

What it really was: a rare cancer (embryonal rhabdomyosarcoma). You only get there by asking the right questions and ordering the right tests.

The game: the rules everyone played by

Doctors and the AI played the exact same game.

  1. Start with a short description of the patient. The diagnosis is hidden.
  2. Each turn, pick one move: ask a question ("Any recent travel?"), order a test (blood work, a scan), or commit to a final diagnosis.
  3. A second AI plays the hospital. It holds the full case file and answers only what's asked. It never gives away the diagnosis.
  4. Diagnosing ends the case. One shot, no take-backs. So the AI has to decide when it's confident enough to stop.
  5. A third AI grades the answer using a scoring guide written by doctors. It counts as correct if it would lead to the right treatment.

Every move had a price

Not real money. A pretend bill to keep the test fair.

MoveWhat it added to the bill
A round of questions$300, like paying for a doctor visit
A testIts real price from a 2023 U.S. hospital price list. A blood test is cheap. An MRI is not.
Without a bill, the AI could order every test in the hospital. The bill shows whether it thinks like a good doctor.
The strategy that won

The AI's secret: one AI, five roles.

Same game, different strategy. Before every move, the AI stops and thinks as five different doctors. It's not five different AIs. It's one AI with five jobs. The real doctors played as themselves.

Dr. Hypothesis

Makes the guess

Keeps a ranked list of the 3 most likely diagnoses and updates it after every new clue.

Dr. Test-Chooser

Picks the next step

Chooses up to 3 tests per round, the ones that best tell the top suspects apart.

Dr. Challenger

Argues against it

Plays devil's advocate: "Are we stuck on our first guess? What doesn't fit?"

Dr. Stewardship

Watches the bill

Pushes for cheaper tests that answer the same question. Can veto expensive, low-value ones.

Dr. Checklist

Checks the work

Makes sure test names are real and the panel's reasoning holds together.

How they work together

Step 1All five debate the case
Step 2They agree on one move
Step 3The "hospital" answers
Step 4Repeat until confident, then diagnose

Microsoft calls the debate a "Chain of Debate." In the budget version, the panel also saw each test's price before ordering it and could cancel tests that wouldn't change the answer.

The results

Right more often. Spent less.

36%
Doctors got about 1 in 3 right, spending about $4,725 per case
81%
The AI panel on a budget got about 4 in 5 right, spending about $2,241 per case
Who playedRight, out of 100 casesAverage pretend bill
Doctors, no references allowed22$3,049
Doctors, with references like UpToDate36$4,725
GPT-5 on its own, no roles79$8,409
GPT-5 with the five roles83$5,430
Five roles plus a budget81$2,241

Doctors struggled. The AI alone was smart but wasteful. The AI with roles was smart and careful. Look at GPT-5 on its own. It was accurate too, but it ordered everything and ran up about $8,400 per case. The panel structure is what brought the cost down.

The structure didn't just make it smarter. It made it less wasteful.

It worked on every AI they tried

Microsoft's tool isn't an AI. It's a set of instructions, the five roles, placed on top of an AI. The main test used GPT-5. Then they swapped the AI underneath: Claude played all five roles by itself, then Gemini, then others. Each AI was tested twice, once on its own and once with the five roles.

Every AI

About 8 points higher

Like a student going from 50 to 58 out of 100, just by using the five roles.

Cheaper AIs

Up to 24 points higher

Google's budget model, Gemini 2.5 Flash, went up about 24. Like a student going from 40 to 64.

Why did the cheaper AIs improve the most? They made more mistakes, so the roles had more to fix. Give a study method to a straight-A student and they improve a little. Give it to a struggling student and they improve a lot.

The trick is the structure, not the specific AI. That means you can use it with whatever AI you already have.

What to keep in mind

Sources: Microsoft AI · Science paper · Methods manuscript · SDBench

Part 2
Karpathy's LLM Council

A different way to get the same result.

Andrej Karpathy is one of the best-known people in AI. He was a founding member of OpenAI, later ran AI at Tesla, and joined Anthropic in May 2026.

He put a free project on GitHub called LLM Council. He called it a fun Saturday hack and said he won't support or update it. So the idea is what matters, not his code.

How it works

  1. First opinions: your question goes to several different AIs. His example used GPT-5.1, Gemini 3 Pro, Claude Sonnet 4.5, and Grok 4.
  2. Review: each AI reads the others' answers with the names hidden, so it can't play favorites, and ranks them for accuracy and insight.
  3. Final answer: one AI acts as the "Chairman" and combines everything into one answer.

Karpathy vs. Microsoft

Karpathy's LLM CouncilMicrosoft's Diagnostic Orchestrator
Who's in the roomSeveral different AIsOne AI playing five roles
How they work togetherAnswer, then rank each otherDebate in rounds, each role has a job
How many triesOne roundMany rounds, gathering more information each time
Cost controlNoneBuilt in

These are two separate projects, built on the same idea.

Don't trust one answer from one AI.

Source: Karpathy's LLM Council on GitHub

Part 3
Why this matters to you

Your AI's first answer is a guess.

Where you can use it

SituationWhat the roles do
Fixing a broken web app"My login works on my laptop but not live." It lists the top 3 suspects and checks the cheapest first: did you push your code? Are your Supabase keys in Vercel?
Before building a feature"Should I add AI chat?" The Challenger asks if users want it. The Budget Keeper estimates the monthly cost.
Pricing$29 a month or $97 one time? It asks about your audience in rounds before answering.
Catching made-up answersThe Checker asks: is this setting real? Is this law real?
Outside of techChoosing a vendor, planning a course, writing a sales email. Any decision where a wrong first guess costs time or money.
This is a way to think, not a way to diagnose yourself. The study had no real patients.
Four ways to do it

Pick the one that fits you.

Way 1 · One AI, five roles

Microsoft's structure, in one prompt.

Works in Claude, ChatGPT, or Gemini. Replace the last line with your problem.

Paste this into any AI
Act as a panel of five experts solving my problem in rounds: 1. The Hypothesis: keeps a ranked list of the 3 most likely answers and updates it each round. 2. The Next Step: picks the one question or check that best tells those answers apart. 3. The Challenger: argues against the top answer. What doesn't fit? 4. The Budget Keeper: pushes for the cheapest, fastest step first and vetoes anything wasteful. 5. The Checker: makes sure everything mentioned is real and the reasoning holds up. Each round, show a short debate, then pick ONE move: ask me a question, ask me to run a check, or give your final answer. Only give the final answer when you're confident, and tell me how confident you are. My problem: [describe it]

Test it yourself, free

  1. Chat 1: ask your question plainly.
  2. Chat 2: ask the same question with the five-role prompt.
  3. Compare. Which answer was more accurate? Which one caught something the other missed? Try it with a fast model and a top model to see who gains more.

Even more independent: separate chats

Open three fresh chats in the same AI and give each one a different role, like "You're a skeptic" or "You're focused on cost." Fresh chats can't see each other, so their opinions are more independent. Then paste their answers into one final chat to judge, like Way 2.

Way 2 · Different AIs, no code

Karpathy's council, by hand.

  1. Ask Claude, ChatGPT, and Gemini the exact same question.
  2. Paste all three answers into one of them, labeled Answer A, B, and C. Leave the names off so it can't favor its own answer.
  3. Use this prompt to rank and combine them.
Paste this with the three answers
Below are three answers to the same question from different AIs, labeled A, B, and C. Rank them for accuracy and insight, and explain why. Show me where they disagree. Then act as the chairman and write one best combined answer. Flag anything that only one answer claimed, so I know to double-check it. My question was: [your question] Answer A: [paste] Answer B: [paste] Answer C: [paste]

Different companies trained these AIs differently, so they disagree in useful ways.

Way 3 · Build your own council

Don't download his. Build your own.

One question box. Your question goes to Claude, GPT, and Gemini automatically, they review each other with names hidden, and a chairman gives you one final answer.

What you need

ToolWhy
OpenRouterOne key that reaches Claude, GPT, Gemini, and more. Pay per use, usually pennies per question.
Claude CodeBuilds the app from one prompt.
GitHub + VercelStores your code and puts it online.
Supabase (later)Not needed to start. Add it for logins, saved history, or to sell it as a subscription.

Steps

  1. Get an OpenRouter key: create an account at openrouter.ai, add about $5 in credit, and create an API key.
  2. Open Claude Code in a new project folder and paste the prompt below.
  3. Put your key in .env.local and test it on your computer.
  4. Push to GitHub. Vercel deploys it.
  5. Add your key to Vercel's environment variables. Your .env.local never leaves your computer, so the live site can't see it until you do this.
  6. Later: add Supabase for logins and history, and Stripe if you want to charge for it.
Paste this into Claude Code
Build me a simple "AI Council" web app with Next.js, deployed on Vercel. 1. One page with a question box. 2. Always send my question to the same 3 models through OpenRouter: one from Anthropic (Claude), one from OpenAI (GPT), and one from Google (Gemini). Use the latest version of each. Set this in one place in the code so I can easily swap a model later. 3. Round 1: show each model's answer side by side. 4. Round 2: send all the answers back to each model, labeled Answer A, B, C with no model names, and ask each one to rank them for accuracy and insight and explain why. 5. Round 3: Claude acts as the chairman. It reads everything, writes one final combined answer, and flags anything only one model claimed. 6. Keep my OpenRouter API key on the server only. Never expose it in the browser. Read it from an environment variable. 7. Add rate limiting so no one can run up my bill. 8. Show roughly what each question cost. Explain each step in plain language before you build it, and walk me through testing it locally, pushing to GitHub, and adding my key to Vercel.
A change on your computer is not a change on your site.
Way 4 · Five roles in Claude Code

Let Claude Code split into five helpers.

Claude Code can create subagents: separate helpers that each start fresh, like separate chats. You don't build anything. Just ask for them in your prompt, inside the project you're already working on.

Paste this into Claude Code
Solve this problem using 5 subagents, one for each role. Run them separately so they don't influence each other: 1. The Hypothesis: list the 3 most likely causes or answers, ranked. 2. The Next Step: find the one check that best tells those apart. 3. The Challenger: argue against the top answer. What doesn't fit? 4. The Budget Keeper: find the cheapest, fastest fix. Don't rewrite anything big. 5. The Checker: make sure everything mentioned is real and actually exists in my project. Then bring their findings together, show me where they disagreed, and recommend ONE next step. Don't change any code until I say yes. My problem: [describe it]

Your homework this week.

  1. Use the five-role prompt on one real problem you have right now.
  2. Run the free test: same question, with and without the five roles. Compare.
  3. Try Way 2: ask three AIs the same question and let one judge.
  4. Next time something breaks in your app, use the Way 4 prompt in Claude Code.
  5. Bonus: build your own council with the Way 3 prompt.
Join the community →

Missed a week? See all past live sessions →