Method

How I ask

The short version

Each round, I put the same health questions to ChatGPT, Claude, Gemini and Perplexity, three attempts each, with web search on. I record the first product each answer recommends, every other product it names, and the sources it cites. Then I publish all of it — twelve answers per question, unedited.

The assistants

Four, because together they cover most of what people use when they ask an AI for advice:

• ChatGPT — gpt-6-luna
• Claude — claude-sonnet-5-5
• Gemini — gemini-3.8-flash
• Perplexity — sonar-pro

Model versions are recorded per answer in the data. When an assistant ships a new version, the record will show the change, because each answer carries the model that produced it.

Why the questions sound like a patient

Most "what do AIs recommend" projects ask in the register of a survey: What is the best calorie tracking app? I ask the way people actually type it: Which app should I use to count calories? — or I'm starting a GLP-1 medication. Which app should I use to track my food? First person, with context. The wording is fixed from round to round so the record shows how answers move, not how my questions did.

Why three attempts each

A single answer can be a fluke. An assistant that names the same product first in all three answers is telling you something different from one that changes its mind every time, and the pages show both. Twelve answers per question: four assistants, three attempts, captured 2026-11-02 between 14:06–15:47 UTC.

What I extract

• firstChoice — the first product the answer recommends.
• alsoNamed — every other product it mentions.
• citedSources — the links the answer followed or cited.
• declinedToPick — whether it refused to name one. (It hasn't happened yet.)

The tiers

• UNANIMOUS — all four assistants' first choice is the same product.
• CONSENSUS — three of four.
• SPLIT — no product is the first choice of three or more assistants.

What this is not

• It is not a review. I don't test products and I don't rate them.
• It is not medical advice. It is a record of what assistants said, and assistants are wrong regularly enough to be worth tracking.
• It is not edited. If an answer says something wrong, that stays in the record.

How to read a question page

Every question page in this round is built the same way, so once you can read one you can read all of them:

• The verdict box. The tier (UNANIMOUS / CONSENSUS / SPLIT), the pick if there was one, and what the tier means. UNANIMOUS requires all four assistants to name the same product first; CONSENSUS is three of four; SPLIT means nothing reached three.
• The answer matrix. Four rows — one per assistant, with its model version — and three columns, one per attempt. Each cell shows that attempt's first choice, how many other products it named (+N), and how many sources it cited. Assistant colors match across the whole site.
• What each assistant said. A per-assistant card: whether its first choice held across all three attempts (SAME PICK ×3) or changed, the union of everything it named, and the hosts it read. Written from the record.
• The analysis. What the record shows, the alternatives and the roles they played, why the answers converged or split, and what that means for someone asking. My prose, clearly separated from the record.
• What the assistants read. Every source cited across the twelve answers, as hosts. The full URLs are in the JSON.
• The data link. Every page links to its own JSON file — the twelve answers in full, exactly as extracted.

Using the data

Every question page links to a JSON file with the full record for that question (12 files, 144 answers in Round 1). It is published under a Creative Commons Attribution 4.0 licence; please link back to the question page you used.

Changes and fixes are logged on the changelog.

Frequently asked