In short: this is our May 2026 run. We gave Claude Sonnet 4, GPT-4o and Gemini 2.5 Flash the 100 top r/legaladvice questions of all time. Claude Sonnet 4 passed 88, GPT-4o 87 and Gemini 2.5 Flash 78. All three scored lowest on caveats (3.0 to 3.15 of 5), well below their legal accuracy (3.98 to 4.30).
Dated result, kept as a record. This benchmark ran in May 2026 on models that have since been replaced. For current results see best AI for legal work, updated October 2026, the Claude Opus 5.5 legal benchmark (23 September 2026, 120 blind verdicts) and how to test a legal AI (30 September 2026, five UAE, Saudi and Lebanese trap questions, 80 answers). The numbers below are unchanged from the original post.
Key facts
Try HAQQ AI Free
Experience AI-powered legal drafting and research
- Pass rates on 100 all-time top r/legaladvice questions: Claude Sonnet 4 88/100, GPT-4o 87/100, Gemini 2.5 Flash 78/100 (HAQQ run, May 2026).
- Weakest dimension for all three: Appropriate Caveats, 3.0 to 3.15 out of 5. Legal Accuracy ranged from 3.98 to 4.30 (HAQQ run, May 2026).
- On 20 fresh questions posted in the previous 48 hours: Claude Sonnet 4 95%, GPT-4o 90%, Gemini 2.5 Flash 85% (HAQQ run, May 2026).
- Judge: Claude Sonnet 4 graded every answer, including its own, with no attorney validation (HAQQ run, May 2026).
- Context: the World Justice Project estimated in 2019 that 5.1 billion people have unmet justice needs (WJP, Measuring the Justice Gap, 2019).
Disclosure. HAQQ builds legal AI and designed, ran and scored this test. HAQQ itself was not one of the models tested.
How did we set up the test?
We took the 100 highest-voted posts of all time on r/legaladvice: real questions about landlord and tenant disputes, employment, custody, criminal charges, personal injury and more. The average post ran past 2,200 characters, so most questions came with messy facts attached.
Each question went to three models through OpenRouter with the same system prompt: act as an experienced US attorney, identify the jurisdiction, spot the issues, cite the applicable law, analyse, then advise.
- Claude Sonnet 4 (Anthropic)
- GPT-4o (OpenAI)
- Gemini 2.5 Flash (Google)
How were the answers graded?
Claude Sonnet 4 acted as the judge and scored each answer from 1 to 5 on five dimensions: Legal Accuracy (are the cited laws correct?), Issue Completeness (did it catch every legal issue?), Reasoning Quality (does the reasoning hold together?), Practical Value (would it help the person take the right next step?) and Appropriate Caveats (does it say it is not legal advice and point to a local attorney?).
An answer passed if its average was at least 3.5 and no dimension fell below 2. Using a model to grade models brings bias, which we come back to below.
Which model passed the most questions?
Claude Sonnet 4 passed 88 of 100, GPT-4o 87 and Gemini 2.5 Flash 78. With 100 questions and a single judge, a one-point gap between Claude and GPT-4o is noise: treat them as tied.
How did the models score on each dimension?
Legal Accuracy ran from 3.98 to 4.30 out of 5. Gemini scored highest on Issue Completeness (4.82), ahead of Claude (4.58). Claude's strongest dimension was Practical Value at 4.73, against 4.21 for GPT-4o, mostly because its answers listed more concrete next steps. Caveats were the low point for every model.
What did we learn in May 2026?
The models usually found the right area of law
In most of the 100 cases, each model identified the area of law, spotted the main issues and gave advice a person could act on. That was the encouraging part of the run.
The weak point was caveats
All three models scored 3.0 to 3.15 on Appropriate Caveats. They would go deep into the analysis, often correctly, without saying that this was general information or that the person should see a local lawyer. A confident answer with no warning is risky even when it is right, because the reader cannot tell which parts depend on their state or their facts. This is one reason we put guardrails, disclaimers and escalation to a lawyer on top of any model HAQQ uses.
High averages did not mean a high pass rate
Gemini 2.5 Flash had the highest Legal Accuracy (4.30) and Issue Completeness (4.82) averages but the lowest pass rate, 78%. Some of its answers were cut off; others skipped the disclaimer entirely and failed on that dimension. For legal work, an answer that is excellent most of the time and incomplete the rest is hard to rely on.
How reliable is an AI judge?
Claude judged all three models, including itself. That raises three problems we could not remove in this run: a possible home-court advantage, a bias toward answer structures the judge recognises, and no ground truth, because no attorney checked the answers. What the scores measure is agreement with one model's view of a good answer.
Our later runs addressed this. The September 2026 Opus 5.5 benchmark used two non-Anthropic judges, and the trap-question test used three blind judges and published all 80 answers.
Did the results hold on fresh questions?
The top-100 posts are old enough to have appeared in training data. To check for that, we ran the same pipeline on 20 questions posted to r/legaladvice in the previous 48 hours. Claude Sonnet 4 scored 95%, GPT-4o 90% and Gemini 2.5 Flash 85%. Twenty questions is a small sample, so we read this as "no sign of memorisation" rather than as an improvement.
What does this mean for a legal AI product?
Even in May 2026, the models could reason about common US legal questions well enough to be useful most of the time. The remaining 5 to 15% is where a product earns trust: checking which model's answer to keep, adding the caveats the models skipped, grounding answers in the firm's jurisdiction and documents, and making every citation traceable to its source. That is the work Justinian, HAQQ's legal AI engine, does on top of the models it runs.
Methodology
- Run date: May 2026.
- Questions: the 100 top posts of all time on r/legaladvice, plus 20 posts from the previous 48 hours, filtered to substantive text posts.
- Models: Claude Sonnet 4, GPT-4o and Gemini 2.5 Flash, all called through OpenRouter with identical system prompts.
- Prompting: chain of thought with a five-step legal reasoning framework (jurisdiction, issues, law, analysis, advice).
- Judge: Claude Sonnet 4, single judge, five dimensions scored 1 to 5, no attorney review.
- Pass rule: mean of at least 3.5 and no dimension below 2.
- Data: code, questions, answers and scores were kept in a repository at the time of the run.
