🔍 Read the full analysis: Why Z.ai’s Legal AI Can Rank Near The Top With About 1 In 5 Tasks Passed on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal achieved a 19.17% all-criteria task pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models in the archived results. The same evaluation reports a 92.5% pass rate on individual criteria, showing how the benchmark’s all-or-nothing task rule sharply lowers the headline score. Z.ai’s claim that it leads open-source models is not fully verifiable from the published information because the comparison set is unspecified and the legal model’s weights were not found in the source review.
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, ranking seventh among 81 models in the archived results. The score means the model met every rubric requirement on about one in five evaluated tasks—not that it passed only one in five individual legal checks: Vals also reported a 92.5% criteria pass rate.
Z.ai describes GLM-5.3-FlashX-Legal as a model built for professional legal workflows, including document understanding, legal reasoning, research, contract review and drafting. Its developer documentation says the model underwent legal-domain mid-training and post-training, with reinforcement learning in what Z.ai describes as tens of thousands of simulated legal-workflow environments. The company recommends using it as an agent connected to services for laws, cases and company data.
The leaderboard lists the legal model at 19.17% of tasks passed and $0.15 per test. Its base model, GLM-5.3-Flash, scored 6.67% at $0.57 per test in the same archived table. That makes the difference 12.50 percentage points, matching Z.ai’s stated improvement. The legal model’s 92.5% criteria pass rate is a separate measure: it reflects individual rubric items, while the task score requires every item for a task to pass.
Z.ai’s October 13 blog post identifies Vals AI as the evaluator; Vals’ archived Version 1 leaderboard, updated October 18, lists the result. Z.ai’s documentation rounds the score to 19.2%. The company also calls the result “state-of-the-art performance among open-source models in this evaluation,” but the material provided does not define which models count as open-source or establish the full comparison set.
Why the All-Pass Score Matters
The result illustrates how a legal AI can rank near the top while still failing to complete most benchmark tasks under a strict definition of completion. Harvey’s benchmark counts a task as complete only if every required criterion passes. A task involving a report, for example, may include checks for facts, conclusions, citations, deadlines, recommendations and formatting. Missing a single required item makes the whole task a failure in the all-pass measure.
That distinction matters to law firms and clients weighing whether to use these systems. A high individual-criteria rate can indicate that a model often gets components right, while the lower task rate warns that outputs may still contain a gap that prevents the work product from being ready for review or use. The figures do not establish that the model is safe for unsupervised legal work, nor do they show how it performs on any particular firm’s matters.
Cost is another point of comparison, but it should not be treated as a measure of quality. Vals lists a low per-test cost for GLM Legal compared with several higher-ranked models. Z.ai’s separate pricing page lists token charges of $0.37 per million input tokens and $1.25 per million output tokens. Those are different cost measures, and neither alone captures the expense of connecting data sources, checking citations or having legal professionals review outputs.
Products Worth Considering
HP Sprocket Portable 2×3 Instant Photo Printer, Lilac, with ZINK Paper Bundle
FLEXISPOT Mochi-Pro Zero-Wall Oversized Power Recliner, Beige
Samsung Odyssey G4 25-inch FHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Harvey’s Benchmark Scores Tasks
Harvey released its Legal Agent Benchmark as an open-source benchmark on May 6, 2026. The benchmark’s public release contains 1,250 tasks across 24 practice areas, graded against more than 75,000 expert-written, binary pass-or-fail criteria. Harvey says each task combines an instruction, a client matter with relevant materials and a requested work product.
The public leaderboard uses a private holdout set that Vals AI says contains 120 tasks and mirrors the public task mix. Internet access is disabled during evaluation. Two AI judges grade submissions, and the benchmark score averages the task pass rates produced by those judges. Vals says its criteria pass rate counts a criterion only when both judges pass it.
Harvey’s May 26 baseline found a best overall all-pass rate of 7.1% among the frontier models it tested, and the company said those models completed fewer than 10% of tasks end to end in aggregate. That was a separate run and should not be treated as a direct comparison with Z.ai’s leaderboard result. It does, however, show why a score below 20% can still rank highly within a benchmark where full completion has been difficult for tested models.
“an all-criteria pass rate of 19.2%”
— Z.ai, in its developer documentation
Open Questions on Model Access and Reliability
The leaderboard result establishes performance on a defined benchmark, not reliability across legal practice or real client matters. It remains unclear how the model performs on tasks outside the holdout set, how results vary across practice areas, or whether it can consistently produce work that meets a firm’s requirements. The score also depends on the benchmark’s tasks, rubric and AI judging process.
The claim of state-of-the-art performance among open-source models needs a clearer comparison set. The source review found no published weights for the legal model, and the available Z.ai materials do not explain what “open-source” means in this claim. A separate four-task spot check described by ThorstenMeyerAI.com found citations to rulings that could not be located. That small check is not a benchmark and cannot establish a general error rate. The review also found errors in other models’ citations, including instances where a real ruling was described as supporting the opposite conclusion.
Information about client-data handling and deployment safeguards is also not settled by the score. The source review says Z.ai’s pages describe outputs as drafts for legal professionals, but it did not find a “not legal advice” notice on the pages examined. Firms would still need to establish who processes their data, under what rules, and how every citation and conclusion will be checked before work leaves the firm.
What Firms and Evaluators Should Check
Further evaluation should clarify the legal model’s availability, whether its weights can be accessed, and the precise set of models behind Z.ai’s open-source comparison claim. Updated leaderboard results or additional benchmark runs may also show whether its ranking holds across versions and evaluation settings. The documentation page cited in the source material did not carry a publication date, while the beta announcement was dated October 13 and Vals lists the release as October 12.
For potential users, the next practical step is not to treat the leaderboard as a deployment decision. Firms would need to test the model on their own workflows, set rules for human review, verify citations against authoritative sources and assess data-processing terms. Z.ai’s announcement said access to the model was through a beta opened by application; the source material does not say how broadly access will expand or when.
Key Questions
What does Z.ai’s 19.2% score mean?
It is the rounded all-criteria task pass rate reported by Z.ai. A task counts as passed only when every required criterion passes; Vals’ archived leaderboard gives the more precise figure of 19.17%.
How can the model pass 92.5% of criteria but only 19.17% of tasks?
The measures use different units. The 92.5% figure counts individual criteria; the task score counts only tasks where all criteria pass. A single missed requirement makes the whole task a failure under the all-pass rule.
Was GLM-5.3-FlashX-Legal the top-ranked model?
No. Vals AI’s archived Version 1 leaderboard placed it seventh among 81 models. The available material does not establish its position on later versions of the leaderboard.
Does the score prove the model is ready for legal work without review?
No. The benchmark measures performance on a specific set of tasks and does not establish that outputs are safe or complete for real matters. The source material describes the outputs as drafts for legal professionals, and human verification remains necessary.
Is Z.ai’s model open source?
The source material does not establish that the legal model’s weights are publicly available. Z.ai claims state-of-the-art performance among open-source models but does not specify the comparison set in the cited materials, so the label requires further clarification.
Source: ThorstenMeyerAI.com
