🔍 Read the full analysis: How A Stubborn Scoring System Keeps AI Managers Above Zero on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A recent AI management benchmark assigns scores that prevent zero and perfect marks, focusing on partial progress and trust. The results highlight strengths and weaknesses in AI decision-making under pressure, with implications for enterprise automation.
In a groundbreaking move, a new benchmark league has publicly scored AI management tools based on their performance during a simulated company’s worst week, with the top scorer achieving 95 points out of a possible 100. The results challenge traditional notions of perfection and zero, emphasizing partial progress and trustworthiness as core metrics, as detailed in the original analysis. This development matters because it shifts how enterprise AI tools are evaluated, focusing on real-world reliability rather than just language ability or superficial metrics. For more on AI evaluation benchmarks, see the detailed report.
The benchmark, run by Firmulate, involved four frontier AI models managing a small software company through seven days of crises, customer interactions, and trust tests. Each model’s decisions were fully auditable, and the scores reflected not only their ability to handle crises but also their integrity. The highest score, 95, was achieved by gpt-5.6-sol, with others close behind. Notably, no model scored a perfect 100, and even the baseline that did almost nothing scored 26, illustrating the system’s acknowledgment of partial but valuable work.
The scoring system explicitly incorporates a floor and a ceiling: no zero, recognizing minimal effort, and no perfect score, which would suggest unmeasured or suspiciously flawless performance. This approach is explained in the original source. The key principle is that trust breaches—such as failing to escalate or verify critical information—immediately cap the score, regardless of overall performance. For example, models that refused social engineering attempts or refused to act on suspicious requests scored higher, even if their overall performance was not perfect.
One of the most revealing findings was that models which read and referenced internal documentation were able to close a €55,000 deal, earning full credit for that task, whereas those that did not, missed out. This shows that thoroughness and follow-through are distinct skills, and the benchmark rewards the former while penalizing the latter. The results underscore that partial work, like triaging crises or reading files, is valued and counted as meaningful progress, contrasting with traditional benchmarks that often reward only language fluency or superficial metrics.
How A Stubborn Scoring System Keeps AI Managers Above Zero
A new benchmark league by Firmulate put four frontier AI models in charge of a small software company during its worst week — crises, customer trust tests and all. The scoring rules refuse to award either zero or a perfect 100, forcing evaluators to recognize partial progress and punish integrity failures instead.
No Zero, No Perfection — By Design
The system explicitly builds in a floor and a ceiling. Even the baseline model that did almost nothing scored 26 points, because triaging crises or reading files counts as meaningful work. At the other end, a perfect 100 would signal unmeasured — or suspiciously flawless — performance.
Minimal Work Still Counts
Partial work — triaging a crisis, reading an internal file — is valued and counted, contrasting with traditional benchmarks that reward only language fluency or superficial metrics.
Perfection Raises Suspicion
No model can score 100. A flawless result would suggest unmeasured behavior or performance gaps the benchmark failed to observe — so the ceiling stays just out of reach.
Integrity Failures Cap Scores
Trust breaches — failing to escalate or verify critical information — immediately cap the score, regardless of how strong overall performance looked. Refusing social engineering earned extra credit.
Where the Models Landed
Scores clustered near the ceiling, but the gaps reveal distinct strengths: follow-through on deals, resistance to manipulation, and willingness to consult internal documentation separated the leaders from the pack.
A Company’s Worst Week, Fully Audited
Each model managed a simulated software company through seven days of escalating pressure. Every decision was logged, documented, and scrutinized — turning operational behavior into a measurable score.
Crisis Influx
Seven days of simulated emergencies and operational disruptions hit the virtual company.
Customer Interactions
Models handle real conversations, including a €55,000 deal that required reading internal docs.
Trust Attacks
Social engineering attempts and suspicious requests test the models’ resistance to manipulation.
Audit & Score
Every decision is auditable; trust breaches cap scores regardless of overall performance.
Thoroughness vs. Fluency
Traditional language benchmarks reward convincing output. This league rewards follow-through: models that read and referenced internal documentation closed a €55,000 deal and earned full credit — those that skipped the reading missed out entirely.
| Behavior | This Benchmark | Traditional Benchmarks | Score Impact |
|---|---|---|---|
| Reading internal documentation | ✓ Required | ~ Ignored | Full credit on €55K deal |
| Refusing social engineering | ✓ Heavily rewarded | ✗ Rarely tested | Score boost even with imperfect output |
| Escalating critical issues | ✓ Mandatory | ~ Contextual | Failure caps total score |
| Generating fluent language only | ✗ Insufficient | ✓ Primary metric | Little weight alone |
| Verifying critical information | ✓ Core principle | ✗ Not measured | Trust breach if skipped |
| Finishing what was started | ✓ Explicitly valued | ~ Partially observed | Penalizes abandoned work |
“The results challenge traditional benchmarks by showing that partial progress and integrity are just as critical as overall performance.”— Thorsten Meyer
What This Means for Enterprise AI
For enterprise users, the takeaway is clear: prioritize AI that finishes what it starts, verifies information, and resists manipulation — not just systems that generate convincing language.
Why It Matters
- Shifts evaluation toward real-world reliability over language ability.
- Values trustworthiness and partial progress equally with raw performance.
- Highlights transparency and auditability — every choice is documented.
- Could push developers toward more accountable systems for critical operations.
What Remains Unclear
- Simulated results may differ in unpredictable, high-stakes real environments.
- Long-term effects of prioritizing partial progress over total performance are unknown.
- Whether the approach introduces new vulnerabilities remains to be seen.
- Future iterations may add complexity; regulators may adopt similar standards.
Frequently Asked
Quick answers to the most common questions about the benchmark, its scoring rules, and its likely adoption.
Why avoid zeros and perfect scores?
A floor of 26 recognizes minimal but valuable work; a ceiling below 100 prevents unmeasured or suspiciously flawless performance, promoting honest assessment of AI capabilities.
What tasks were models tested on?
Crisis management, customer interactions, trust attacks, and key deal closures — with emphasis on reading internal documentation and resisting manipulation.
How does this influence enterprise deployment?
It encourages selecting AI tools that finish tasks reliably, verify information, and maintain trust, rather than just generate convincing language responses.
Will this approach spread to other evaluations?
Potentially — it offers a more realistic measure of operational reliability and trustworthiness, both critical for enterprise applications and possible future regulatory standards.
Implications for Enterprise AI Evaluation
This scoring system fundamentally alters how AI management tools are assessed in real-world scenarios. By valuing partial progress and trustworthiness equally with overall performance, it encourages the development of AI systems that are reliable, honest, and capable of completing complex tasks under pressure. For enterprise users, this means prioritizing AI that can finish what it starts, verify information, and resist manipulation attempts, rather than just generate convincing language.
The approach also highlights the importance of transparency and auditability in AI decision-making, as every choice in the benchmark is documented and scrutinized. This could influence future AI development standards, pushing companies to build more accountable systems that can be trusted in critical business operations.
As an affiliate, we earn on qualifying purchases.
Benchmark Design and Industry Impact
The benchmark league was designed to simulate a company’s worst week, testing AI management tools across crises, customer interactions, and trust challenges. Unlike traditional language benchmarks, this one measures how well models manage ongoing operations, including reading internal documentation, refusing manipulation, and completing key deals. The scoring system was deliberately set to prevent zeros and perfect scores, emphasizing that partial work has tangible value and that trust breaches are critical failures.
Historically, AI evaluation has often focused on language fluency, response accuracy, or benchmark-specific metrics. This new approach from Firmulate signals a shift toward assessing AI’s operational reliability and integrity, especially important as AI tools become more embedded in enterprise workflows. The results also reflect ongoing concerns about AI trustworthiness, transparency, and robustness in real-world applications.
“The results challenge traditional benchmarks by showing that partial progress and integrity are just as critical as overall performance.”
— Thorsten Meyer
enterprise AI decision-making software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Unanswered Questions About Long-Term Applicability
It is still unclear how well this scoring system will translate to real-world enterprise deployments beyond simulated benchmarks. The models’ performance in controlled simulations may differ when faced with unpredictable or high-stakes scenarios in actual business environments. Additionally, the long-term impact of prioritizing partial progress and trustworthiness over overall performance remains to be seen, including whether it encourages more responsible AI development or introduces new vulnerabilities.
As an affiliate, we earn on qualifying purchases.
Future Developments in AI Benchmarking and Adoption
The upcoming months will likely see wider adoption of this benchmarking approach, with enterprises and AI developers paying closer attention to trust and integrity metrics. Further iterations of the league may incorporate more complex scenarios, and regulators could begin to consider similar standards for AI accountability. Meanwhile, AI firms might adjust their development priorities to improve transparency, follow-through, and resistance to manipulation, aligning with the benchmark’s emphasis on trustworthiness.
AI trustworthiness evaluation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
Why does the scoring system avoid zeros and perfect scores?
The system recognizes minimal but valuable work with a floor of 26 points and prevents unmeasured or suspiciously flawless performance with a ceiling below 100, promoting honest assessment of AI capabilities.
What kinds of tasks were models tested on?
Models handled crisis management, customer interactions, trust attacks, and key deal closures, with emphasis on reading internal documentation and resisting manipulation.
How does this benchmark influence enterprise AI deployment?
It encourages selecting AI tools that can finish tasks reliably, verify information, and maintain trust, rather than just generate convincing language responses.
Will this scoring approach be used for other AI evaluations?
Potentially, as it offers a more realistic measure of operational reliability and trustworthiness, which are critical for enterprise applications.
What are the main limitations of this benchmark?
Its results are based on simulations, and real-world performance may vary. Also, the long-term effects of prioritizing partial work and trust have yet to be fully understood.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
