📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project trained a large-scale Italian LLM from scratch, achieving impressive technical results but performing poorly on academic benchmarks. This reveals scaling challenges in sovereign-language models and questions the investment needed for true language and knowledge depth.
Italy’s Minerva-3B, a large language model trained from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, revealing significant challenges in achieving country-specific language and knowledge depth despite substantial investment.
The Minerva project, led by Sapienza University of Rome’s NLP group and supported by Italy’s national AI infrastructure, trained models ranging from 350 million to 7 billion parameters using a dataset of 2.5 trillion tokens, half of which was Italian. The project aimed to develop a sovereign Italian LLM from scratch, contrasting with approaches like Portugal’s AMÁLIA, which extended multilingual models with smaller native-language data.
While Minerva’s models outperformed comparable multilingual models on Italian benchmarks, the 3B model’s performance on the INVALSI Italian school-exam benchmark was only 4.9%, close to chance. Researchers noted that dataset size and parameter count are more critical for complex language tasks than pre-training data composition alone, indicating that scale remains a limiting factor.
This empirical result suggests that even significant native-language investment may not suffice at current parameter scales to produce models with deep country-specific knowledge, raising questions about the effectiveness and future direction of sovereign-LLM strategies in Europe.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Fine-Tuning Large Language Models: From Custom Datasets to High-Performance AI Models Using Modern Toolchains
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
LLM Performance Evaluation: How to Build Automated Testing Pipelines, Benchmark Models, and Validate AI Applications Before Production
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Accelerate Everything with Tensor Cores: A Developer’s Guide to High-Performance AI, Efficient Training, and Scalable Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Mastering Small Language Models: A Practical Guide to Building Lightweight NLP Systems with Python, Transformers, and Quantization Techniques
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications of Minerva’s Benchmark Performance
The results demonstrate that large-scale training from scratch, even with substantial native-language data, may not be enough to achieve meaningful country-specific knowledge in LLMs. This challenges assumptions that simply increasing native-language data and parameters will produce models capable of complex academic and cultural tasks, emphasizing the need for more targeted or scaled investments.
For European sovereign AI initiatives, the findings highlight a potential scaling barrier, suggesting that current parameter sizes might be insufficient for truly deep language and knowledge understanding, which could influence future funding and strategic decisions.
European Sovereign LLM Development Strategies
Italy’s Minerva project represents a deliberate choice to build a large, native-language LLM from scratch, contrasting with approaches like Portugal’s AMÁLIA, which focused on continuation pre-training of multilingual models with smaller native-language data shares. The project was supported by Italy’s national AI strategy, involving extensive government funding, supercomputing resources, and a dedicated research team.
Despite the technical success in outperforming multilingual models on benchmarks, Minerva’s poor performance on academic content tests underscores a broader debate about the scale and scope of native-language investments needed for effective sovereign-LLMs. Prior European efforts have largely focused on multilingual models or smaller native-language models, but Minerva’s results suggest that scale remains a fundamental challenge.
“Even with 50% Italian data on 660 billion tokens, the model performs near chance on academic benchmarks, indicating scale limitations.”
— Research team member, Orlando et al.
Unanswered Questions on Scaling and Effectiveness
It remains unclear whether increasing parameter sizes further or adopting different training methodologies could significantly improve Minerva’s performance on complex tasks. The long-term scalability and the optimal investment level for sovereign-language models are still subjects of ongoing research.
Additionally, how these findings translate to other languages and domains in Europe is not yet established, and the impact on future policy and funding decisions remains to be seen.
Next Steps in European Sovereign-Language Model Research
The Minerva team plans to continue iterating on training methodologies, including ongoing experiments with continual training and larger models. Further evaluations on diverse benchmarks are expected to clarify whether scaling alone can overcome current limitations.
European policymakers and researchers will likely reassess native-language investment strategies in light of these results, potentially shifting focus toward larger models or alternative architectures to achieve deeper country-specific knowledge.
Key Questions
Why did Minerva perform poorly on the Italian school exams?
Despite extensive training on a large native-language dataset, the model’s limited parameter size and the inherent complexity of academic tasks likely contributed to its low performance, highlighting scaling challenges.
Does this mean native-language models are not worth the investment?
Not necessarily. The results suggest that current parameter scales may be insufficient; larger models or different training approaches might be needed to realize the full potential of native-language models.
How do Minerva’s results compare to multilingual models?
Minerva’s models outperform multilingual models on Italian benchmarks but still fall short on complex academic tests, indicating that native-language focus alone does not guarantee deep knowledge without sufficient scale.
What are the implications for European AI policy?
The findings suggest that achieving country-specific AI capabilities will likely require significant scaling investments, potentially influencing future funding and strategic priorities across Europe.
Source: ThorstenMeyerAI.com
