Lately I’m seeing more and more AI models calling themselves “open source”. Llama, Mistral, Falcon… they all claim to be “open”. But are they really? How open are they actually?
I recently discovered the European Open Source AI Index (OSAI), a European initiative doing excellent work systematically evaluating how open generative AI models really are.
What is the OSAI Index?
The OSAI Index is a community-driven public resource hosted at Radboud University (Netherlands). Its mission is simple but important: to evaluate the real openness of generative AI models (text, images, code, video, and audio) beyond the marketing.
What’s interesting is that it’s not a subjective opinion. They have a 14-parameter evaluation system that measures different aspects of a model’s openness.
The 14 Evaluation Parameters
What I like most about OSAI is that their evaluation is very detailed. It’s not just a simple “yes/no”. They analyze:
- Base Model Data - Are training datasources documented and available?
- End User Model Data - Same for the model the user interacts with?
- Base Model Weights - Are the weights freely available?
- End User Model Weights - And for the user model?
- Training Code - Is all pipeline code available?
- Code Documentation - Is it well documented?
- Hardware Architecture - What hardware was used for training?
- Preprint - Are there technical preprints available?
- Paper - Any peer-reviewed scientific publications?
- Modelcard - Standardized model documentation?
- Datasheet - Datasheet following Gebru et al. standard?
- Package - Packaged for easy installation (PyPI, etc.)?
- API and Meta Prompts - Unrestricted API available?
- Licenses - OSI-approved licenses for EVERYTHING (data, code, weights)?
The “Openwashing” Problem
This is where the OSAI Index really shines. They identify what they call “openwashing”: when a model is marketed as “open source” but doesn’t meet real open source standards.
A very recent example: Lumo, Proton’s AI assistant. OSAI analyzed it and classified it as “the least open ‘open’ model we’ve ever evaluated”.
So it was sold as open but was actually almost as closed as any traditional proprietary solution.
Why This Matters for Us as Developers
I think this is important for several reasons:
Technical Transparency
When you choose a model for a project, you need to know:
- Can I inspect how it works?
- Can I audit training data?
- Can I modify and redistribute it?
- What legal restrictions do I have?
Independence
If the model isn’t truly open:
- You depend on the provider for changes
- You can’t audit behavior
- You can’t self-host
- You’re tied to their terms of service
Regulatory Compliance
With the EU AI Act and other regulations, you need models you can audit. Real open source gives you control; “openwashing” gives you false security.
The Ranking: Truly Open Models
According to the OSAI Index, the most open models currently include:
- OLMo (Ai2) - Very highly rated for openness
- BLOOMZ (BigScience Workshop)
- Pythia (EleutherAI)
- OpenLLaMA
- SmolLM (HuggingFace)
And curiously, very popular models like Llama (Meta) or Mistral don’t fare as well when evaluating all 14 parameters. Yes, they release the weights, but many things are missing: training code, data, complete documentation…
My Personal Opinion
I’ve been following the open source world for years, and I believe transparency matters especially in AI.
With traditional software, if it’s not open source, you can at least read the spec, understand the API, use it. With AI, the model is an opaque block: you don’t know what it learned, nor how, nor what biases it has.
If a model claims to be “open source” but:
- Doesn’t share training data
- Doesn’t document the hardware used
- Doesn’t allow real redistribution
- Uses licenses with hidden restrictions
…then it’s not open source. It’s “openwashing”.
How to Use the OSAI Index
My recommendation is simple: before choosing a model for a serious project, check the OSAI Index.
The process:
- Go to osai-index.eu
- Filter by type (text, image, code…)
- Compare models you’re interested in
- Review the 14 parameters
- Make an informed decision
For example, if you’re evaluating between three language models:
- Check if they have proper modelcards
- Verify if licenses are OSI-approved
- Look for peer-reviewed papers
- Check if you can audit the training
Conclusion
The European Open Source AI Index is a necessary and timely tool.
In a world where everyone claims to be “open”, you need an objective evaluation. OSAI’s 14 parameters give you exactly that: a way to distinguish marketing from reality.
For us developers and system architects, this means we can make more informed decisions about which models to use, how to deploy them, and what risks we’re taking.
And that, at the end of the day, is what matters: real transparency, not marketing slogans.
Resources:
Comments