Evaluating open source projects is a permanent challenge that OW2 has chosen to meet by defining a unique composite indicator now applied to its projects. This indicator facilitates the evaluation of open source projects from the point of view of corporate information systems managers. The growth of open source software is taking place in two main directions. First, by moving up the layers of information systems: from the operating system to the business applications, open source software is increasingly used by non-IT specialists. Second, by becoming “mainstream”, open source is reaching out to decision-makers who are unfamiliar with open source. For these new users, educated in the commercial practices of proprietary software vendors, open source remains a counter-intuitive model; its technical, legal and community specificities are a source of uncertainty that is not very favourable to positive decisions. Mainstream decision-makers must hear a language they understand. This is the role of the Market Readiness Levels (MRL) method developed by OW2 for evaluating open source projects. With MRL, decision-makers have a familiar indicator that positions open source projects according to the “business” decision criteria they are used to. Open source is moving towards them. The talk begins with a presentation of the MRL method, its three levels of analysis and the hundred or so criteria taken into account. It then gives the example of a few OW2 projects evaluated by the method and explains what benefits it brings to the development teams, but also to the end-users and the open source in general

1. Cedric Thomas, 2021
The OW2 Market Readiness Levels
Improving Productivity and
Market Readiness of OW2 projects
SFScon, Bolzano 2021

2. 2021
About OW2 A global non-profit open source organization
Promote the development of a portfolio of open-source components for
enterprise information systems.
Foster the growth of a community and business ecosystem.
OW2 is a community-driven organization
Code base of some 100 third-party projects developed by communities
and members. OW2 does not own project IP.
Independent, open source governance: Board of directors, Technology
Council and Project Leader accountability.
Focus on project quality and market readiness
Launch of third generation support program: Market Readiness Levels
methodology applied to OW2 projects.
Independent technical infrastructure and community governance system
provide input to market readiness assessment.

3. 2021
Meet the Conventional
Decision Maker
Not an open source activist
May be wary of open source
Educated with proprietary software
A corporate executive
May be away from technology
Reports to a boss (CIO, CEO, GM, etc.)
Makes a business decision
Looks for long-term value
Decides without drilling down into details

4. Nov 11, 2021 4
2021
Aligning OSS Projects
with Market Expectations
Users do not want just code: Product attributes
create value for OSS projects
Market Readiness Levels (MRL) assessment
helps create market value for OW2 projects
Developer User
Code
POCs
Use-cases
Demonstrators
Doc.
Tutos.
Testing
Upgrades
Bug-fixing
Training
Support
Packaging
Case
studies
Collateral
Pricing
Contracts
Early
adopters
Partners
Complements
Etc.
Predictability
Quality
Trust
Market Value

5. Nov 11, 2021 5
2021
http://www.frankichamaki.com/wp
-content/uploads/2014/01/
nasa-trl.jpg

6. 2021
Developer User
Helping Decision Making with the
MRL Single Composite Indicator
Too much data creates confusion.
From detailed data to Market Readiness
Levels.
Rough Code
(useless)
Fully Supported Product
(useful)

7. 7
2021
OW2 Market Readiness Levels (MRLs) Inspired by the NASA TRLs
Significant market share and global customer base. Properly
financed and organized business support. Global active community.
Customer base of mainstream users. Appropriate financing. Active
community support and contributions. Recognized software
Established product. Customer base of early and mainstream users.
Stable financing. Open to community support and contributions.
Proven product. Customer base of early users. Project fit for third
party contributions. Implicit community governance
Some customers, recent market opening, Core team of developers,
untested open source governance
Several users, project leadership well established
MVP stage One declared user (can be company internal) with
declared project leader
POC stage. Basic R&D code developed with one demonstrated use
case, some documentation
Basic R&D code developed
MRL 9 – Market Leader
MRL 8 – Established Outsider
MRL 7 – Established Business
MRL 6 – Sizable Adoption
MRL 5 – Fair Adoption
MRL 4 – Usefulness Verified
MRL 3 – Fledgeling Usefulness
MRL 2 – Product Development
MRL 1 – Basic Early Stage
“flight proven”
“flight qualified”
“prototype in space”
“proven demo”
“relevant envt validation”
“lab validation”
“proof of concept”
“application formulation”
“basic principles”
Nasa TRLs

8. 8
2021
Stage 1: Best Practices Mapping
Project Communication
Project Community
Project Documentation
Development Environment
Project Organisation and Mgt
Project Licenses
Development Process
Testing Process
Release Management
Security and Vulnerability Mgt Attention to security vulnerabilities, mechanisms and resources to deal with them.
Enablement and maintenance of the integrity of the product and its releases.
Implementation, quality and maintenance of the testing process.
Development practices that help improve code quality.
Open source licenses and copyright handling by the project.
Management of requirements and planning and estimates of project activities.
Provisioning of development resources and facilitation of bug reports and commits contribution.
Documentation facilitating usage of the project and contribution to it.
Status of the community and mechanisms that support third party contributions.
Willingness and ability of the project to communicate, to be easy to deal with.
●
50 Control points
●
Verified self-declared form

9. 9
2021
Community
Activity
Compliance
Quality
Sustainability
Stage 2: Data Collection and
Attribute Modeling
0.0
1.0
2.0
3.0
4.0
5.0
Community
Quality
Activity
Compliance
Sustainability
●
20 Control points
●
Automatic data collection

10. 10
2021
Stage 3: Market Capabilities Assessment
0 1 2 3 4 5
Product No-user, R&D,
POC
Beta, Packaged,
Documentation
Full Q&A, Doc
etc.
UI, APIs and
Interoperability
Product Range Platform Sub-
Project
Generation
Contributors Research
partners
Declared OSS Active OSS
project leader
Occasional third
party contrib.
Regular third
party contributors
Broad active
project
community
Support No support Help contact
identified
Community best
effort
Committed bug
fixing
Direct
commercial
support
Direct and
partner support
Customers No customers
nor users
Early users, not
customers
First customers Early customer
base
Growing
customer base
Established
customer base
Sales No sales Random sales by
founders
Fledgeling sales
by sales
organisation
Sales from new
customers
Repeat business
from customer
base
Sales from
customer base
and partners
Finances R&D subsidies Founders,
Friends and
Family
Sponsor or
professional early
stage
Early stage plus
customer money
Appropriate
financing
Positive cash
flow
Recognition No market
recognition
Basic identity,
website
Developing
marketing
Active marketing Market
recognition
Leadership
recognition
●
42 Market-oriented situations
●
Expert interview

11. 11
2021
Computing the Score
OSS best practices via self-declared form
Project metrics from development platform
Market capabilities via expert interview
X X
X
X
X
Stage 1 Stage 2 Stage 3

12. 12
2021
MRL Is Applied
to 22 Projects
in the OW2
Code Base
ASM
Authzforce
CLIF
DocDokuPLM
FusionDirectory
Imixs-Workflow
Joram
Knowage
LemonLDAP::NG
Lutece
OCS-Inventory
Proactive Scheduler
Rocket.Chat
Sat4j
SeedStack
Spoon
Sympa
Telosys
Waarp
WebLab
XWiki
ZenRoom

13. 2021
MRL Runs On our Own Dedicated
Infrastructure
Simplified and fully maintained by OW2
Data
Collection
Jira
ScanCode
SonarQube
GitLab
OMM Xwiki
GitHub
Storage and
Compute
SQL DB
Project
dashboard
MKT Xwiki
Best practices
(project lead form)
Metrics (automatic
data collection)
Market capabilities
(expert interview)

14. 14
2021
1- Metrics
2- Collectors
3- Models 4- MRL
5- Promotion
Collaterals
Presentations
Exhibitions
Sources
Lists
Formats
Extensions
Updates
Automation
Forms
Updates
Normalisation
OSCAR
CMR:MRL
Updates
IHM
Webpages
Feedbacks
Comparable
Integration
Project Phasing

15. 15
2021
MRL 2.0
Beyond OW2
Project Timeline
2018-2019 2020 2021
Code base deployment
OW2 Mature projects
Code base deployment
OW2 mature projects
2022
Tools development
Feasibility testing
Concept & Methodology
Acceptability verification

16. 2021
Enhancing OSS Confidence
and Predictability
For project users
Assess reality of OSS strategy: transparency on
OSS code quality, stability, maintenance.
Be sure the OSS project is build-ready... and is
somewhat real (no fake “community edition”)
For the project team
Continuous project analysis
Dashboard to follow issues and contributions
management, code quality, compliance.
Weekly project builds
Detection of build error (like broken
dependencies), not just upon commits (like CI).
MRL is helpful for both
developers and users.
Good MRL rating
shows the OSS code
builds, how the project
is alive and healthy,
and how it is already
adopted.

17. 2021
Conclusion: Reality Check Strengths
It works!
Useful for developers and users
State of the art OSS expertise
Transparent, open, not a black box
Simple, accessble, adaptable
Limitations
Not an evaluation of functions or features
Business perspective, not a scientific one
Snapshots of projects in evolution
Calibrated for the OW2 code base
Still to do: Robustness/Sensitivity analysis

18. 18
2021
www.ow2.org
For more details please contact Cedric Thomas, OW2 CEO, cedric.thomas@ow2.org
And now let's talk
Q&A
Disagreements
Complements
Feedback
etc.
Thank You