Managing the evolution of F/OSS with Model Driven Techniques

Università degli Studi dell’Aquila

Alfonso Pierantonio, Davide Di Ruscio
Dipartimento di Informatica
Università degli Studi dell’Aquila (Italy)
alfonso.pierantonio@univaq.it

2
→ Linux distributions are
among the most complex
software ecosystems

→ More than 29,000
interdependent packages
→ Keynote at SLE11 by Krzysztof
ca. 10,000 features only in the
kernel
GTTSE 2011 Alfonso Pierantonio – Managing the evolution of F/OSS with Model-
Driven Techniques

3

explicit dependencies among linux packages

GTTSE 2011
Alfonso Pierantonio – Managing the evolution of F/OSS with Model-Driven Techniques

4
Free and Open Source Software
→ From tarballs to distributions
→ Package technology

Problem: upgrade failures
Abstracting the System
→ Modeling languages for describing the System
→ Maintainer Script Analysis

→ Injectors

Analyzing the System
→ Upgrade simulator
→ Fault detector

Conclusions
GTTSE 2011

5

Today
Alfonso Pierantonio
→ A gentle introduction
→ Models of Complex Systems

July 6, 2011 – 14:45
Davide Di Ruscio
→ Concrete Artifacts
→ Demo

GTTSE 2011

6

FOSS distributions are among the most complex
software systems known, being made of thousands
components evolving rapidly without centralized
coordination

Large numbers (components/developers)
→ SourceForge.net contains more than 123,736 projects,
1,342,153 “users”
→ Debian GNU/Linux is distribuited with more than
29,000 pre-compiled packages

GTTSE 2011

7

Numbers like that have the potential to radically
change the way we produce and study software, in
particular the complex systems of the future

complexity dependencies

8

In the beginning it was the tarball
Before the advent of distributions, the peculiar way
to install free software on client machine was:

server client
side side

bazaar proj 1 user
proj 2 installations

proj n

GTTSE 2011

9
To answer these problems, GNU/Linux distributions have
born as intermediaries between FOSS projects and their
users

FOSS proj 1 server client
bazaar proj 2 side side

user
proj n installations

package
management
meta-
installer
distribution
editors

package repository

10

FOSS distributions are among the most complex
software systems known, being made of thousands
components evolving rapidly without centralized
coordination

FOSS components are provided in packaged form by
distribution editors
Packages define the granularity at which components
are managed using package manager applications

GTTSE 2011

11

GNU/Linux distributions
Central notion in distributions (to abstract over the
complex underlying infrastructure):
→ package, together with package management software

package
management

distribution
editors

package repository

GTTSE 2011

12

GNU/Linux distributions
Central notion in distributions (to abstract over the
complex underlying infrastructure):
→ package, together with package management software

package
management

distribution
editors

And yet, doing things right can be
package repository

extremely difficult!

GTTSE 2011

13

Installation intricacies
In state of the art distributions:
→ knowing if the installation process terminates without errors
is undecidable (reason: configuration scripts) … today
→ there is no guarantee that distribution repositories contains
only installable packages
→ a large number of errors faced by users are dependency
resolution errors
─ more than 29,000 packages (and 10,000 sources) in Debian
─ more than 200.000 inter-package relationships

GTTSE 2011

14

PROBLEM: UPGRADE FAILURES

15

→ each phase can fail – it actually happens quite often!
→ efforts should be made to identify errors as early as
possible, if not predict

16

Upgrade failures
Essentially they can fail for the following classes of
problems
→ Consistency breach due to administrator operations at a
finer level of granularity than that of packages
─ eg. needed resources are manipulated or removed from the
command-line
→ Implicit dependencies are not considered in the
installation/removal script in the upgraded packages
→ Script erroneous behavior

GTTSE 2011

17

Installing package
libapache-mod-ssl
Configuration 1

> apt-get install libapache-mod-ssl

+

Configuration 2

21

once mod_ssl is
installed, it is enabled in
apache

GTTSE 2011

22

removing package
libapache-mod-ssl
Configuration n+1

- > apt-get remove libapache-mod-ssl

Configuration n

23

GTTSE 2011

24

It disables mod_ssl in apache.

What happens if the maintainer
does not write this statement ?

GTTSE 2011

25

An “inconsistent” configuration is
reached and it is detected only at
run-time:

the package dependency
metadata are not enough !

GTTSE 2011

26

Problem: Upgrade failures
Current tools are able to predict a limited set of
upgrade failures before deployment

When trying to predict upgrade failures, existing tools
only consider static package metadata and the
behaviour of the maintainer scripts is completely
ignored
→ This leaves a wide range of failures unpredicted

GTTSE 2011

27

Upgrade failures classification
Static deploy-time occur when a static requirement is
violated during the upgrade.
→ the low-level package manager fails at deploy-time, aborting the
upgrade process
Dynamic deploy-time occur when a maintainer script fails
→ such failures can originate from errors ranging from syntax to
failures in external tools
→ they are not addressed by state of the art package managers
Undetected failures remain undetected through upgrade
deployment
→ the upgrade has been completed successfully, but the obtained
system configuration contains undetected incoherences

GTTSE 2011

28

IDEA : ABSTRACTING AND ANALYZING THE SYSTEM

29

Idea
Extracting relevant information from a linux box
(«snapshot») at a given moment of time
Analysis and upgrade
simulation on the
«snapshot»

injectors

«snapshot»

Linux box

30

Ingredients 1
Modeling languages for describing the several aspects of a
linux distribution
→ Packages
─ including maintainer scripts
→ System Configuration
─ Installed packages
─ Configuration files
─ MIME-type handlers
─ Alternatives
─ etc

Injectors for harvesting the system and building the models
→ collection of injectors

GTTSE 2011

31

Ingredients 2
Modeling language for the maintainer scripts
→ scripts are written in POSIX languages whose semantics is far
from being simple, although
→ maintainer scripts does not harness the full expressivity of
such languages (template-based “macro-language”)

Maintainer scripts as
models
→ which semantics ?

GTTSE 2011

32

Ingredients 3
Transformational semantics for simulating the
behavior of the maintainer scripts on the system
«snapshot»
→ M2M transformations obtained by «compiling» the
maintainer scripts into ATL

Fault detector, a general mechanism for performing
queries over the «snapshot» for digging the model and
search for inconsistencies

GTTSE 2011

33

Configuration 1
?
Configuration 2
package injection
system injection

M2M
transformation
M2M transformation

«snapshot» 1 «snapshot» 2

34

A model-based approach is introduced to support the
package upgrades and enhance the failure detection
possibilities:
→ the simulator is used to predict the effect of maintainer
script executions (deploy-time failures)
→ the fault detector is used to deal with undetected failures

35

EVOSS

A model-based approach is introduced to support the
package upgrades and enhance the failure detection
possibilities:
→ the simulator is used to predict the effect of maintainer
script executions (deploy-time failures)
→ the fault detector is used to deal with undetected failures

36

MAINTAINER SCRIPT ANALYSIS

37

Rigourous maintainer script analysis (1/2)
Script analysis for Debian and RPM based distributions
To support the analysis and represent the results a
database of scripts has been developed
→ Already known templates have been identified
─ By taking into account annotations in the scripts (when available,
eg. in case of Debian scripts)
─ Exact match of existing templates in the scripts
→ New templates have been identified
─ By using an “exact match” between scripts
─ By analyzing block statements (if, while, case, for, etc.)
─ by using a similarity based approach

GTTSE 2011

38

Rigourous maintainer script analysis 2/2

http://evoss.di.univaq.it

40

Used to represent in terms of
models packages and system
configurations

GTTSE 2011

41

Configuration and package injectors

By means of the model injection, given a linux system a
corresponding model is obtained
Injectors inspect software artifacts and inject relevant
information into corresponding models
The process is driven by the metamodels
GTTSE 2011

42

Configuration injector
Debian Mandriva Caixa Magica Ubuntu
Model Injector Model Injector
… Model Injector Model Injector

Mancoosi
Model Injection Infrastructure

Mancoosi Model Management

cfginject [-d <distribName>] [-full]
→ It creates a system configuration model starting from a real system. It is
possible :
─ to specify the distribution name
─ to force the complete regeneration of the model. By default incremental
changes of the model are performed in order to keep the system and the
corresponding model synchronized

DEMO

43

Package injector

pkginject [--installed]
[--package <package file name>]
[--cachedir]
→ It creates package models. By default, a model for each installed
package is created (--installed)
→ It is possible to inject only one package by specifying the package file
name (--package)
→ It is also possible to inject all the packages which are in the cachedir (--
cachedir)
DEMO

45

Upgrade Simulator

Used to simulate system
configuration upgrades

GTTSE 2011

46

Input
Output
Upgrade plan: New
{(p1,u1),(p2,u2)…,(pn,un)} Configuration
Model
Configuration Upgrade
Model Simulator Error
Model

{p1, p2, …, pn}
Package Model1
Package Package Model2
Injection …
Package Modeln

GTTSE 2011

47

Configuration
Upgrade plan: Model
{(p1,u1),(p2,u2)…,(pn,un)
}

For 1≤i ≤ n : Package Model1
retrieve the
state of pi Package Model2
…
ui pi state Package Modeln

For each pi simulate
upgrade scenario New
Configuration
valid simulation Model
error found
Error
Upgrade Simulator Model

GTTSE 2011

48

install Not installed
Configuration
Model Package Modeli

preinst install
“Files are unpacked” OK FAILED

FAILED
postinst configure postrm abort-install

Half-configured
OK OK

Upgrade scenario FAILED
Installed

New Not installed
Half installed
Configuration “Reinst required”
Model
Error
Model

GTTSE 2011

49

Package Script Simulator
Model
Script
Script
Script Retrieval of Model
transformations
St1
St2
… Repository of
Stn Model Model transformation
transformations orchestration
Configuration generation
Model
Model
New transformation
Configuration
Model
orchestration
execution
Error
Model

GTTSE 2011

50

Configuration
Tst1
Model

CM1
Script
Tst2
st1
Orchestration
st2 Error CM2 of model
… Model transformations
stn …
CMn-1
New
Configuration Tstn
Model

GTTSE 2011

51

GTTSE 2011

52

GTTSE 2011

53

swi-prolog
5.6.58
<?xml version="1.0" encoding="UTF-8"?> [Not-Installed]
<selectionStates xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<selectionState type="install"> install
<param name="package" value="swi-prolog"/>
<param name="version" value="5.6.58"/>
swi-prolog
<param name="architecture" value="all"/>
5.6.58
</selectionState> [Installed]
<selectionState type="remove">
<param name="package" value="swi-prolog"/> remove
<param name="architecture" value="all"/> swi-prolog
</selectionState> 5.6.58
<selectionState type="install"> [ConfigFiles]
<param name="package" value="swi-prolog"/>
<param name="version" value="5.6.58"/> install
<param name="architecture" value="all"/>
</selectionState> swi-prolog
<selectionState type="install"> 5.6.58
<param name="package" value="swi-prolog"/> [Installed]
<param name="architecture" value="all"/> upgrade
</selectionState>
</selectionStates> swi-prolog
5.7.59
[Installed]
Sample upgrade plan

DEMO

55

Used to check system configurations for faults
that might give place to failures in the real
system

GTTSE 2011

56

Fault detector
Discovering implicit dependencies among packages: we are
able to discover dependencies that are not declared into the
package's meta-information
Discovering missing configuration files: according to the system
configuration model, some configuration files are required but
they are not available in the system
Discovering Mime-type dangling handlers: according to the
available information, the considered system should be able to
manage a mime type, but the corresponding handler is missing
in the system
Discovering missing services: the init.d file contains services
that should start at the system start-up; by querying the
configuration model, it is possible to detect missing services
…

GTTSE 2011

57

Fault detector architecture
MANCOOSI Server

-Libraries (.jar files) and OCL queries User machine
-Catalogue of faults
-Catalogue of solutions

FD-Server FD-Client

Web Portal

User

DEMO

58

Conclusions 1
The prediction of the package upgrade impact has
been enhanced: a larger number of cases which lead
to faulty behaviors can be detected
The core ingredients have been defined
→ Modeling languages for the different aspects
→ Injectors: system configuration and packages

→ Transformational Script Semantics
─ based on ATL
→ Fault detection: query-based global knowledge base
─ OCL and JAR queries to detect faults
─ Contribution from the community

GTTSE 2011

59

Conclusions 2
The main difficulty is in building the model injectors
→ labor intensive, deep platform knowledge required,
ad-hoc techniques
→ Interesting (academic) tools available
─ GRA2MOL – grammarware / modelware bridging
─ WIRES* – Model transformation orchestration
Monolithic real scale metamodels
→ Maintenance of the developed metamodels has been an issue:
interdependencies, ripple effects
→ Difficult stabilization of the support tools
Model comparison for validating the simulation
→ EMF Compare

GTTSE 2011

60

Conclusions 3
This work has been done within the
→ EU FP7 ICT STREP MANCOOSI
http://www.mancoosi.org

Industrial partners
→ Mandriva
→ Caixa Magica

→ IBM Ilog

GTTSE 2011

62

Questions?

Università degli Studi dell’Aquila

64
Upgrade plan:
{(p1,u1),…,(pn,un) Package
Package Upgrade
Package CM1 4.Simulator
} Model
Model
Model

1.Upgrade Package System
Planner 2. Injector 3.Injector CM2

(p,u) Real
System
Packages
5.Upgrade

Upgraded Real System
6. Injector CM3
System

Delta Configurations
Model 7. compare

Managing the evolution of F/OSS with Model Driven Techniques

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Managing the evolution of F/OSS with Model Driven Techniques

Semelhante a Managing the evolution of F/OSS with Model Driven Techniques (20)

Mais de Alfonso Pierantonio

Mais de Alfonso Pierantonio (14)

Último

Último (20)

Managing the evolution of F/OSS with Model Driven Techniques