This a talk I gave at the 13th CREST Worshop. See more details at: http://crest.cs.ucl.ac.uk/cow/13/

1. 13th CREST Open Workshop 12th-13th of May 2011, London Search-Based Software Engineering for Model-Based Testing Event-B in a Nutshell Test Generation Approaches SBT Challenges Test Data Generation Finite Model Learning Conclusions Alin Stefanescu - University of Pitesti, Romania *) joint work with colleagues from DEPLOY project

2. [Event-B in a Nutshell]

4. Inventor of the Z and B formal methods.

5. Z – developed in the 70s

6. B – developed in the 90s, successfully deployed in industry

7. Event-B – born with the 21st century

8. Evolution of B for system level specification

9. Developement supported by French and European projects: FP6 RODIN and FP7 DEPLOY

11. Bosch, Siemens, SAP, SSF

12. 3 industrial service providers

13. Systerel, ClearSy, Cetic

14. 7 academic partners

15. Newcastle, Aabo, Düsseldorf, ETH Zurich,

16. Southampton, Pitesti, Bucharesthttp://www.deploy‐project.eu

18. Theorem proving as core technology

19. Many other Rodin Plug-ins

20. ProB:animation, consistency and model-checking

21. Animators (AnimB)

22. Decomposition

23. Modularisation

24. Team-work

25. Code generation

26. UML-B

28. set theory as mathematical language

29. refinement as basic modeling approach

30. Contexts

31. carrier sets (domains)

32. constants

33. axioms

34. Machines

35. global variables

36. invariants

37. events that update the variables

38. Events

39. local parameters

40. guards

41. actionsITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876},{it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098}) items : Powerset(ITEMS)

42. [Test Generation Approaches]

43. Test generation based on Event-B Model-Based Testing (MBT) Model-based testing (MBT) is a newlyintroduced topic in DEPLOY priority topic for industrial partners like SAP challenges due to the sheer size of the state space of real-life scenarios Weinvestigate search-based testing (SBT) techniques for Event-B.

44. Future MBT plugin in RODIN Event-B model MBT Plug-in MBT Users { Model-checking } { Constraint-based } Extra test information { Search-based } Test cases University of Pitesti and University of Dusseldorf Tool developers:

45. Test generation from Event-B SBT Opportunity! 1. Generate a set of tests (sequence of events with concrete param.) 2. Optimize test suite (according to some criteria) – if still needed Event-B model ev2(..),ev5(..),...ev3() ev4(..),ev2(..),...ev4(..) ... ... ... ... ... ... ... ... ev3(..),ev7(..),...ev5(..) ... ... ... ... ... ev6(..),ev5(..),...ev8 Global variables: var1, var2, var3, ... Events ev1(p11,...), ev2(p21,...), ev3(p31,...), ... ev2(..),ev5(..),...ev3() ev4(..),ev2(..),...ev4(..) ... ev3(..),ev7(..),...ev5(..)

46. What is the explicit state space State Space of the Event-B model (3,4,{a,b},...) Abstract machine Event-B model ev3(5) Global variables: var1, var2, var3, ... Events ev1(p11,...), ev2(p21,...), ev3(p31,...), ... ... ... ... ... States given by the values of global variables Transitions labeled by events with concrete parameters

47. Test generation from Event-B State Space of the Event-B model (3,4,{a,b},...) Approach 1: Explore the state space using the ProB model checker state space explosion mainly due to data Try: guide the search ev3(5) ... ... ... SBT Opportunity!

48. Test generation from Event-B – part II State Space of the Event-B model (3,4,{a,b},...) Approach 2: Explore state space ignoring the data (i.e. local parameters) Problem 1: still large state space Then: construct approximations of state space up to depth K using finite automata Try 1: using machine learning and static analysis Try 2: using evolutionary algorithms? Problem 2: infeasible sequences Try 1: constraint solving for path feasibility Try 2: test data generation with metaheuristics ev3 ... ... ... SBT Opportunities!

49. [Search-Based Testing Challenges]

50. Let’s take a look at some of specific challenges for Event-B... More details in: A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. Towards Search-Based Testing for Event-B Models. To appear in Proc. of 4th International Workshop on Search-Based Software Testing (SBST), 2011.

51. No explicit state space Fact Event-B has no explicit states like the EFSMs no control state (as in EFSMs) Problem Large (possibly infinite) state space testing coverage criteria must be defined only recent work addressing SBT for EFSMs Possible ideas: coverage of all events (or a given subset of them) or coverage of all test paths of length < K many other coverages possible, so industrial guidance is needed consider the class of Event-B models with a special state variable (see industrial use cases from SAP, SSF, Bosch and UML-B models)

52. Non-numerical types Fact Event-B is based on set theory set relations, powersets, functions, set comprehensions, products, records, etc. Complex structured data (e.g. business domain) Problem fitness functions in literature mostly defined for numerical types Possible solutions design new fitness functions for set-based (non-numerical) types efficient encoding of mixed non-numerical/numerical test data

53. Hierarchical models Fact Event-B supports different types of hierarchy refinement from abstract to concrete levels model decomposition modularity most industrial models use some sort of hierarchy (due to size) Problem no much previous work on SBT addressing hierarchical models Possible ideas: adapt existing work on test selection for hierarchical state machines use the existing ProB model checker that can partially deal with hierarchy

54. Non-determinism Fact Event-B has different types of non-determinism :| or :∈ operators (e.g. x :∈ {item1, ..., item20}) non-deterministic choice of the event to be executed when several enabled non-deterministic choice of parameters (ANY construct) non-deterministic initialisation of variables satisfying the set of invariants Problem no much previous work on SBT addressing non-deterministism Possible ideas: devise fitness functions that improve the chance of choosing a given path in a non-deterministic model (under certain assumptions) make the non-determinism visible (model instrumentation)

55. [Test Data Generation]

56. Generating test data for a path Problem Given one path of events, provide the test data (event parameters) that enables the execution of the path. Approach genetic algorithms encoding of sets into binary genes mixed choromosomes (numerical and binary genes) More details in: I. Dinca, A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. Test Data Generation for Event-B Models using Genetic Algorithms. In Proc. of 2nd International Conference on Software Engineering and Computer Systems (ICSECS'11). CCIS Series, vol. 181, pp. 76-90, Springer, 2011.

57. Test data generation with genetic algorithms Event-B model Encoding of variables … 0 1 1 0 1 0 0 1 1 0 1 0 1 0 1 0 0 1 0 1 1 0 0 1 1 1 1 0 1 items ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876},{it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098}) items : Powerset(ITEMS) “Chromosome” Crossover 0 Selection Mutation Fitness evaluation End? Fitness functions Simulator (ProB)

58. Fitness functions for one path fitness := approach level + normalized branch level New objective functions for set types Classical Tracey’s objective for numerical types

59. Examples from the benchmark

61. Random Testing (RT)

62. on 18 paths covering 5 Event-B models

64. [Finite Model Learning]

65. Generating finite models from Event-B Problem There is no explicit state space of an Event-B model Approach Finite automata learning (adapted L* algorithm ) Aproximation through cover automata K-bound on the length of executions Use finite automata for conformance test generation More details in: F. Ipate, I. Dinca, A. Stefanescu: Model Learning and Test Generation for Event-B using Cover Automata. Submitted to SEFM’11.

66. First experiments SBT Opportunity?! Preliminary approach Approximation through cover automata for bound l Incremental -> fits very well with model refinements Minimal finite automata Sometimes difficult to find counterexamples (to the approximation) Scales for medium size models: Boundl ev3 ... ... ...

67. [Conclusions]

69. Test data generation for one path with search-based algorithms

70. Construct finite models with evolutionary algorithms

71. Combine ProB model-checker with meta-heuristics

72. Combine ProB constraint-solver with meta-heuristics

73. Experiment with different search algorithms (PSO, ACO, SA,...)To be answered until end of project (April 2012): Which of the above work good in practice?