Study and development of methods and tools for testing, validation and verification of multi-agent systems

Study and development of methods and
tools for testing, validation and
verification of multi-agent systems
University of Murcia
Department of Information and
Communication Engineering
Author:
Emilio Serrano Fernández
Supervisors:
Juan Antonio Botía Blaya
José Manuel Cadenas Figueredo

Outline
1
2
3
4
5
2
Motivation
Quantitative analysis for general MASs
Qualitative analysis for MASs with protocols
semantically annotated
Analysis for MABSs used as model of AmI
systems
Conclusions and future lines

Outline
1
2
3
4
5
3
Motivation
systems

● Software is used for more life-critical applications every year.
● Bugs cost the U.S. economy an estimated 59.5$ billion
annually.
● Bugs must be found by testing, verification, validation and
debugging.
● The most common debugging strategy is by brute force.
● Multi-agent systems (MASs): kind of distributed system
● Interaction among agents is performed by complex
dialogues.
● Debugging a MAS in which intelligent or emergent
behaviors may appear is much more complex.
● This thesis deals with the analysis of interactions in
large-scale MASs.
Motivation
4
Motivation | Quantitative analysis | Qualitative analysis | Analysis for MABS | Conclusions

● Categorized in the forensic analysis field
● Providing a series of simple displays
● Defining and testing protocols automatically
● Discovering knowledge in the event log
● Shortcomings which motivate this thesis and goals
● Order of the logged events
● Abstract of the presented information
● Detection of undesired emergent behaviors
● Usability
● Analyzing semantics
Three main MAS analysis approaches
5

● General large-scale FIPA MASs
● Very few elements are assumed.
● MASs with protocols semantically annotated.
● Annotations allows to study semantics.
● Multi-agent based simulations (MABS) which
model Ambient Intelligence (AmI) applications.
● Why such a concrete system?
Types of MAS studied
6

Outline
1
2
3
4
5
7
Motivation
systems

Abstract of the thesis
8

● General approach to provide forensic analysis of runs
● Capturing, ordering, and representation of messages
● Capturing: Aspect oriented programming (AOP) is employed.
● Relational data base (RDB) stores the messages.
● Transparent to developer
● Ordering: The global state is distributed and that a common
time base does not exist
● Vector clocks: an array of n integers in a MAS with n
agents
● m1_ v1 happened before m2_ v2 if v1 < v2
● Management required → AOP
Infrastructure for Forensic Analysis of Multi-Agent Systems
9

● They help to find typical bugs:
● Uninitialized agent, failure to
send, wrong recipient, message
sent multiple times, and wrong
message sent.
● Sequence diagrams are very popular
● There is an information loss.
● The use of order graphs is proposed
● Concurrent events are clearly
shown.
● Acyclic directed graph OG(V,E),
where V are the selected
messages and E the transitivity
reduction of the less-strict relation
between vector clocks.
Studying messages via simple displays
10

Order graph for case study, fire example
11

● When the number of messages and agents grows, the utility of
the visual representation decreases.
● Abstract graphs and collaboration graphs
Summarizing the order graphs
12

● How to detect undesired emergent behaviors?
● Database M with messages → Data mining
● Fields: sender, receivers, performative and
content.
● Patterns can be detected
● Association rules and Clustering
Intelligent Data Analysis over the Database for Forensic Analysis
13

● Association rules: technique to discover relations
between variables in large databases
● Example:
at1=v1 and at2=v2 then at3=v3 and at4=v4 (0.5)
● Relations between fields in the messages
● Goals: bugs detection and automatic testing
● MAS typically has a non-deterministic behavior
Discovering relationships among attributes by association rules
14
PERFORMATIVE=CFP ==> SENDER=client conf:(1)
PERFORMATIVE=ACCEPTPROPOSAL ==> SENDER=client conf:(1)
PERFORMATIVE=REJECTPROPOSAL ==> SENDER=client conf:(1)
PERFORMATIVE=PROPOSE ==> RECEIVER=client conf:(1)
PERFORMATIVE=INFORM ==> RECEIVER=client conf:(1)
...

● Discovering groups of agents which behave similarly
● Similarity → agents which communicate with same agents
● ROCK algorithm is applied → categorical data
● M in the form of messages is not valid anymore
● Two agents are neighbors if they communicate with the same agent
● Clusters → agents which share a high number of neighbors.
Discovering agents behaving similarly by the ROCK clustering algorithm
15

● Discovering agents which collaborate
● Collaboration → large interchange of messages
● The k-means algorithm is applied → vectors of real numbers
● Vectors must reflect the communication activity
● Groups are formed with agents which had a high communication activity
among them
Discovering agents interacting together by the k-means clustering algorithm
16

Collapsed similarity and collaboration graphs for a case study, cinema example
17

● Clustering approach presents some limitations
● Intra and inter cluster visual information
● Information about the most important elements
● Amount of knowledge required by the user
● A solution is the use of techniques based on social networks
analysis (SNA)
● Collaboration graph → social network
● Pathfinder algorithm generates pathfinder networks (PFNETs)
● They allow to reveal the underlying organization of a system
● P.e. psychology and scientometrics
● It keeps only those links whose weights do not violate the
triangle inequality
Analysis by Pathfinder Networks, Improving the Clustering Approach
18

● A PFNET(r, q) is a septuple (N, E, W, LLR, LMR, r, q)
● Only (A, E, W) is considered
● Similarity PFNET
● W function bounded by (0,1]
● 1 if agents interact with same agents using same
number of messages
● Collaboration PFNET
● W function bounded by [0,1)
● 0 if agents do not interact at all
Obtaining Pathfinder Networks for agents
19

Collaboration PFNET for a case study, cinema example
20

Implementation, ACLAnalyser
21
Presentation: videosACLAnalyser.mp4
Project website: aclanalyser.sourceforge.net/

● Quantitative analysis for general large-scale MASs
● General infrastructure for forensic analysis
● AOP, RDB, vector clocks
● Simple displays for MASs runs.
● Order graphs, sequences diagrams, abstract graphs,
collaboration graphs
● Data mining
● Apriori, ROCKS, K-means
● Social network analysis
● Pathfinder → PFNETs (similarity and collaboration)
● Implementation
● ACLAnalyser
Contributions of this part
22

Outline
1
2
3
4
5
23
Motivation
systems

24

● Qualitative approach
● The target MAS cannot be general → Requirements to capture
semantics
● Protocols specification
● Protocol results + semantics + data mining → context models
● Logical theories that capture regularities in previously observed
interactions
● Utility?
● Summary
● Predictions, infer the mental states definitions, trustworthiness
Qualitative Analysis from Interactions, Using Protocols Semantically Annotated
25

Protocols Semantically Annotated, a negotiation
26
Protocol paths + ground terms = context = tuples for data mining

● Data mining algorithms usually assume a fixed number of
attributes
● Strategies for preprocessing
● Different paths → Different messages/constraints/variables
● Different data set for each observed path
● Using all messages/constraints and “?”·
● Iterations → Several constants g1,g2...gn
● N “copies” of each variable can be kept
● Considering only the first/last ground term g1/gn
● Class
● Protocol result/Constraint result
Mining protocol executions: Dealing with paths, loops, and variables
27

Results for a case study, car negotiation
28

Implementation, Protocol miner
29

● Qualitative analysis for MASs with protocols
● Context models = data mining over paths and
semantics
● Formal framework for protocols
● Strategies for the preprocess
● Experimental results
● Implementation
30

Outline
1
2
3
4
5
31
Motivation
systems

32

● Multi-agent based simulations (MABS)
● Ideal technology for the validation of AmI applications.
● Ambient Intelligence (AmI)
● Applications are endless
● Smart homes, health monitoring and assistance, hospitals,
transportation, emergency services, education, workplaces,
etc.
● Direct validation may be impractical.
● Methodology for the development and analysis of MABSs to validate
AmI applications.
● MABSs are complex → forensic analysis and new techniques.
Forensic Analysis of MABSs as Tool to Develop AmI Systems
33

34
AmI Systems

AVA, An Agent based methodology for the Validation of AmI systems
35

Forensic analysis infrastructure for MABS
36

Results for a case study, UbikSim
37

Implementation, UbikSim
38
First release: videosUbikSimHD.mp4
Current version: videosUbikSim-office.avi
Project website: ubiksim.sourceforge.net/

●AVA methodology
●Analysis of MABSs which model AmI
systems
●Simpler simulations
●Reality injection
●Forensic analysis
●Implementation, UbikSim
39

Outline
1
2
3
4
5
40
Motivation
systems

● Contribution: advancing the state of the art in the analysis of
multi-agent systems (MASs) interactions.
● Based on a forensic analysis.
● Very few elements that can be assumed for the analysis of a
general MAS interactions.
● Database with messages.
● An infrastructure for forensic analysis has been provided.
● AOP, RDB, Vector clocks, Order graphs, Sequences
diagrams, Collaboration graphs, and Abstract graphs.
● Interaction among agents can generate unwanted emergent
behaviors.
● Learning techniques can be applied: association rules and
clustering.
Conclusions I
41

● Human intervention is required to assess the results of data mining
● Pathfinder networks (PFNETs)
● Clues about unwanted behaviors which must be confirmed
● Restrictions on the MAS design allows a more a powerful analysis
● Protocols semantically annotated allow to study not only the
“envelope”
● Definition, creation, and analysis of Multi-agent based simulations
(MABSs) used as model to validate Ambient Intelligence (AmI)
systems
● Testing AmI complex and costly → MABS are also complex
systems → forensic analysis methods and tools
● We have to keep working
Conclusions II
42

● Quantitative analysis
● Studying new aspects in agent societies to be represented.
● Studying alternative techniques.
● Improving the degree of automation.
● Qualitative analysis
● Using more real-world examples.
● Studying the performance for different strategies.
● Employing more advanced machine learning methods.
● Analysis for MABS
● Automatic construction and verification of the environment
model.
● Using learning agents.
● More cases study.
Future lines
43

Journals
• Validating Ambient Intelligence based Ubiquitous Computing Systems by Means of
Artificial Societies. Emilio Serrano and Juan Botía. Information Sciences. In press
(Impact Factor 2010, 2.833, Q1)
• Debugging complex software systems by means of pathfinder networks. Emilio Serrano,
Arnaud Quirin, Juan Botia and Oscar Cordón. Information Sciences 180 (5) (2010) 561--
583. (Impact Factor 2009, 3.291, Q1) .
• Ubik: a multi-agent based simulator for ubiquitous computing applications. Emilio
Serrano, Juan A. Botia, and Jose M. Cadenas. Journal of Physical Agents, 3(2), 2009.
Indexed in Scopus.
• Intelligent Data Analysis applied to Debug Complex Software Systems. Emilio Serrano,
Jorge J. Gomez-Sanz, Juan A. Botia, Juan Pavon. Neurocomputing, 72(13-15):2785 --
2795, 2009. (Impact Factor 2009, 1.440, Q2).
Relevant Publications
44

Conferences
• Mining Qualitative Context Models from Multiagent Interactions. Emilio Serrano, Michael Rovatsos and Juan Botía.
Extended Abstract in AAMAS2011.
• Human behaviours simulation in ubiquitous computing environments. Teresa Garcia-Valverde, Francisco
Campuzano, Emilio Serrano and Juan A. Botia. Workshop on Multi-Agent Systems and Simulation (MAS&S) at
MALLOW10.
• Social simulation for AmI systems engineering. Teresa Garcia-Valverde, Emilio Serrano and Juan A. Botia.
International Conference on Hybrid Artificial Intelligence Systems (HAIS'10), 2010.
• Incremental deployment of large-scale AmI system by means of social models. Juan A. Botia, Emilio Serrano, Teresa
Garcia-Valverde and Antonio Gomez-Skarmeta. Proceedings of International Workshop on ``Simulation of Complex
Social Systems'' (SiCoSSys 2009).
• Artificial societies immersed in an Ambient Intelligence Environment. Teresa Garcia-Valverde, Emilio Serrano, Juan
A. Botia, Antonio Gomez-Skarmeta and Jose M. Cadenas. Proceedings of The 1st Workshop on Social Simulation on
International Joint Conferences on Artificial Intelligence, 2009.
• Infrastructure for forensic analysis of multi-agent based simulations. Emilio Serrano, Juan A. Botía Blaya and Jose M.
Cadenas. Seventh international Workshop on Programming Multi-Agent Systems. Promas 2009
• Construction and debugging of a multi-agent based simulation to study ambient intelligence applications. Emilio
Serrano, Juan A. Botía Blaya, and Jose M. Cadenas. International Work-Conference on Artificial Neural Networks
(IWANN2009).
• Infrastructure for forensic analysis of multi-agent systems. Emilio Serrano and Juan Botia. In Programming Multi-
Agent Systems: 6th International Workshop, PROMAS 2008.
• Testing and debugging of MAS interactions with INGENIAS. Jorge J. Gómez-Sanz, Juan Botia, Emilio Serrano, and
Juan Pavón. In Agent-Oriented Software Engineering IX: 9th International Workshop, AOSE 2008
Relevant Publications II
45

• Jorge J. Gómez-Sanz and Juan Pavón
• Complutense University of Madrid
• Oscar Cordón and Arnaud Quirin
• European Centre for Soft Computing
• Michael Rovatsos
• University of Edinburgh
• Teresa Garcia-Valverde, Francisco Campuzano, and Andrés
Muñoz
• My group
• Acknowledgements: Thesis supported by the Spanish Ministry of Science and
Innovation under the grant AP2007-04080 of the FPU program, and in the scope
of the Research Projects CARONTE, DIA++ and through the Fundación Séneca
within the Program “Generación del Conocimiento Científico de Excelencia”
Collaborators and acknowledgements
46

University of Murcia
Department of Information and
Communication Engineering
Thank you very much
for your attention
"Science is forever a search, never really a finding. It
is a journey, never really an arrival"
Karl Popper
47

Study and development of methods and tools for testing, validation and verification of multi-agent systems

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (9)

Destaque

Destaque (20)

Semelhante a Study and development of methods and tools for testing, validation and verification of multi-agent systems

Semelhante a Study and development of methods and tools for testing, validation and verification of multi-agent systems (20)

Último

Último (20)

Study and development of methods and tools for testing, validation and verification of multi-agent systems