Kalman Graffi - IEEE HPCS 2013 - Comparative Evaluation of P2P Systems Using PeerfactSim.KOM
1. Comparative Evaluation of Peer-to-Peer
Systems Using PeerfactSim.KOM
Kalman Graffi
Technology of Social Networks
Institute of Computer Science
Heinrich Heine University in Düsseldorf
eMail: graffi@cs.uni-duesseldorf.de
Matthias Feldotto
Theory of Distributed Systems
Institute of Computer Science
University of Paderborn
eMail: feldi@mail.upb.de
See: www.peerfact.org
2. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 2
Agenda
Background on Evaluation Approaches
Requirements on Simulating P2P Systems
PeerfactSim.KOM
Comparative Evaluation in PeerfactSim.KOM
Conclusions
3. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 3
Analysis
– Often simplified model
• Homogeneous nodes, probabilistic actions
– Leads to proofs (under specific conditions)
Example:
– Given: Weighted DAG
– Results:
• Proofs
• Complexities - O (log N)
Good: General results
Weakness:
– Details lost in abstraction
– Sometimes the constants are important
General Evaluation Methods
4. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 4
Simulation
– Advanced and heterogeneous model
• Specific node characteristics, capacities, behavior
– Investigates emerging behavior
– Often focus on quality of service
• Response times, induced traffic, specific node load
• A response time of 1s to 5s matters!
Examples:
Given:
– 10.000 nodes, capacity distribution X
– 70% altruistic nodes, 20% selfish nodes, 10% malicious nodes
– Protocol XY, workload Z
Results:
– Statistics on quality of service over time
General Evaluation Methods
5. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 5
Prototype – in Testbed / in real world
– Deployment of code in real testbed (e.g. PlanetLab)
– Most adequate models, unpredictable user behaviour
– Challenging to coordinate the tests, gather results
– Logging and coordination might disturb the results
Example
Given:
– 733 PlanetLab nodes
– Full protocol stack: IP, TCP/UDP, middleware, application, virtual
users
– Deployment in global PlanetLab
Results:
– Behavior under realistic network conditions
• Delays, jitter, node load …
General Evaluation Methods
6. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 6
Agenda
Background on Evaluation Approaches
PeerfactSim.KOM
Requirements on Simulating P2P Systems
Comparative Evaluation in PeerfactSim.KOM
Conclusions
7. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 7
PeerfactSim.KOM (see www.peerfact.org)
Type
– Event-based simulator
– Written in Java
– Simulations up to 100K peers possible
– Focus on simulation of p2p systems on various layers
• User
• Application
• Services: monitoring, replication …
• Overlays
• Network models
Invitation to join the community
– Several universities use and extend the simulator actively
– Used and heavily extended in the project
8. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 8
Layered View
Layered Architecture
– Easy exchange of components
– Testing of new applications
– Testing of new mechanisms
Main idea
– Layers have several implementations
– Enables testing of individual layer
mechanisms
• on its own
• in combination with other layers
See www.peerfact.org
Application
Overlay
User
SimulationEngine
Network
Service
Transport
Application
Overlay
User
SimulationEngine
Network
Service
Transport
10. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 10
Analyzers and Final Graph Generation
CSV filesCSV files
GraphsGraphs
gnuplot
11. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 11
Agenda
Background on Evaluation Approaches
PeerfactSim.KOM
Requirements on Simulating P2P Systems
Comparative Evaluation in PeerfactSim.KOM
Conclusions
12. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 12
Requirements on Simulating P2P Systems
For what do we want to use simulators?
– Evaluation one or several protocol through
– Parameter studies with multiple seeds
What we want: specify once – collect all graphs at the end
– Specify all setups in the begin
• E.g. overlays=pastry,chord,kademlia network=simple,gnp seed=10
• Resulting in 60 simulations: 10 simulations each of pastry-simple,
pastry-gnp, chord-simple, chord-gnp, kademlia-simple, kademlia-gnp
– Automated execution
• Parallelity configurable, individual maximum memory
– Automated results processing and plotting
• Combine all results of a setup – calculate means and std.deviations
• Plot results of all setups in graphs
13. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 13
Motivation and Essence of the Paper in Summary
Before
Simulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot results
Combine all individual
simulation results
Combine all individual
simulation results
Check how well your
protocol performed
Check how well your
protocol performed
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Combine all individual
simulation results
Combine all individual
simulation resultsCombine all individual
simulation results
Combine all individual
simulation resultsCombine all individual
simulation results
Combine all individual
simulation results
14. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 14
Motivation and Essence of the paper in Summary
Now
Simulation
setup
Simulation
setup
Start
simulation
Start
simulation
Check how well your
protocol performed
Check how well your
protocol performed
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Combination of all
simulation results
Combination of all
simulation results
Automated plottingAutomated plotting
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
15. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 15
Agenda
Background on Evaluation Approaches
PeerfactSim.KOM
Requirements on Simulating P2P Systems
Comparative Evaluation in PeerfactSim.KOM
Conclusions
16. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 16
Approach
1. Common interfaces
2. General analyzers
3. Configurations and automated result compilation
17. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 17
Analyzers
Collecting
– Events are monitored and reported to analyzers
– Analyzers maintain overview on metrics
Hierarchy of output on
three different layers:
– Comparative evaluation (for
all parameter combinations)
– Multiple-seed evaluation
for one parameter setup
– Single evaluation of one
setup with one seed
Output style:
– ordered by the time
– grouped by the single peers
– grouped by the single peers
and sorted by their values
18. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 18
Analyzers by Layers
DefaultAggregationAnalyzer: query events in aggregation nodes
DefaultDHTOverlayAnalyzer: lookup and store events in a DHT overlay
DefaultKBROverlayAnalyzer: query and routing events in a KBR overlay
DefaultUnstructuredAnalyzer: connections and queries in an unstructured
overlay
DefaultTransAnalyzer: sent and received messages in the transport layer
DefaultNetAnalyzer: sent and received messages in the network layer
DefaultChurnAnalyzer: session times and inter-session times
DefaultConnectivityAnalyzer: on- and offline events of nodes
DefaultOperationAnalyzer: operations in the simulator
DefaultMemoryUsageAnalyzer: memory used by the simulator
DefaultTimeUsageAnalyzer: time used by the simulator
19. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 19
Sample Evaluations: All Overlays
Setup: 1000 nodes
Overlays
– Unstructured: Gnutella06 and Gia
– Structured: Chord, EpiChord, ReChord, CaReChord, Pastry and
CAN
Seeds: 20
Actions:
– 120 min joining phase
– 60 min publishing phase
– 120 min lookup phase
– Publish and lookup files from a file set based on a Zipf distribution
Network: Simple network, no churn
Interest: Successful lookups/search, bandwidth consumption
22. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 22
Sample Evaluations: Structured Overlays
Setup: 1000 nodes
Overlays
– Structured: Chord, EpiChord, ReChord, CaReChord, Pastry and CAN
Seeds: 20
Actions:
– 120 min joining phase
– 180 min lookup phase (1 lookup every 30-120 seconds)
– Publish and lookup files from a file set based on a Zipf distribution
Network: Global Network Positioning, no churn
Interest: Hop count, traffic / bandwidth usage
27. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 27
Sample Evaluations: Chord Variants
Setup: 1000 nodes
Overlays
– Structured: Chord, EpiChord, ReChord, CaReChord
Seeds: 20
Actions:
– 120 min joining phase
– 180 min lookup phase (1 lookup every 30-120 seconds)
– Publish and lookup files from a file set based on a Zipf distribution
Network:
– Modular network layer: measurement-based delay, packet loss,
jitter and bandwidth.
– Churn: based on measurements in KAD by Steiner et al.
Interest: Hop count, traffic / bandwidth usage
32. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 32
Conclusions
Simulation
– Common approach to evaluate large scale (p2p) networks
Requirements
– Setup / configure once, collect all results at the end
– Comparative evaluation of all parameter combinations
Simulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot resultsSimulation
setup
Simulation
setup
Start
simulation
Start
simulation
Compile and
plot results
Compile and
plot results
Combine all individual
simulation results
Combine all individual
simulation results
Check how well your
protocol performed
Check how well your
protocol performed
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Simulation
setup
Simulation
setup
Start
simulation
Start
simulation
Check how well your
protocol performed
Check how well your
protocol performed
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Combination of all
simulation results
Combination of all
simulation results
Automated plottingAutomated plotting
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
Sim.Sim.
NowNow
33. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 33
Join PeerfactSim.KOM (see www.peerfact.org)
Type
– Event-based simulator
– Written in Java
– Simulations up to 100K peers possible
– Focus on simulation of p2p systems on various layers
• User
• Application
• Services: monitoring, replication …
• Overlays
• Network models
Invitation to join the community
– Several universities use and extend the simulator actively
– Used and heavily extended in the project
34. Jun.-Prof. Dr.-Ing. Kalman Graffi 15. Juli 2013 34
Thank You for Your Attention
Jun.-Prof. Dr.-Ing. Kalman Graffi
Technology of Social Networks Group
Institute of Computer Science
Heinrich-Heine-Universität Düsseldorf
eMail: graffi@cs.uni-duesseldorf.de
Web: www.p2pframework.com
Web: www.peerfact.org
??