NeuViz, a data processing and visualization architecture for network measurement experiments. Presented at "50° Congresso Nazionale AICA" - Fisciano (SA), 19th September 2013

1. 50° Congresso Nazionale AICA
Visualizing Internet-Measurements Data for
Research Purposes:
the NeuViz Data Visualization Tool
G. Futia*
E. Zimuel+
S. Basso*
J.C. De Martin*
*
Nexa Center for Internet & Society, Politecnico di Torino (DAUIN)
+
R&D Department, Zend Technologies Inc
Fisciano (SA), 19th September 2013
19th September 2013
Giuseppe Futia – Politecnico di Torino
1

2. Introduction
●
Our objective:
●
●
As a Center for Internet & Society we study the net neutrality with a
multidisciplinary approach
What we have:
●
Neubot (the Network neutrality bot), a free-software Internet bot that
performs active, lightweight network-performance tests:
–
–
BitTorrent (BitTorrent protocol)
–
●
Speedtest (HTTP protocol)
And other tests
What we need:
●
NeuViz (the Neubot Visualizer): a data processing and visualization architecture
for network measurement experiments to detect cases in which a protocol
seems discriminated
19th September 2013
Giuseppe Futia – Politecnico di Torino
2

3. NeuViz Architecture (I)
19th September 2013
Giuseppe Futia – Politecnico di Torino
3

4. NeuViz Architecture (I - a)
●
The Producers:
●
●
●
PULL static datasets from Neubot and other projects expressed in
different format (CSV or JSON)
PUSH (via API) the results of the experiments of network-measurement
tools just after the experiments are run
The Frontend and the Consumers:
●
●
The Web API connects the Frontend and the Backend: the default
Consumer is a Web browser that exploits the NeuViz Web interface
(developed using the D3.js JavaScript library)
Feedback mechanism: we plan to extend the Neubot Master Server to
fetch data from NeuViz to provide better suggestions to Neubot
instances
19th September 2013
Giuseppe Futia – Politecnico di Torino
4

5. NeuViz Architecture (II)
19th September 2013
Giuseppe Futia – Politecnico di Torino
5

6. NeuViz Architecture (II - a)
●
The Importer Stage:
●
●
●
Written in Python, it is composed of many modules to import data
from a specific network-measurement tool and format (also via a
Submit API)
It converts the input data into a JSON document and it adds
geolocation if needed (GeoLite Database)
The Raw Database:
●
●
It receives heterogeneous data organized in a uniform format by the
Importer Stage
Implemented using MongoDB: it is not practical to reduce all the input
data to the same schema, because each network experiment saves
different metrics
19th September 2013
Giuseppe Futia – Politecnico di Torino
6

7. NeuViz Architecture (II - b)
●
The Analysis Stage:
●
●
●
●
Written in Python, it is a collection of modules to produce aggregate
data for the visualization
For both tests it computes the median number of different metrics, for
each hour of the day, for each month, along geographical (country,
cities) and business dimensions (ISPs)
Scalability: we designed the code to exploit Map/Reduce techniques
on cloud services
The Analysis Database:
●
●
Conceptually-separated databases to store data ready to use (or to
visualize)
Implemented with MongoDB: useful for managing the results of
complex analysis that require long time to process
19th September 2013
Giuseppe Futia – Politecnico di Torino
7

8. Dimensions of Data
19th September 2013
Giuseppe Futia – Politecnico di Torino
8

9. Number of tests in Canada
19th September 2013
Giuseppe Futia – Politecnico di Torino
9

10. Median upload difference
19th September 2013
Giuseppe Futia – Politecnico di Torino
10

11. A case to be investigated...
Empirical Probability Density Function (PDF) of the difference
of the median upload speed of US and Canada in 2013
19th September 2013
Giuseppe Futia – Politecnico di Torino
11

12. Comparison of Speedtest and
BitTorrent performance
In 2013 the median upload difference between Speedtest and
BitTorrent in Canada was very often positive, while the same
difference was very often negative in the US
●
●
●
the BitTorrent protocol seems to be discriminated in Canada, but
we need to gather more information
we have, in fact, to investigate other possible causes of the
observed bias, including the implementation of both tests
an advanced Master Server could learn about similar anomalies
and ask Neubot instances to conduct detailed experiments
19th September 2013
Giuseppe Futia – Politecnico di Torino
12

13. Conclusions and Future Works
●
●
●
NeuViz proved to be a very useful tool to identify cases in which a
protocol seems discriminated, to be investigated with more specific
network tests
We plan to extend NeuViz to automatically raise warnings and to
cooperate with the Neubot Master Server to trigger more-specific
network experiments
We envision to integrate social networks to have an overview on
net neutrality discussions
19th September 2013
Giuseppe Futia – Politecnico di Torino
13

14. Thank you
Giuseppe Futia (giuseppe.futia@polito.it)
Nexa Center for Internet & Society (http://nexa.polito.it)
Dept. of Computer and Control Engineering (DAUIN)
Politecnico di Torino, Italy
The NeuViz git repository is available at:
http://github.com/neubot/neuviz/
19th September 2013
Giuseppe Futia – Politecnico di Torino
14