DATA SCIENCE Lesson 3 Data Architectures Data Processing Modeling -Algorithm - Method conference Jean-Antoine Moreau

LLeeccttuurree 33
DDaattaa AArrcchhiitteeccttuurreess SSyynntthheessiiss
DDaattaa PPrroocceessssiinngg
DDaattaa MMiinniinngg
MMooddeelliinngg --AAllggoorriitthhmm –– MMeetthhoodd
 IInntteeggrraattiioonn iinnttoo tthhee IInnffoorrmmaattiioonn SSyysstteemm
Conference - Study : Jean-Antoine Moreau (Engineer - Lecturer)
© Jean-Antoine Moreau
This lesson, copying and reproduction prohibited
Managing my copyright: ADAGP.

DDAATTAA SSCCIIEENNCCEE
LLeessssoonn 33
Conference - Study : Jean-Antoine Moreau (Engineer - Lecturer)

JAM 3 / 175
Contact http://www.jean-antoine-moreau.fr.nf
« Tell me and I forget. Teach me and I
remember. Involve me and I learn. »
Benjamin Franklin

JAM 4 / 175
DDAATTAA SSCCIIEENNCCEE -- LL33
• Volumes of data are:
– more and more important;
– distributed;
– interconnected;
– heterogeneous;
– mobile.

JAM 5 / 175
• Data volumes are more difficult to:
– integrate
– analyze in real-time

JAM 6 / 175
These new requirements
these new conditions
involve
new data models
new methods of modeling

JAM 7 / 175
new data models
new methods of modeling
information systems architectures
Large-Scale Distributed Data Intensive Information Systems.

JAM 8 / 175
Simple modeling:

JAM 9 / 175
Layers’s modeling

JAM 10 /
Massively parallel architectures

JAM 11 /
diagram

JAM 12 /
data streaming
streaming bulk data

JAM 13 /
Basic diagram :

JAM 14 /
• Changeover / transition of DBMS
(DATABASE MANAGEMENT SYSTEM) to
NoSQL or Hybrid DBMS.

JAM 15 /
summary table

JAM 16 /
Data Base use

JAM 17 /
• New data models:
– tables;
– key values;
– documents;
– graphs;
– streams;
• multiple sizes;
• size (volume) changing in time;
• different versions.

JAM 18 /
Modeling includes :
– in-depth of data;
– the position in time;
– release management.

JAM 19 /
Modeling includes :
– Real time applied to each object;
• each attribute;
• each relationship between objects;
– Simple, composition, coupling, cardinality;
• each relationship between process.

JAM 20 /
Modeling of data distributed
multi-site architectures
and cloud

JAM 21 /
• Massively parallel architectures;
• Massively parallel processing;
• Massively parallel computing.

JAM 22 /
• Online Storage
• Database Machine
MapReduce
Oracle
…

JAM 23 /
• Using
– In-Memory DataBase (IMDB);
– Memory DataBase System (MMDB);
– Memory Resident DataBase :
• database management system that primarily relies
on main memory for computer data storage.

JAM 24 /
Using the In-Memory
New form of modelisation

JAM 25 /
Example of modelisation:

JAM 26 /
SSttrreeaammiinngg ddaattaa
• How to analyze data streams in real time?
• How to analyze process flows in real time?
• How to use and cross these streams with the
information system data?

JAM 27 /
• Data evolution :
– Open data;
– Interconnection of data:
• social networks;
• linked open data;
– Big Data:
• volume;
• variety;
• velocity;
• truthfulness;
• validity.

JAM 28 /
• Evolution data:
– Internet:
• Data distribution;
• Heterogeneity of the data;
• Data mobility.
– Flows:
• Continuous data stream;
• Complex events.

JAM 29 /
in-depth data
time
Location/ localization
Data Stream
Version xX
Data Stream
Version xY
Data Stream
Version xZ

JAM 30 /
Computer Applications
The issues
• IT operations in real-time from distributed data;
• Crossing and visualization of public and private data;
• Complex Big Data analysis.

JAM 31 /
Big Data
Data Analytic

JAM 32 /
Transactions distributed over the network
data stream processing in real time
Complex Event Processing (CEP)

JAM 33 /
Technology
• Flash memory;
• HDD (Hard Disk Drive) - SSD (Solid-State Drive);
• Multi-core processing;
• Combination CPU (Central Processing Unit) / GPU (Graphics
Processing Unit);
• Prioritization of parallel computing;
• Server virtualization (VM virtual Machines);
• …

JAM 34 /
Technology
• Virtualization of disk storage:
– NAS (Network Attached Storage)
– SAN (Storage Area Network);
• New architectures DBMS (database management system).

JAM 35 /
SAN
Storage Area Network
with a Cluster

JAM 36 /
Databases
• Adaptation:
– patterns;
– requests;
– transactions;
– views;
– integrity constraints;
– triggers;
– managing indexes;
– changing rules of consistency;
– traceability rules.

JAM 37 /
Databases
ACID database models
ACID :
– AAtomicity;
– CConsistency;
– IIsolation;
– DDurability.

JAM 38 /
Data Modeling in a Hybrid NoSQL / Relational

JAM 39 /
Data Model
Data Modeling

JAM 40 /
Data models
Relational :
– value domain;
– link;
– algebra and calculus;
– concept of zero.
SQL 2:
– the data type;
– insulation levels;
– the portablité.
SQL 3:
– Nested tables;
– Complex types;
– Extensions (expand) objects.

JAM 41 /
New models:
– key values;
– tables;
– documents;
– graph;
– stream.

JAM 42 /
Access to the database
• The object-relational impedance mismatch.
• API (Application Programmable Interface):
– Ado (Microsoft ActiveX Data Objects);
– JDBC (Java DataBase Connectivity);
– SQLJ (program can call the JDBC );
– JPA (Java Persistence API);
– JDD (Java Decision Diagrams);
• Using Language Integrated Query (example : Microsoft .NET
Framework ).

JAM 43 /
Data Analysis
• Decision-making system and OLAP
– multidimensional databases;
• OLAP system (On-Line Analytical Processing);
• OLTP (Online transaction processing) - class of software programs capable of
supporting transaction-oriented applications;
• TPC (TPC-H / TPC-E)
– Transaction Processing Performance Council;
• Simulates a Decision Support System or Business Intelligence database environment.
– Multi-dimensional data.

JAM 44 /
Data Analysis
• Data cubes:
– array of data;
– multidimensional matrix.

JAM 45 /
array of data multidimensional matrix

JAM 46 /
Business Intelligence
Data Mining
Extraction of knowledge from the data.

JAM 47 /
ddiissttrriibbuutteedd aarrcchhiitteeccttuurreess

JAM 48 /
• multi-tier architectures:
– 2-Tiered architecture;
– architecture technology to 3-tier architecture;
– N-tier architecture.

JAM 49 /
2 Tiered :

JAM 50 /
Three-Tier Architecture

JAM 51 /
Three-Tier Architecture

JAM 52 /
« We know what we are, but know not
what we may be. »
William Shakespeare
A short break,
You can ask your questions.

JAM 53 /
Architectures
• RIA: Rich Internet application;
• RDA: Rich Desktop Application.

JAM 54 /
RIA: Rich Internet application

JAM 55 /
client network server
thick
thin
rich
application

JAM 56 /
• Using :
– HTML 5 (HyperText Mark-Up Language);
– CSS 3 (Cascading Style Sheets);
– Java Script;
– Ajax (Asynchronous JavaScript and XML).

JAM 57 /
Coupling Server / Application
Data server

JAM 58 /
Data Protection :
– encryption;
– discretionary access control;
– control of multi-level access.

JAM 59 /
Service-Oriented Architecture
• Web services
method of communication between two electronic devices over a network;
• WSDL
Web Services Description Language;
• UDDI
Universal Description, Discovery and Integration;
• SOAP
Simple Object Access protocol;

JAM 60 /
• SOA (Service-Oriented Architecture), deployment of distributed
services on ESB (Enterprise Service Bus);
• EAI functions (Enterprise Application Integration – Workflow Process).
• WOA (Web Oriented Architecture)
– use technologies :
• POX (Plain Old XML);
• REST (Representational State Transfer) .

JAM 61 /
• Massively distributed architectures;
• Massively distributed systems;
• Massively distributed computing.

JAM 62 /
• Cluster :
– Parallelism;
• Grid :
– Virtual Infrastructure;
– Parallelism and Distribution;
• P2P :
– Distribution;

JAM 63 /
Diagram
Architecture Master/Slaves

JAM 64 /
• and the cclloouudd:
 parallelism of computer centers distributed across the
planet
 GGlloobbaalliizziinngg tthhee tteecchhnniiccaall aarrcchhiitteeccttuurreess .

JAM 65 /
Example :
Cloud architecture
including API to connect
on a social network.

JAM 66 /
Management of the distributed data
• Integration of heterogeneous data;
• Unifying the distributed data;
• DBMS transaction (database management system) distributed.

JAM 67 /
Integration of heterogeneous data
• Federation of data sources;
• Mediator & Adapter / Architectures;
• Integration of schema.

JAM 68 /
• Data models for the integration :
– Relational model;
– Object;
– XML;
• data model that's based on an XML file
• the XML Data Model makes it easier to integrate XFDL forms with
other applications.
– XFDL : Extensible Forms Description Language is a class of the
Extensible Markup Language (XML) originally specified in World
Wide Web Consortium (W3C) ;
– key-value model.

JAM 69 /
Visualization of the integrated data :
– Dataviz (software) ;
– Dashboards;
– …

JAM 70 /
Unifying distributed data
• Query optimization processing;
• Intermediate results of integration;
• Products (some examples) :
– Google Fusion Tables (or simply Fusion Tables) is a web service provided by Google
for data management;
– IBM InfoSphere Information Server for Data Integration ;
– Oracle Data Integrator (data integration platform);
– SAP BusinessObjects (Data Integrator and Data Quality Management);
– Talend for Data Integration (Talend Open Studio for Data Integration).

JAM 71 /
Distributed DBMS
Transaction Management in Distributed Database.

JAM 72 /
• Transaction Management in Distributed Database:
– Two-Phase Commit Protocol (2PC)
• type of atomic commitment protocol (ACP);
– Fault tolerance;
– Scalability;
• Data replication:
– consistency of copies;
– propagation of updates;
– adjusting the consistency rules of the databases;
ORACLE, IBM DB2, Microsoft SQL server, SAP sybase, PostgreSQL and Postgres-XC.

JAM 73 /
• Parallel data processing :
– Massive parallelism of data;
• massively data-parallel applications on GPU clusters;
– (GPU Graphics Processing Unit);
• data-parallel ...
– Database machine;
– Also the basic techniques.

JAM 74 /
Massive parallelism of data
GGooaallss

JAM 75 /
• Goals :
– Linear Speed up;
– Linear Scale up;
– Scale out;
– Elasticity;
– Adaptability.

JAM 76 /
Shared-Memory
Multiprocessor Architectures
SMP
shared memory multiprocessor
NUMA
Non Uniform Memory Architecture

JAM 77 /
SSMMPP
shared memory multiprocessor

JAM 78 /
NNUUMMAA
Non Uniform Memory Architecture

JAM 79 /
Multiprocessor Architectures
• Cluster shared-disk;
• Cluster shared-computing;
• …

JAM 80 /
Example 1 :

JAM 81 /
Example 2:

JAM 82 /
using servers

JAM 83 /
tree – organization - cluster

JAM 84 /
• Types of data parallelism:
– Inter-application;
– Application;
– Inter-operation;
– Pipeline.

JAM 85 /
Example : Data Processing

JAM 86 /
Example : Signal Processing

JAM 87 /
Parallel data processing
basic techniques

JAM 88 /
Data storage
Partitioning and placement strategies in database machines
Partitioning - Sharing replication indexing

JAM 89 /
performances
Parallelism of applications Balance of charges Transactions

JAM 90 /
Fall-over
management
High availability
Savepoints

JAM 91 /
DDaattaabbaassee MMaacchhiinnee
Products :
• IBM DB2 pureScale;
• SQL Server 2012 Parallel Data Warehouse;
• ORACLE RAC (Real Application Clusters) et Exadata;
• SAP (sysbase);
• Teradata Aster Data;
• MySQL cluster (write-scalable, real-time, ACID-compliant transactional
database );
• HP Vertica (software-based analytics platform);
• ParAccel (Analytic DataBase and Plateform);
• EMC Greenplum (Data Computing Appliance DCA , massively parallel processing
database MPP).

JAM 92 /
MMaappRReedduuccee

JAM 93 /
MMaappRReedduuccee

JAM 94 /
MMaappRReedduuccee

JAM 95 /
MMaappRReedduuccee Map Reduce
Big Data Result

JAM 96 /
MapReduce
• Data storage;
• Architectures;
• Parallel programming;
• Integration.

JAM 97 /
MapReduce Layers - Architecture
With Api, for example using IBM API.

JAM 98 /
MapReduce
Processing
Split: the input data-set into independent chunks

JAM 99 /
Data Storage
• DDistributed FFile SSystem:
Google File System (scalable distributed file
system for large distributed data-intensive
applications );
Hadoop Distributed File System (HDFS) is a
distributed, scalable, and portable.

JAM 100 /
HDFS
layers

JAM 101 /
HDFS diagram :

JAM 102 /
HDFS diagram
Simple architecture :

JAM 103 /
HDFS
Cluster diagram

JAM 104 /
Flows on Global File System / Cluster

JAM 105 /
Master / Slave Diagram:

JAM 106 /
MMaappRReedduuccee using
• GFS (Global File System)
– shared disk file system for Linux computer
clusters;
– symmetric Cluster Architecture;
• HPFS
– High Performance File System.

JAM 107 /
Example :

JAM 108 /
Meta Data on Serveur / Cluster

JAM 109 /
Map Reduce
GFS / HPFS Architectures :
– Master node;
• The architecture of GFS follows the master-slave model.
– Distribution of chunks (fragment of information);
– Replicating chunks.

JAM 110 /
Map Reduce
IT architecture models
• Partitioning according to the input files;
• Fault tolerance;
• Balancing loads.

JAM 111 /
Map Reduce
• key–value model;
• Function
Map
Reduce

JAM 112 /
Business Model IT Architecture Model

JAM 113 /
Business IT architecture and process

JAM 114 /
Internet
Load Balencer
Vm (1) Vm (2) ….. Vm (n)
Real machine ( 1 …n)
Virtual

JAM 115 /
diagram - modeling :

JAM 116 /
Managing my copyright: ADAGP. A short break,
You can ask your questions.
« The direction in which
education starts a man
will determine his future in
life. »
Plato

JAM 117 /
Hadoop & MapReduce

JAM 118 /
Switch
Hadoop slave
servers
Hadoop master
and slave server Test server

JAM 119 /
Switch
Hadoop slave
servers
Hadoop master
and slave server Test server

Hadoop HDFS
JAM 120 /
Hadoop
Manage File Transfer
Hadoop Gateway Server
inspect process loading Loading
aggregate automate load

JAM 121 /
Hadoop BD Architecture and Layers

JAM 122 /
Cluster

JAM 123 /
Cluster
Flow

JAM 124 /
MapReduce Architecture

JAM 125 /
MapReduce

JAM 126 /
methods and algorithms

JAM 127 /

JAM 128 /
Map Reduce

JAM 129 /
PPaarraalllleell pprrooggrraammmmiinngg with MapReduce framework
Tools:
– Pig (workflow interface – Pig script);
– Hive (SQL Interface);
– …
• Software publisher:
– cloudera, datameer, greenplem HD, hurence, Intel, Talend,Vmware.

The software layers
Apache ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and
providing group services
JAM 130 /
Apache Ambari is an operational framework for provisioning, managing and monitoring Apache Hadoop clusters.

JAM 131 /
MapReduce integration into the information system
• Database Machine;
• Cloud:
– Outsourcing of the Big data processing;
• Tools :
– Google cloud (Platform, Print, BigQuery …), Amazon web services, Microsoft
HDInsight (Microsoft's Hadoop-based service), IBM InfoSphere BigInsights (built on
the Apache Hadoop software framework - for analyzing massive volumes) ,
ORACLE BigData Appliance (engineered system for acquiring, organizing, and
loading unstructured data into an Oracle Database), Cloudera Impala (open source
massively parallel processing (MPP) SQL query engine for data stored in a computer
cluster running Apache ).

JAM 132 /
Model of Information Systems
in the Cloud environment.

JAM 133 /
User Applications
Integrated
Services
Single Data Strore
Service
Information System
Internet
and mobile
and tablet
ERP
data warehouse

JAM 134 /
Exemple for data warehouse

JAM 135 /
Cloud Architecture
Layers

JAM 136 /
Cloud Architecture
Platform as a service
Infrastructure as a service

JAM 137 /
Cloud Architecture:
Access Security.

JAM 138 /
Data streaming
• SCA Streaming Computing Architecture;
• Streaming SQL;
• Processing and stream management of the
data flow.

JAM 139 /
SCA Streaming Computing Architecture
• Concepts:
– Data stream;
– Continuous queries;
– Temporary data;
• Flow modeling:
– Sequences;
– Time series;
• Parallel streaming.

JAM 140 /
SCA Streaming Computing Architecture
• Concepts:
– Data stream;
– Continuous queries;
– Temporary data;
• Flows modeling:
– Sequences;
– Time series;
• Parallel streaming.

JAM 141 /
Streaming Graph

JAM 142 /
Streaming SQL
• Filtering;
• Merger;
• Flows aggregation ;
• Join query between flows and the persistent
data of the information system.

JAM 143 /

JAM 144 /
Processing data flows
• Query stream;
• Set of queries on the stream;
• Stream mining;
• Automatic classification of flows.
• Products: Tibco streambase (push-based real-time analytics solution), ORACLE
Streams, IBM Infosphere Streams (stream computing solution to enable real-time
analytic processing of data), Microsoft streamInsight (platform for complex event
processing), Telegraph CQ, LighStream (solution for real-time messaging over HTTP
and WebSockets), Apach Samza (stream processing framework), Storm (has use cases:
realtime analytics, online machine learning, continuous computation, distributed RPC,
ETL ).

JAM 145 /
A short break, You can ask your questions.
“Better three hours too soon than a minute too late.”
William Shakespeare

JAM 146 /
NNooSSQQLL DDBBMMSS

JAM 147 /
NoSQL DBMS
• NoSQL architecture;
• NoSQL approaches;
• Integrating SQL into NoSQL.

JAM 148 /
aarrcchhiitteeccttuurree
ssooffttwwaarree llaayyeerrss
example

JAM 149 /
aarrcchhiitteeccttuurree
ssooffttwwaarree llaayyeerrss
example

JAM 150 /
NoSQL architectures
• Scalability;
• Architecture 3 Tier in the cloud (cloud computing);
• NoSQL :
– RDBMS object;
DBMS XML.

JAM 151 /
Architecture in the cloud
example

JAM 152 /
AArrcchhiitteeccttuurree iinn tthhee cclloouudd
example

JAM 153 /
• DBMS key value:
– Amazon dynamo DB (NoSQL database service that is offered by
Amazon.com), Amazon simple DB (distributed database written in
Erlang by Amazon.com), Apache Cassandra (massively scalable
open source NoSQL database), Voldemort (a distributed key-value
storage system, Data is automatically replicated over multiple
servers, Processing Data Streams at LinkedIn), LinkedIn Espresso
(horizontally scalable, indexed, timeline-consistent, document-oriented,
highly available NoSQL data store).

JAM 154 /
Using, Schema

JAM 155 /
Example, Diagram

JAM 156 /
• Large DBMS table :
– Google Big table
• a compressed, high performance, and proprietary data storage
system built on Google File System;
– Hadoop Hbase
• open source, non-relational, scalable, distributed database
modeled;
– Apache Acumulo
• high performance data storage and retrieval system with cell-level
access control.

JAM 157 /

JAM 158 /
Hadoop Hbase
Layers

JAM 159 /
Hbase in the
Hadoop Ecosystem

JAM 160 /
• Document database (managing document-oriented information):
– MongoDB;
• Agile and Scalable;
– Ubuntu Couche DB.

JAM 161 /
• Document database :
– MongoDB;
– Ubuntu Couche DB

JAM 162 /
• Graph database :
Graph databases apply graph theory to the
storage of information about the relationships between
entries;
Graph database is a database that uses graph
structures with nodes, edges, and properties to represent
and store data.

JAM 163 /
• Graph database :
– Google Pregel (designed for the Google cluster
architecture), (use for processing graph algorithms);
– Neo4j (highly scalable, robust – ACID, native graph
database, JAVA Graph Database;
– DEX (scalable graph database management system -
allows for the integration of multiple data sources).

JAM 164 /
Graph Model

JAM 165 /
Graph properties
Model

JAM 166 /
Graph roles
Model

JAM 167 /
SQL / NoSQL integration
Decrease in consistency.

JAM 168 /
Synthesis scheme

JAM 169 /
The management of the calculations distribution.

JAM 170 /
EntryProcessor
MapEntry
 send the code (computer program) to the data
rather than sending the data to the code (computer program).

JAM 171 /
EntryProcessor
MapEntry
 send the code (computer program) to the data
rather than sending the data to the code (computer program).
COHERENCE

JAM 172 /
EntryProcessor
• Decrease the amount of data passing
through the network;
• Répartion of the load on different nodes of
the cluster;
• Distribution of concurrent operations
without explicit lock;

JAM 173 /
Parallel Execution Process
• Method;
• Distributed execution on a cluster;
• Execution parallelized on the cluster;
• Each cluster node handles its local data.

JAM 174 /
« We are born weak, we need strength;
helpless, we need aid; foolish, we need
reason. All that we lack at birth, all that we
need when we come to man's estate, is the gift
of education. »
Jean-Jacques Rousseau

JAM 175 /
End of the part of this lesson published on the internet.

DATA SCIENCE Lesson 3 Data Architectures Data Processing Modeling -Algorithm - Method conference Jean-Antoine Moreau

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a DATA SCIENCE Lesson 3 Data Architectures Data Processing Modeling -Algorithm - Method conference Jean-Antoine Moreau

Semelhante a DATA SCIENCE Lesson 3 Data Architectures Data Processing Modeling -Algorithm - Method conference Jean-Antoine Moreau (20)

Mais de Jean-Antoine Moreau

Mais de Jean-Antoine Moreau (20)

Último

Último (20)

DATA SCIENCE Lesson 3 Data Architectures Data Processing Modeling -Algorithm - Method conference Jean-Antoine Moreau