FGD Big Data

12/19/2018 FGD 06.11.2018 - Arsitektur Big Data - Eueung Mulyana
https://telematika.org/remark/bigdata6/#1 1/691 / 69
FGDFGD
Big Data
Perspektif Teknologi
Eueung Mulyana
https://telematika.org/remark/bigdata6
06 Nov 201806 Nov 2018 | CC BY-SA

Eueung Mulyana
Dr.-Ing.Dr.-Ing. (TUHHTUHH/DE 2006)
M.Sc.M.Sc. (Uni KarlsruheUni Karlsruhe/DE 2001)
ST.ST. (ITBITB/ID 1997)
Institut Teknologi Bandung
Institute for Innovation & Entrepreneurship Dev.Institute for Innovation & Entrepreneurship Dev.
Division Chief - Industrial & Business IncubatorIndustrial & Business Incubator
School of Electrical Engineering and InformaticsSchool of Electrical Engineering and Informatics
Head of TelematicsTelematics Laboratory

https://telematika.org/remark/bigdata6/#1 3/69
OutlineOutline
TinjauanTinjauan
ArsitekturArsitektur Umum
Studi KasusStudi Kasus Implementasi Big Data
3 / 69

Tinjauan Singkat
Big Data
4 / 69

Arsitektur Umum
13 / 69

https://telematika.org/remark/bigdata6/#1 16/69Traditional Data WarehousingData Warehousing (Ref: J. Szwabinski) 16 / 69

https://telematika.org/remark/bigdata6/#1 17/69Modernizing an Existing DW 17 / 69

https://telematika.org/remark/bigdata6/#1 19/69Lambda Architecture 19 / 69

Studi Kasus
Implementasi Big Data
20 / 69

https://telematika.org/remark/bigdata6/#1 22/691 PB1 PB Elastic Cluster & 192-Core192-Core Spark Cluster 22 / 69

https://telematika.org/remark/bigdata6/#1 23/69BigData @BL - >1.5 PB1.5 PB (Ref: I. Arief - 2017) 23 / 69

"jika dulu mengolah data faktur bisa sampai 8 bulan, kini cuma membutuhkan 2-3 menit saja"
(Ref: I. Djuniardi) 24 / 69
Hadoop
Big Data
@Ditjen Pajak

PoC (2015)
10 nodes (40 cores)
storage 10 TB
1. Data Integration 504504 Cores, 3.53.5 TB RAM,
100100 TB Storage
2. Data Platform 408408 Cores, 4.24.2 TB RAM,
465465 TB Storage
3. Enterprise DWH 112112 Cores, 22 TB RAM,
235235 TB Storage
Ref: [komite.id]

Cloudera
Big Data
1. Bank Mandiri
2. Bank Danamon
3. Telkomsel
Ref: [The Jungle of Big Data]

BigDataBigData@PoliTO
ClouderaCloudera
Initally 10 Nodes
3030 worker nodes - storage available 768768
TBTB, memory 2TB2TB (8GB per computing
core)
1818 nodes - single node maximum storage available 36 TB36 TB
1212 nodes (prev. cluster - 2013); single node maximum
storage available 10 TB10 TB
33 master nodes (prev.)
44 switch 48-port layer 2

BigDataBigData@ANAC
ClouderaCloudera

77 Nodes
11 head node
(1CPU 8 core, 3232 GB RAM, 300300 GB HDD)
33 master nodes
(1CPU 8 core, 3232 GB RAM, 22 x 8080 GB SSD)
33 worker nodes
(1CPU 8 core, 6464 GB RAM, 66 x 300300 GB HDD)

https://telematika.org/remark/bigdata6/#1 31/69Monitoring Architecture @General Electric (Ref: G. Gri ths) 31 / 69

https://telematika.org/remark/bigdata6/#1 32/69Data Pipeline - Java vs. Go (k8s) 32 / 69

https://telematika.org/remark/bigdata6/#1 33/69Kafka & Kubernetes 33 / 69

https://telematika.org/remark/bigdata6/#1 34/69Example Data Pipeline 34 / 69

Refs/Resources
35 / 69

Selected References
1. Big data analytics for policy making, Deloitte for European Union, 2016
2. M. Coates, Designing a Modern Data Warehouse + Data Lake, 2017
3. J. Szwabinski, Introduction into Big Data analytics
4. W. El Kaim, (Big-)Data Architecture (Re-)Invented
5. Digital Governance @Ditjen Pajak, iCIO Community
6. G. Gri ths, Building Resilient Data Pipelines, GE Digital
7. I. Arief, Big Data @ Bukalapak, SARCCOM & BCA Tech Talk, Oct 2017
Credits
Icons made by Eucalyp from Flaticon
Photos by Stig Ottesen & Bagus Ghufron on Unsplash
36 / 69

ENDEueung Mulyana
06 Nov 201806 Nov 2018 | CC BY-SA

Notes
38 / 69

Hadoop
45 / 69

Cases
49 / 69

DW & Data Lake
56 / 69

Kafka
64 / 69

ENDEueung Mulyana
06 Nov 201806 Nov 2018 | CC BY-SA

FGD Big Data

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a FGD Big Data

Semelhante a FGD Big Data (20)

Mais de Eueung Mulyana

Mais de Eueung Mulyana (20)

Último

Último (20)

FGD Big Data