Introduction into Big Data and its current status as well as showing 1.5 approaches to apply it.

1. 1 Big Data – Is it a hype or for real? 22.12.2015
Software
Solutions for
Manufacturing
Introduction into Big Data and 1.5 applications
Dr. Dirk Ortloff
Big Data – Is it a hype
or for real?

2. 2 Big Data – Is it a hype or for real? 22.12.2015
 Company overview
 Introduction
 Status of Big Data
 Big Data ≠ Hadoop
 1.5 example applications
 camLine eTrace SaaS
 The research project PRO-OPT
 Conclusions
Agenda
Picture from: https://commons.wikimedia.org/wiki/File:BigData_2267x1146_white.png

3. 3 Big Data – Is it a hype or for real? 22.12.2015
Company overview

4. 4 Big Data – Is it a hype or for real? 22.12.2015
Service Offering Portfolio

5. 5 Big Data – Is it a hype or for real? 22.12.2015
Product areas

6. 6 Big Data – Is it a hype or for real? 22.12.2015
Products for your Manufacturing Excellence
PRODUCTS
LineWorks
•WIP, MaMa,
CarMa, mDICE
•iGate
•Pulse, MaiMa
•SPACE, SQM
•Recipe
Management
•…
Cornerstone
•Explorative
Data Analysis
(EDA)
•Design of
Experiments
(DoE)
•Six Sigma
InFrame
Synapse
•SME’s MES
•Complete
tracking and
tracing
XperiDesk
•Technology
Development
Management
platform
•Virtual
manufacturing
enablement
•Electronic Lab
notebook
(ELN)
Solutions
•Manufacturing
Execution
System (MES)
•Quality
Management
•Equipment
Efficiency (OEE)

7. 7 Big Data – Is it a hype or for real? 22.12.2015
Introduction – Big Data

8. 8 Big Data – Is it a hype or for real? 22.12.2015
The first documented use of the term “big data” appeared in a 1997 paper by
scientists at NASA, describing the problem they had with visualization (i.e.
computer graphics) which “provides an interesting challenge for computer
systems: data sets are generally quite large, taxing the capacities of main memory,
local disk, and even remote disk. We call this the problem of big data. When data
sets do not fit in main memory (in core), or when they do not fit even on local
disk, the most common solution is to acquire more resources.”
Gartner: Big data is high-volume, high-velocity and/or high-variety information
assets that demand cost-effective, innovative forms of information processing that
enable enhanced insight, decision making, and process automation.
What is Big Data?

9. 9 Big Data – Is it a hype or for real? 22.12.2015
What is Big Data?
Picture from: http://www.ibmbigdatahub.com/sites/default/files/infographic_file/4-Vs-of-big-data.jpg

10. 10 Big Data – Is it a hype or for real? 22.12.2015
Data: symbols
Data is raw. It simply exists and has no
significance beyond its existence (in and
of itself). It can exist in any form, usable
or not. It does not have meaning of itself.
In computer parlance, a spreadsheet
generally starts out by holding data.
Information: data that are processed to be useful; provides answers to "who", "what",
"where", and "when" questions. Information is Data that has been given meaning by
way of relational connection.
Knowledge: application of Data and Information; answers "how" questions. Knowledge
is the appropriate collection of information, such that it's intent is to be useful.
Wisdom: is the ability to increase effectiveness. Wisdom adds value, which requires the
mental function that we call judgment. The ethical and aesthetic values that this implies
are inherent to the actor and are unique and personal.
Following the DIKW model: http://www.systems-thinking.org/dikw/dikw.htm
Definitions

11. 11 Big Data – Is it a hype or for real? 22.12.2015
“Fact: 80 percent of the
digitized information in
a typical company is in
the form of unstructured
data such as documents, e-mail,
and images. “1
“Fact: The amount of unstructured
content in a typical business grows
by 50 percent every year. “1
What it boils down to:
1: Oracle: Information Management – Get control of your Information
Picture is property of: www.yakidoo.com

12. 12 Big Data – Is it a hype or for real? 22.12.2015
Status of Big Data

13. 13 Big Data – Is it a hype or for real? 22.12.2015
Is Big Data still a hype to go away?
Picture from: http://na2.www.gartner.com/imagesrv/newsroom/images/hype-cycle-pr.png;wada20fd4bd7891509

14. 14 Big Data – Is it a hype or for real? 22.12.2015
Is Big Data still a hype to go away?
Picture from: http://na1.www.gartner.com/imagesrv/newsroom/images/HC_ET_2014.jpg;wadf79d1c8397a49a2

15. 15 Big Data – Is it a hype or for real? 22.12.2015
Is Big Data still a hype to go away?
Picture from: http://na2.www.gartner.com/imagesrv/newsroom/images/emerging-tech-hc.png;wa0131df2b233dcd17

16. 16 Big Data – Is it a hype or for real? 22.12.2015
Big Data
Almost anybody does something
with Big Data…
Picture from: https://commons.wikimedia.org/wiki/File:Peak_hour_traffic_in_melbourne.jpg

17. 17 Big Data – Is it a hype or for real? 22.12.2015
But everthing is pretty company
specific and mostly isolated
applications
Picture from: https://commons.wikimedia.org/wiki/File:Palau_archipelago.jpg

18. 18 Big Data – Is it a hype or for real? 22.12.2015
Big Data ≠ Hadoop

19. 19 Big Data – Is it a hype or for real? 22.12.2015
Hadoop – the myths
Data Lake
42

20. 20 Big Data – Is it a hype or for real? 22.12.2015
“The Apache™ Hadoop® project develops open-source software for reliable,
scalable, distributed computing.
The Apache Hadoop software library is a framework that allows for the
distributed processing of large data sets across clusters of computers ….”
Hadoop itself consist only out of :
• Hadoop Common
• Hadoop Distributed File System (HDFS™)
• Hadoop YARN
• Hadoop MapReduce
Hadoop – the reality

21. 21 Big Data – Is it a hype or for real? 22.12.2015
Big Data Landscape

22. 22 Big Data – Is it a hype or for real? 22.12.2015
Big Data Landscape – Example Usage

23. 23 Big Data – Is it a hype or for real? 22.12.2015
Paradigm shift – Technically
Picture from: http://zcom.ro/wp-content/uploads/docs/pdf/3.2.-Eric-Dong-Huawei-Big-Data-How-to-Face-the-Data-Wave-in-Cloud-Era.pdf

24. 24 Big Data – Is it a hype or for real? 22.12.2015
In theoretical computer science,
the CAP theorem, also known
as Brewer’s theorem, states that
it is impossible for a distributed
computer system to
simultaneously provide all three
of the following guarantees:
• Consistency (all nodes see the
same data at the same time)
• Availability (a guarantee that
every request receives a
response about whether it
was successful or failed)
• Partition tolerance (the system
continues to operate despite
arbitrary message loss or
failure of part of the system)
Principle differences in data storage approaches
Picture from: http://www.abramsimon.com/

25. 26 Big Data – Is it a hype or for real? 22.12.2015
Important – Keep history and evolvement in mind
Picture from: http://www.digitale-technologien.de/DT/Redaktion/DE/Downloads/Publikation/smart-data-boden-introduction-flink.pdf

26. 27 Big Data – Is it a hype or for real? 22.12.2015
Paradigm shift – Culturally

27. 28 Big Data – Is it a hype or for real? 22.12.2015
What is the future business flow …???

28. 29 Big Data – Is it a hype or for real? 22.12.2015
1.5 solution approaches
camLine eTrace

29. 30 Big Data – Is it a hype or for real? 22.12.2015
Cloud eTrace SaaS Edition
ETrace
eTrace SaaS Edition
• Real time process by each upload wafer test result files
• Real time test result analysis and notification per file
• Unlimited user
• Various Analysis plugins are provided by camLine
Analysis Storage
Real Time
Eval.
Source Parser
Foundries
Test
House
Foundries

30. 31 Big Data – Is it a hype or for real? 22.12.2015
The Challenges
Reduce
integration and
maintenance
effort for tester
raw data
Online and long
term data
storage and
analysis
Reduce cost of
ownership

31. 34 Big Data – Is it a hype or for real? 22.12.2015
Search by Lot and View summary for each parameter online
Analysis – Wafer Map

32. 35 Big Data – Is it a hype or for real? 22.12.2015
Online Wafer Map View for each test result on
given lot
Ease pick list of
tested result set
for given wafer
Drill down

33. 36 Big Data – Is it a hype or for real? 22.12.2015
SPACE Offline Chart Plugin Viewer - DMP
Full Wafer Map
with interpolated
values.

34. 37 Big Data – Is it a hype or for real? 22.12.2015
Reduce cost of
ownership
eTrace Solution Values
Reduce
integration and
maintenance
effort for tester
raw data
Online and long
term data
storage and
analysis
Saving in effort in integration and
test programing by engineers
Potential of 30% reduction
for data storage capacity
Reduce customer
resources to maintain
the system

35. 38 Big Data – Is it a hype or for real? 22.12.2015
1.5 solution approaches
Overview research project PRO_OPT

36. 39 Big Data – Is it a hype or for real? 22.12.2015
The PRO-OPT Research Project
Project start 1.1.2015,
3 years of funding,
www.pro-opt.org

37. 40 Big Data – Is it a hype or for real? 22.12.2015
The PRO-OPT Mission and Research Group
Evaluation Partner and Data Supplier
for Automotive Diagnostics
Technology and Evaluation Partner for
Production Systems
Research Partner for Data Mining and
Integration of System Components
Project Lead, Technology and
Evaluation Partner for Automotive
Diagnostics
Technical Project Lead,
Research Partner for
Access Restriction, SW Architecture,
Simulation, Data Generation
Multiplicator
Data Supplier and
Evaluation Partner
PRO-OPT aims at identifying valuable data and making it available for
creating additional benefit for all members of a Smart Ecosystem.
Associated Partners

38. 41 Big Data – Is it a hype or for real? 22.12.2015
Product, Delivery Notes, …
Motivation
Module Supplier OEM Production OEM After-Sales
Electronic
Component Supplier
OEM Engineering
Design and Engineering Data, Configuration Data, …
Is there more
information available
here?
Is there even more
Information
we ignore?
Why is there so little
feedback here?

39. 42 Big Data – Is it a hype or for real? 22.12.2015
Is this how we Look at the Diagnostic Data of a
Car?
3rd-Party After-Sales,
Automobile
Associations

40. 43 Big Data – Is it a hype or for real? 22.12.2015
Shouldn‘t it be like this?

41. 44 Big Data – Is it a hype or for real? 22.12.2015
The PRO-OPT Mission
• Profitable exchange of data for all involved parties
• Protection of „private“ data and intellectual
property
New
Business Models
• Identify and describe available data sources
• Consider data quality and usage restriction
Modeling
• Platform for integrative big data applications
• Data analysis and visualization methods and tools
• Support data ownership and usage restriction
Platform and Methods
for Collaborative
Analytics

42. 45 Big Data – Is it a hype or for real? 22.12.2015
Benefits
Transparency improves overall
product quality
Lower warranty costs
among the supply chain
Better reputation wrt. quality
Getting ready for
growing product complexity

43. 47 Big Data – Is it a hype or for real? 22.12.2015
Big Data Landscape – Technology@PRO-OPT

44. 48 Big Data – Is it a hype or for real? 22.12.2015
camLine – Use Case Overview
camLine
Conti
camLine
SDC
WIP
EvaProd
CarMa
MaMa
…
…
camLine mDice camLine
Cornerston
e
Trace KPI
camLine
SPACE
Data Storage
Data ProcessingData Ingestion
Cluster
Zookeeper
RDBMS
Bulk
Uploader
(TBD)
store read
store
store readstore read
Spark
Streamin
g
MLLib GraphX SparkR
Blink
DB
Spark
SQL
Cluster
ETL
UC9
UC1
UC2+3UC4
UC5
UC5+6UC5+6UC5+7UC8

45. 49 Big Data – Is it a hype or for real? 22.12.2015
Conclusions

46. 50 Big Data – Is it a hype or for real? 22.12.2015
• Big Data is here to stay – Gartner recons it is not hype anymore it is in the
market
• Big data ≠ Hadoop
• Big data is not a technical – it is also a cultural change
• New "Ideology": Collect first, Collect everything, think later what to do with it...
storage is cheap, lost data is expensive
• Example applications already exist – there is a longer way to the data lake
Conculsions

47. 51 Big Data – Is it a hype or for real? 22.12.2015
Thank you for your attention