Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (20)
Semelhante a IoT Connected Brewery
Semelhante a IoT Connected Brewery (20)
Último
Último (20)
IoT Connected Brewery
- 1. 1© Cloudera, Inc. All rights reserved. Harnessing Data within Hadoop in the Connected Brewery: Kafka, Spark Streaming, and Kudu Jason Hubbard Jason.hubbard@cloudera.com Cloudera
- 2. 2© Cloudera, Inc. All rights reserved. Internet of Things (IoT) $1.7 Trillion In Value 20% Annual Growth 30 Billion Things 250 Million Connected Vehicles Source - IDC & Gartner Estimates Internet of Things IoT Markets - 2020
- 3. 3© Cloudera, Inc. All rights reserved. IoT Will Drive An Explosion of Data… Data expected to explode to 44 ZB by 2020 Source: IDC 44 Trillion GB!80% of data will be unstructured
- 4. 4© Cloudera, Inc. All rights reserved. Value is maximized when data is combined with other sources Value of Data is multiplied when you combine and correlate it with other data from relevant sources Improvement in value that can be unlocked by combining data from multiple IoT applications and sources SOURCE: McKinsey Global Institute analysis Interoperability would significantly improve performance by combining sensor data from different machines and systems to provide decision makers with an integrated view of performance 40%
- 5. 5© Cloudera, Inc. All rights reserved. The IoT Ecosystem Consumer Industrial IoT Gateway Data Center Data Analytics Sensors/ Things Data Characteristics • Un-structured • Intermittent • Volume & Variety Gateway • Data Routing • Edge-Processing • Edge-Storage Sensors/ Things •To grow by 50X •Drop in prices by 70% in last 5 years Data Storage, Processing & Analytics IOT Data Characteristics • More processing in the cloud • Analytics on the cloud IOT Data Analytics • Key to Value Creation • Combine data from multiple sources & types • Drive business insights IOT Data Characteristics • Distributed Data Processing • Cloud & On-Premise Cloud
- 6. 6© Cloudera, Inc. All rights reserved. IoT Attributes • Low powered devices, possibly battery powered • Highly Distributed • Gateway/Controller possibly mesh network • Compact messages
- 7. 7© Cloudera, Inc. All rights reserved. IoT Challenges • Multiple protocols (Z-wave, Zigbee, Thread, etc) • Distributed, low power may mean data coming from multiple locations • May power off to save battery or away from controller, need to handle late data • Calibration between devices may be limited • Very fast and bursty traffic • Low bandwidth last mile
- 8. 8© Cloudera, Inc. All rights reserved. Use Cases • Yes, Contrived • But a good excuse to: • Brew Beer • Buy more sensors and microprocessors • Sorry Wife
- 9. 9© Cloudera, Inc. All rights reserved. Use Case - Calibration • Sensors need to continually be calibrated • Calibration takes resources and down time • Instead use historical raw data • Calibrate on known values • For temperature sensors use bowling point and triple point • Temperature sensor is typically linear between these points • Fit a curve instead
- 10. 10© Cloudera, Inc. All rights reserved. Use Case - Optimize Models • Kalman Filter is used to estimate variable with presence of noise • Need to know accuracy of sensor • Usually published by manufacturer but generalized • Accuracy can degrade over time • PID Controller • 3 parameters control performance • Parameters different for each application
- 11. 11© Cloudera, Inc. All rights reserved. Use Case - Predictive Maintenance • No, not just for heavy machinery • Sensors fail too • Can save money by not replacing too early • More importantly, schedule downtime • Better Model with more data – Sensors same application many factories
- 12. 12© Cloudera, Inc. All rights reserved. Technologies • Apache Kafka • Messaging Framework – Scalable, Fault Tolerant • Pub/Sub • Retains Data • Apache Spark • General Purpose Distributed Processing Framework • Multiple Components including Streaming • Streaming continually processes data • Apache Kudu
- 13. 13© Cloudera, Inc. All rights reserved. Kudu for IoT Why it matters
- 14. 14© Cloudera, Inc. All rights reserved. Kudu use cases Kudu is best for use cases requiring a simultaneous combination of sequential and random reads and writes • Machine data analytics • Example: IOT, Connected Cars, Network threat detection • Workload: Inserts, scans, lookups • Time series • Examples: Streaming market data, fraud detection / prevention, risk monitoring • Workload: Insert, updates, scans, lookups • Online reporting • Example: Operational data store (ODS) • Workload: Inserts, updates, scans, lookups
- 15. 15© Cloudera, Inc. All rights reserved. How would we build the Analytics System Today? • HDFS Excels at: • Full table scans • Ad-hoc analytics Click to enter confidentiality Sensors Kafka / Pub-sub Events Today’s Partition Yesterday’s Partition Historic Data AnalystIngest 1. Have we accumulated enough data? 2. Flush into HDFS
- 16. 16© Cloudera, Inc. All rights reserved. Handling Late Arriving Data Click to enter confidentiality /cars/01-13/ /cars/01-14/ /cars/01-15/HDFS (Storage)
- 17. 17© Cloudera, Inc. All rights reserved. Hybrid big data analytics pipeline Before Kudu Sensors Kafka / Pub-sub Events HBase Consumer HDFS (Storage) Random Reads Analyst Analytics Snapshot & Convert to Parquet Compact late arriving data
- 18. 18© Cloudera, Inc. All rights reserved. Hybrid big data analytics pipeline After Kudu Sensors Kafka / Pub-sub Events Kudu ConsumerRandom Reads Analyst Analytics Kudu supports simultaneous combination of sequential and random reads and writes
- 19. 19© Cloudera, Inc. All rights reserved. What Kudu is *NOT* • Not a SQL interface itself • It’s just the storage layer • Not an application that runs on HDFS • It’s an alternative, native Hadoop storage engine • Not a replacement for HDFS or HBase • Select the right storage for the right use case
- 20. 20© Cloudera, Inc. All rights reserved. Kudu Trade-Offs (vs Hbase) • Random updates will be slower • HBase model allows random updates without incurring a disk seek • Kudu requires a key lookup before update, Bloom lookup before insert • Single-row reads may be slower • Columnar design is optimized for scans • Future: may introduce “column groups” for applications where single-row access is more important
- 21. 21© Cloudera, Inc. All rights reserved. Demo
Notas do Editor
- Lets start by taking a look at the market potential for IoT: Billions of devices include everything from cars, homes, airplanes, parking meters, factories, oil rigs, heavy machinery to wearables will be connected to the internet and more importantly will be interconnected enabling businesses to work smarter, faster and more profitably. If you look at the market potential for IoT, We are talking about significant growth and some big numbers here. By 2020 we are talking anywhere from 30 – 50 Billion connected things depending on who you talk to… There will be around a quarter billion connected cars on the roads. And It is estimated that IoT will generate ~1.7 Trillion US Dollars in value in the next 4-5 years with an approx. growth rate of 20% YoY.
- Data is the key to IoT – all of the ability to gain insights out of all of this data However, IoT isn’t just about the things or connecting these objects to the Internet; IoT is really going to be all about the data. With 30 Billion things connected, IoT Will Drive An Explosion of Data… The amount of data on the planet is set to grow 10-fold to around 44ZB. . If you are wondering how much that is – That is about 44 trillion GBs of data. Not only that – over 80% of that data is going to be unstructured/ semi-structured. So the question then becomes - how can you effectively manage, store, process, analyze and drive insights into all of this data that IoT is going to generate?
- Data coming in from just one sensor has value, but limited value. Real value from this data can be exploited by combining with data from other IoT sensors or combining it with Internal & external data. So for example – its good to know that your brake pads need to be replaced in your care, through sensors, but auto manufacturers are taking it to the next level – They want to combine that data with other data about the customer including what make and model is the car, where does the customer live, how does he or she like to shop and then send targeted offers to the customer saying – Here is an offer for your brake pad change, at your favorite body shop and you here is a coupon for 15% off your brake pad replacement service. ------------------------------------- McKinsey estimates that situations in which two or more IoT systems must work together can account for about 40 percent of the total value that can be unlocked by the Internet of Things. For ex. Interoperability would significantly improve performance by combining sensor data from different machines and systems to provide decision makers with an integrated view of performance across an entire factory or oil rig. While most use cases involve an immediate response — e.g., when a sensor detects a water leak — the bigger value may be in analyzing historical data or combining it with other data sets.
- 30-70% Drop in the price of MEMS sensors in past five years – McKinsey Research Diverse data types – from intermittent sensor readings of temperature and pressure to real-time location data or streaming live videos for video analytics Given the flexible, scalable nature of cloud-based infrastructure and the fact that machine data often originates off premises, we expect a lot of IoT data to be stored and processed in the cloud. The ideal IoT data platform can be deployed either on premise or in a public, hybrid, or private cloud environment. It should be possible to administer the platform via both a web-based interface and API calls. Gateways collects, aggregates, and optionally processes the data generated by the devices. The gateway can also accept and route commands sent from the backend to the respective device. Gateway is responsible for authenticating and authorizing the devices to participate in the workflow. It ensures secure communication between the devices and the centralized command center. The gateway is capable of dealing with multiple protocols and data formats. Response to edge analytics: Having access to all of your data is important, but with access comes responsibility and you a need a strategy about which data needs to be collected at the atomic level, which data needs to be rolled up and aggregated, and which data needs to be used to run your business. We are not saying all and every bit of the data generated by every sensor needs to make it way back to the data center. For some data it might make sense to collect, store, interpret and respond to locally. But organizations need a strategy about which data needs to be collected at the atomic level, which data needs to be rolled up and aggregated, and which data needs to be used to run your business. You will have to decide what happens at the edge, at the core, and perhaps in-between. For example, rather than send all sensor data to a central location, an edge device or software solution may send a summary of the data or trigger an automatic alert based a threshold-level status change. However there are few things you need to mind 1) you need to ensure you are not building up hundreds of different data silos that sits out there and you lack a centralized/ comprehensive view of the business – That is really a huge step backwards from both a business and IT perspective and 2) Security & Governance – Do you want sensitive data, customer data sitting in thousands of edge sensors or gateways significantly increasing your risk of a breach.