Log I am your father
•Transferir como PPTX, PDF•
1 gostou•2,252 visualizações
Log I am your father
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (20)
Semelhante a Log I am your father
Semelhante a Log I am your father (20)
Mais de DataWorks Summit/Hadoop Summit
Mais de DataWorks Summit/Hadoop Summit (20)
Último
Último (20)
Log I am your father
- 1. Copyright © 2016 Splunk Inc. Log I am your father: search your data, you know it to be true James Hodge, Principal Product Manager
- 2. Spelunking: Splunking: to explore underground caves to explore machine data
- 3. Explore your IoT data
- 4. 4 The data we know and use The available data we don’t know or use
- 5. 5 Nice, structured, tidy data
- 6. It began with logs…
- 7. 7 Machine data time series, in motion, unstructured
- 8. It can be big data…
- 9. …and it is slightly crazy “data”…
- 10. 10 … and it is messy and lazy
- 11. SECURITY IT OPERATIONS CUSTOMER EXPERIENCE RISK & COMPLIANCE PRODUCT PERFORMANCE IOT But it has many uses and lots of untapped value
- 13. 13 Make machine data accessible, usable and valuable to everyone. 13
- 14. [2014-09-04-14.45.54.608000] proc_source="B24A", tmst_target="2013-09-04-14.45.54.724000", serv_id="ISS", proc_input="MAST", proc_target="B24H", interface_acq="BNET_1", interface_iss="02008", cod_msg="XJYZ", oper_rrn="XJYZ", card_id="52xxxxxxxxxxx", oper_amount="000000000050", oper_currency="978", oper_country="380", term_id="0059XXXX", circuito="", sett_merc="4722", bin_acq="XXXX", id_merc="32xxxxxxxxxx", prcode="XYZ", action_code="XXX", ... ... auth_rout_id="HISO_AUTH", msg_subst="", ndg="00000xxxxxxxx", station_acq="STA-BNET-MI1", acceptor="A COOL SHOP", tmst_ins="2013-09-04-14.48.56.277466", ... Mastercard Circuit Client IDMerchant name AmountCard ID (masked) Merchant category IDMerchant ID There’sgold in thattheredata
- 15. Turning Machine Data Into Business Value 15 Platform for Machine Data Application Delivery IT Operations Security, Compliance and Fraud Business Analytics Industrial Data and Internet of Things
- 16. HA/DR Admin Data Security Apps SDKs/APIsScale Collect Data Index Data Enrich Data Search & Explore Analyze & Predict Report & Visualize Alert & Action 16 Fully Integrated Enterprise Platform
- 17. Technology Telecommunications Travel and Leisure Education Healthcare Energy and Utilities Manufacturing Financial Services and Insurance Media Proven at 11,000+ Customers in 100 Countries More Than 80 of the Fortune 100 Retail Cloud and Online Services Government
- 18. Splunk Capabilities for IoT Data Ingest at Scale Partner Ecosystem Advanced Analytics and Visualization Sense and Respond
- 19. IoT and Industrial Machine Data DevelopVisualize PredictAlertSearch Engineers Data Analysts Security Analysts Business Users Native Inputs TCP, UDP, Logs, Scripts, Wire, Mobile SDKs and APIs Java, JS, C#, Python, Ruby, PHP Modular Inputs MQTT, AMQP, COAP, REST, JMS HTTP Event Collector Token Authenticated Events Real-time Technology Partnerships Kepware, ThingWorx, Cisco, Palo Alto Maintenance Info Asset Info Data Stores External Lookups/Enrichment OT Industrial Assets IT Consumer and Mobile Devices
- 20. HTTP Event Collector Supports DevOps and IoT data analysis needs at scale 20 • Standard API and logging libraries send events directly to Splunk • Libraries integrated with popular platforms and services Scales to Millions of Events/Second AWS IoT EVENT COLLECTOR API
- 21. Advanced Analytics Splunk ML Extensions • New/Enhanced Commands: Data Sampling, Fit, Apply, Summary, Predict • Access to Python Data Science Library • Model storage and export to production Splunk ML App • Step-wise guidance to create, test and deploy custom ML models • Purpose-built visualizations • Sample data and best practices Optimizing enterprise operations with predictive ML analytics
- 23. How VW Visualizes Connected Car Data VW Data Labs Connected Car program Post-sales big data visualization Customer loyalty & retention IoT analysis & prediction of customer needs
- 24. Sense and Respond Use Splunk Alerts and Custom Alert Actions to trigger & automate workflows ● Allows packaged integration with third-party applications ● Simple admin/user configuration ● Developers can build, package, and publish alert actions within an app ● Growing list of integrations available 24
- 25. Splunk’s IoT and Industrial Partner Ecosystem 25 SDKs UI Ingest and Platforms IoT and ICS SecurityAdvanced Analytics and ML Custom User Interfaces Services and Delivery
- 26. Customer Success
- 28. Zebra Technologies Corporation 28 Founded in 1969, Zebra offers customers a complete end-to-end solution – from mobile computers and scanners to specialty printers, RFID, software and services – for identifying, tracking and managing critical assets, people and transactions. Headquarters: Lincolnshire, IL Offices: 122 offices across 81 countries Employees: 7000+ globally Market Capitalization: $5,640M1 Revenue: $2,275M 1 Profit (EBITDA): $439M 1 Mobile Computing Printers Data Capture (barcode scanners) RFID Location solutions Wireless LAN About Global presence Financials Products 2,000+ NFL players tagged and tracked since 2014, generating more that 68+ billion bytes of player position data 1 Zebra corporate fact sheet, summer 2015
- 29. Challenge for Zebra Printer Division 29 Limited data analysis capabilities . . . • Data not being collected for long-term use • Reports not being generated • Data not shareable across departments • Changes to traditional database took a long time • Minimal analytics capabilities (e.g., Excel) • Unable to scale to increasing data volumes Reactive mindset: Teams were reacting to issues after they occurred Little continuous improvement: Company was unable to leverage data to improve • Product design • Manufacturing • Repair and service Resulted in . . . Goal is to design a system that: • Captures complete device and test history • Is flexible and can be easily modified • Is capable of robust analytics, trending, & alerting capabilities • Can easily share data to enable better business decisions
- 30. Solution developed by Zebra 30 Test History History of the test performed by the test utility on the device Test Data Any data deemed necessary to log and store Device (Printer) Data Raw data output of device (4 data formats) 1 manufacturing center 8 service centers 3 reconfig centers 30+ applications 600+ computers globally Allows for real-time view of production data to address low yields and drive quality improvements Communicates product performance over time, leveraging statistical methods to baseline product performance Anomaly detection analytics utilized to capture known defect patterns that prevent product shipment Dashboard Reports Analytics
- 31. Value realized by Zebra 31 Quick access to test history and device data allows for enhanced customer experience Faster customer resolution Enhanced product development Reduced return rate Improved business decisions Reduced cost Data being captured early in development drives improvements in design Anomaly detection analytics used to capture known defect patterns that prevent product shipment Data being easily shared across Quality, Product, Technical Support, and Services enables data-driven decision making process Quantified cost reduction of $90,000 per year through media reduction alone 1 2 3 4 5
- 32. Planes, Trains and Automobiles (and Coke, Zombies, Floods, Buildings, Crops and Medicine)
- 33. Data-driven Refreshment Aggregate machine data from Coca-Cola freestyle® machines Insights into customer interactions and decisions Reduced Downtime and Increased Consumer Satisfaction Vending machine performance and diagnostics
- 34. More people buy Coke just before The Walking Dead
- 35. 35 How Gatwick Airport Ensures Better Passenger Experience With Splunk Cloud On-time efficiency & dramatic queue reduction with 925 flights per day Real-time, predictive airfield analytics - mobile app & CEO’s apple watch Data from airport gates, board pass scans, x-ray, travel, passenger flow
- 38. Saving The US Rail Industry A Billion Dollars And 250 Million Acres Of Trees in CO2 Train sensor data Fuel savings Better trained drivers
- 39. 24 Hour DB Hackathon Highlight defect impact & rail construction issues Predictive maintenance & reduced disruption Transport, infrastructure, environment & journey data How Deutsche Bahn Analyzed Tracks in 24 Hours
- 41. Top ten types of notifications about issues that have occurred in the transportation infrastructure
- 42. What kinds of defects occur and quantify how big the deviations are Correlated with materials of the track sleepers: concrete (“Beton”) vs. wood (“Holz”) Track deviations with indication notifications. Blue bars denote “no known issues”
- 43. Width of Sankey bar shows amount of track deviations between different destinations Fulda and Frankfurt has high track deviations (it is the widest bar) Indicates the need for upcoming repair, maintenance and possible renewal
- 44. Water Level Sensor Water Level Sensor Data in Monitors Real-time Data In Advance of Future Impending Flood Situations Alert Mobile Devices
- 45. Robot Analytics to Reduce Costs in the Supply Chain 4%Increased Throughput per Distribution Center Aggregate machine data from robots Failure pattern detection and reporting Preventative maintenance scheduling
- 46. 46 Remote Freight Train Monitoring Energy Efficiency Calculations Driving Strategy Recommendations Over $1 Billion Saved
- 47. Sensor Data Delivering Millions of Dollars in Energy Savings
- 48. + Content browsed, purchased and watched. All tracked by time and MAC address Customer behavior analytics Customer profile and MAC address / device assignments Understanding Customer Behavior
- 49. IoT WORKLOADS Blurring the Lines Between Digital & Physical Security Ops Center Business Ops Center IT Ops Center CLOUD WORKLOADS ENTERPRISE IT WORKLOADS ADVANCED ANALYTICS RAPID SOLUTION DEVELOPMENT DATA INGEST AT SCALE
- 50. Thank You
Notas do Editor
- At Splunk, our mission is to make machine data accessible, usable and valuable to everyone. And this overarching mission is what drives our company and product priorities.
- Splunk products are being used for data volumes ranging from gigabytes to hundreds of terabytes per day. Splunk software and cloud services reliably collects and indexes machine data, from a single source to tens of thousands of sources. All in real time. Once data is in Splunk Enterprise, you can search, analyze, report on and share insights form your data. The Splunk Enterprise platform is optimized for real-time, low-latency and interactivity, making it easy to explore, analyze and visualize your data. This is described as Operational Intelligence. The insights gained from machine data support a number of use cases and can drive value across your organization.
- Splunk provides an open, fully integrated platform. That means you can collect, index, analyze, report and predict on machine-generated data from a single product. It’s enterprise-ready with high availability and disaster recovery features, role-based access control and scales to index hundreds of terabytes per day. It’s an open platform with over 500 Splunk Apps available and allows for custom development.
- More than 10,000 customers in 100 countries have purchased the enterprise license of Splunk. This includes a majority of the Fortune 100. Enterprises, service providers and government agencies in 100 countries use Splunk to improve service levels, reduce IT operations costs, mitigate security risks and drive new levels of operational visibility. As they gain new visibility into their real-time and historical machine data, Splunk’s customers are finding answers and solving the most challenging issues facing IT and the business.
- Now you can onboard data directly from any application or device– opening up new types of machine data to the benefits of Splunk analysis. The new Event Collector makes it simple and efficient to collect this data, scaling to millions of events per second, using a developer-friendly, standard HTTP/JSON API and logging libraries And NO FORWARDERS. Today it is possible to send data directly to Splunk using Modular Inputs or a TCP connection, however this is not an efficient or scalable solution. While log files and forwarders provide an efficient mechanism for typical log and syslog files, use of files and forwarders is not possible or necessarily a desired data collection method for the world of custom applications DevOps, Docker, and other packaged application environments. The same is true for the world of IoT event data, where devices/apps need have no local storage, and even intermediate event collection systems and partners would prefer to use a real-time interface to Splunk rather than create specific log files and use forwarders. The HTTP Event Collector (EC) uses a standard API and high-volume Splunk endpoint to allow events to be directly sent/collected at extreme velocity. The HTTP/JSON API is a developer-standard, whose simple but powerful functionality will be attractive to DevOps and custom application developers and operations managers. Without requiring new system configuration, log creation or administration support, developers can instrument their applications to understand usage flows, performance, error conditions and more. The interface/functionality is also a fit for IoT software developers to connect their devices either directly or via intermediate collection services. The data volumes supported by Splunk are ideal for the transactional and diagnostic data of devices such as Point-Of-Sale systems, vending machines, gaming consoles, automobiles and other devices/systems – opening up a new world of machine data to the benefits of Splunk analysis An ecommerce saas provider uses EC to analyze the performance of their Docker apps A large video game company uses EC to monitor security of a popular sports game. 16B events collected in first 2 months
- NY Air Brake – Optimizing fuel efficiency per route Large Telecom – Main use case is to forecast possible telecom network outages - “intelligent performance-based alarming”. Happy ITSI customer who wants to do more free form ML with more data. Another Large Telecom – Proactive cell tower outage prediction and avoidance Online real-estate marketplace and app – First use case is website app troubleshooting; Second is identifying fraudulent web visitors. Electronics company specializing in video games – Predicting and avoiding online services outages Financial Services company – Detecting lateral movement inside network Another Financial Services company – ML services to app developers and support Large food and beverage company – Real-time inventory forecasting
- Industry: Automotive/Manufacturing Use case: IoT and Analytics More can be found at: http://www.techworld.com/news/big-data/volkswagen-group-uses-splunk-underpin-iot-experiment-3575744/ Splunk have been working with VW’s data labs on their connected car and IoT program. Splunk is being used to analyse sensor data from VW’s E-Up cars. Splunk’s ability to search, alert, report and analyse IoT data allowed VW to see a fleet of E-Up cars at CeBIT and analyse metrics such as speed, RPM, battery level, range, temperature, when the doors were open and when windscreen wipers were on. This data could also be combined with location, maps and smart watches to show where vehicles had been, heatmaps, and driver heart rate
- Custom Alert Actions provide the ability to use Splunk Alerts to trigger custom actions or pre-packaged integrations with 3rd party products such as trouble ticketing or support systems. Developers can build and publish integrations or custom action packages that users or admins can use via a simple menu within the Splunk Alert Interface. Splunk and partners provide a growing set of integrations including, ServiceNow, xMatters, Webhooks and more. Previously these integrations were complex, ad-hoc efforts requiring custom scripts. The new scheme makes it simple for partners (and customers) to create and contribute out-of-the-box integration templates, and for customers to use them via a simple pull-down menu.
- There are many free add-ons and Apps for Splunk software that simplify the connection and collection of data from both industrial systems and the Internet of Things. These include: Rest API Modular Input: Poll local and remote REST APIs and index the responses. Amazon Kinesis Modular Input: Index data from Amazon Kinesis, a fully managed service for real-time streaming data. Apache Kafka Modular Input: Index messages from Apache Kafka messaging brokers, including clusters managed by Zookeeper. DB Connect 2: Integrate structured data sources with your Splunk real-time machine data collection. Universal Forwarder for Linux (ARM – Raspberry Pi): Dedicated Splunk package for Linux and ARM based systems where data needs to be collected directly from embedded devices such as the Raspberry Pi. MQTT Modular Input: Index messages from MQTT, a machine-to-machine connectivity protocol, by subscribing Splunk software to MQTT Broker Topics. AMQP Modular Input: Index data from message queues provided by AMQP brokers. JMS Modular Input: Poll and index message queues and topics from messaging queues and topics, including MQTT messages, provided by message providers, including TibcoEMS, Weblogic JMS and ActiveMQ. Protocol Data Inputs: Recieve data via a number of different data protocols such as TCP , TCP(s) ,HTTP(s) PUT/POST/File Upload , UDP , Websockets , SockJS. Splunk App for Stream: Capture, filter and index real-time streaming wire data and network events. COAP Modular Input: Index messages from a COAP (Constrained Application Protocol) Server. SNMP Modular Input: Collect data by polling SNMP attributes and catching SNMP traps from datacenter infrastructure devices providing cooling and power distribution. In addition, Splunk has a powerful ecosystem of technology partners. Kepware Technologies – Connects Splunk software with thousands of industrial devices communicating on over a hundred proprietary industrial protocols. Stream real-time data to Splunk from industrial control systems, including SCADA. Carvoyant – Connected car platform, integration with Splunk software allows enterprises to monitor their automobile fleets, including geo-location, engine parameters and diagnostics. B&B SmartWorx – Intelligent sensors and gateways. Integration with Splunk (Splunk App) will include sensor data collection (via MQTT), and gateway and sensor network diagnostics and cyber security. Bluvision– Intelligent beacons. Integration with Splunk (Splunk App) will include beacon data collection (via Websockets). Powerful retail applications. ThingWorx (PTC) – The leading IoT Application Development Platform. Seamless data exchange between ThingWorx applications and Splunk Enterprise and Splunk Cloud, and ThingWorx customers can access Splunk search and analytics through the ThingWorx mashup builder. Buddy.com – Cloud services for connected devices. Integration (Splunk App) will allow Splunk to stream data from any device connected to the buddy platform. Octoblu (Citrix) – IoT developer platform. Has created libraries that allow any Octoblu-enabled device to stream its data to Splunk software and allows those same devices to use Splunk search and analytics to inform their own decisions and logic Red Balloon Security – Security platform for the defense of embedded systems in the enterprise (IP Phones, Printers, switches and routers, etc). Uses proprietary firmware level protection and appliance-based endpoint monitoring, and is integrating (Splunk TA/ES Compliant CIM) with Splunk software to allow Enterprise Security monitoring of threats to embedded enterprise devices. Bayshore Networks – Content-aware cyber security platform for industrial networks. Is integrating (Splunk TA/ES Compliant CIM) with Splunk software to allow Enterprise Security monitoring of threats to SCADA and other industrial networks. Foxguard Solutions– Cyber security and compliance solutions for industrial networks built with Splunk. NERC-CIP compliance specialists. UltraElectronics-3eti – Cyber security platform for industrial networks. Building ES compliant TA to allow collection and analysis of security relevant ICS data in Splunk. Distrix – Software defined networking for industrial networks and the internet of things. Simplifies connectivity and delivers and enhances data over extremely complex networks. Distrix’s SDN supports Splunk to Splunk communication, and can enhance other data, including timestamping and meta-data enrichment, for ingestion in Splunk. Prelert – Anomaly detection app for Splunk Enterprise. Valuable app for management of sensors and devices where rapid identification of anomalies in sensor readings or operations are critical. Predikto – Leverages the power of Predictive Analytics enabling organizations to use their data to predict future asset failures. N3N – Custom, advanced user interfaces for Splunk specializing in isometric views of industrial facilities. R Project App – harness the power of R statistical processing language directly from Splunk interfaces and search processing language. D3.js – Data driven documents for powerful user experiences. HTML5 – Advanced web interfaces and applications for browser and mobile based user experience.
- Coca-cola uses Splunk to understand connected vending machines, including the Coca-Cola Freestyle. They are able to better understand patterns in consumption as well as the performance of the machines, both of which ultimately lead to better customer satisfaction.
- Splunk have been working with VW’s data labs on their connected car and IoT program. Splunk is being used to analyse sensor data from VW’s E-Up cars. Splunk’s ability to search, alert, report and analyse IoT data allowed VW to see a fleet of E-Up cars at CeBIT and analyse metrics such as speed, RPM, battery level, range, temperature, when the doors were open and when windscreen wipers were on. This data could also be combined with location, maps and smart watches to show where vehicles had been, heatmaps, and driver heart rate
- Visuals need to be worked on
- New York Air Brake’s Train Dynamic Systems Division is using Splunk to manage inter-train forces, the “slinky factor” inherent in large freight trains with 6 inches of flex between cars. With splunk, they are able to produce insight and reports allowing the owners of the locomotives they manage to better train the engineers, and better manage the acceleration and braking of the trains throughout thousand mile journeys. Managing this data with Splunk, they can produce 5-10% fuel savings for customers. For their largest customers this can mean a billion dollars in savings a year.
- Energy efficiency is one of the most critical factors in any building project. It's estimated that commercial buildings consume nearly 20% of all energy in the U.S., the majority of which is used for lighting and indoor climate control. From site selection and layout to building materials and mechanical systems, energy efficiency is a primary goal at every stage in a building's lifecycle. In early 2012, McKenney's was enlisted to assist Gulf Power and its partner Chevron Energy Solutions in implementing a new energy management system at Eglin Air Force Base through Gulf Power's GSA contract services in Florida. At 724 square miles, Eglin is one of the largest military bases in the world and includes hundreds of buildings and a base population of about 17,000. The Enterprise Intelligence Group leveraged its in-house experience with Splunk to provide Eglin with continuous collection and aggregation of data from most of the energy management, building control systems and utility metering and monitoring systems on base, helping to monitor and analyze tens of thousands of sensors and data inputs from HVAC systems in more than 100 Eglin buildings. The new Eglin energy management system (EMS) will leverage the Splunk to provide dashboards that will help base maintenance staff to assess building performance and energy efficiency, generate automated Air Force/DoD energy usage reports, compare current energy usage with historical data, and enable the deployment of load shedding and load shifting strategies to take advantage of favorable electric rates. The project is projected to save about $2.5 million annually, with a payback period of less than three years.
- Comcast Corporation (Nasdaq: CMCSA, CMCSK) (www.comcast.com) is one of the world’s leading media, entertainment and communications companies. Comcast has many different Splunk use cases. One of their use cases involves taking data from the set-top boxes to gain real time insights in to customer interaction with content served up by the set top box. Each set top box has a media access control (MAC) address that is unique and is associated with a specific customer. The set top box is capturing all customer interaction with device including which content the customer searched for, what the date of search was, what search results were displayed (this information is recorded a unique identifiers called IDA numbers) and what content was purchased. However, the set top box does not have any information on the customer including their profile. That information is stored in the billing system. Comcast is using Splunk to correlate data across set top boxes and billing systems to gain real-time business insights. Using the correlation criteria of MAC address, content displayed in search and time of purchase, Comcast is gaining a broad range of business insights into their customers. For example, these insights are helping Comcast understand revenues driven by search. By overlaying this information with geo location data, they are able to improve content mix and drive higher monetization. These insights are also helping Comcast improve content promotion based on region. Comcast is using the Splunk and Hadoop integration to visualize Comcast setbox log information. The setbox data comes to Hadoop, get pre processed and moved to Splunk for visualization. Hadoop Input = High volume of data from many systems along a complex workflow, Developers expressing artistic prerogative on log formats, Many different data sources and formats Splunk Output = Drive operational intelligence, Improve user experience, Troubleshooting, root cause analysis, Track and measure success, Reports, alarms