SplunkLive! Washington DC May 2013 - Big Data Architectural Patterns
•
6 gostaram•1,944 visualizações
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (14)
Semelhante a SplunkLive! Washington DC May 2013 - Big Data Architectural Patterns
Semelhante a SplunkLive! Washington DC May 2013 - Big Data Architectural Patterns (20)
Mais de Splunk
Mais de Splunk (20)
Último
Último (20)
SplunkLive! Washington DC May 2013 - Big Data Architectural Patterns
- 1. Copyright © 2013 Splunk Inc. Raanan Dagan May 2013 Splunk Big Data Architectural Patterns: Hadoop and Database
- 2. Agenda • Architectural Patterns - Splunk and Hadoop • Hadoop Connect Demo • Architectural Patterns - Splunk and Database • DB Connect Demo
- 3. 3 Developer Platform Report and analyze Custom dashboards Monitor and alert Ad hoc search SQL Splunk Hadoop Connect • Reliable bi-directional integration to Hadoop Splunk DB Connect • Real-time integration to relational DBs Splunk: A Platform for Big Data Integration
- 4. Architectural Patterns Splunk and Hadoop
- 5. 5 Splunk Hadoop Connect Delivers reliable integration between Splunk and Hadoop Export events to Hadoop Explore and Browse Hadoop directories Import and Index Hadoop data into Splunk
- 6. Hadoop Connect: 3 Patterns Splunk then Hadoop 1 Hadoop then Splunk Combination and Search Commands 2 3 6
- 7. Splunk then Hadoop Splunk Ingest and provides: Real-time Analytics, End-user Security and Visualization 7 Splunk for Analytics Data Sources Hadoop for ETL
- 8. Hadoop Connect - Export 8 1. Splunk forwarders move data to an indexer 2. Search head stream data into a local directory 3. Periodically Splunk compresses the file and puts it into the HDFS directory (location set by users)
- 9. Hadoop Connect: 3 Patterns 9 Splunk then Hadoop 1 Hadoop then Splunk Combination and Search Commands 2 3
- 10. Hadoop then Splunk 10 Splunk for Analytics Data Sources Hadoop for ETL Hadoop Ingest. Splunk provides Interactive Analytics, End-user Security, and Visualization
- 11. Hadoop Connect – Import and Index 11 1. Splunk detects any updated or new file in the HDFS directory 2. Splunk imports the data into Splunk indexers 3. In Splunk you can apply access controls to the data as well as search, report and visualize your data
- 12. Hadoop Connect: 3 Patterns 12 Splunk then Hadoop 1 Hadoop then Splunk Combination and Search Commands 2 3
- 13. Splunk for Analytics Data Sources Hadoop for ETL • Splunk and Hadoop Share the data • Splunk for real-time Analytics • Hadoop for ETL Data flows in both directions 13
- 14. Hadoop Connect – Explore Enables Splunk to browse and navigate HDFS directories and files from the Splunk search head user interface Explore User interface wraps ‘hdfs lsr’ and ‘hdfs read’ 14 | hdfs read hdfs://kiru-demo-01.sv.splunk.com:9000/home/HadoopConnect/twitter/File1.gz | hdfs lsr hdfs://kiru-demo-01.sv.splunk.com:9000/home/HadoopConnect/twitter/
- 16. Architectural Patterns Splunk and Database
- 17. Splunk DB Connect Enrich search results with additional business context Easily import data into Splunk for deeper analysis Integrate multiple DBs concurrently Simple set-up, non-evasive and secure Reliable, scalable, real-time integration between Splunk and traditional relational databases Microsoft SQL Server JDBC Database Lookup Database Query Connection Pooling Other Databases Oracle Database Java Bridge Server 17
- 18. DB Connect: 3 Patterns 18 Database Lookup 1 Import Database Tables Search Commands 2 3
- 19. 19 Media Server Logs (Machine Data) Mar 01 19:18:50:000 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct start for 2172618992@splunktel.com 10.164.232.181 from 12.130.60.5 recorded OK. 2013-03-01 19:18:50:150 10.2.1.34 GET /sync/addtolibrary/01011207201000005652000000000053 - 80 - 10.164.232.181 "Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3" 503 0 0 825 1680 Mar 01 19:18:50:163 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct stop for 2172618992@splunktel.com 10.164.232.181 from 12.130.60.5 recorded OK. Track ID Artist Title Format ID Run time 01011207201000005652000000000053 Maroon 5 Moves like Jagger MP3 4:30 Phone # Subscriber ID 2172618992 53546 Subscriber ID First Name Last Name Age State Customer Score 53546 Jim Morrison 25 CA 93 Customer, Product Databases Phone Number IP Address Track ID Enrich Machine Data with additional Business Context DB Connect – Database Lookup
- 20. DB Connect – Database Lookup 20 1. Use ‘lookup’ in Splunk Search 2. Keys are sent from Splunk to Databases (Product ID) 3. Values are returned if Keys are matched (Product Name, Product Price) SQL KEYS VALUES
- 21. DB Connect: 3 Patterns 21 Database Lookup 1 Import Database Tables Search Commands 2 3
- 22. DB Connect - Import Database Table 22 Provide deeper analysis Machine Data Machine Data Machine Data RDBMS
- 23. DB Connect – Import Database Table 23 Two input types can be used to import rows from the database: • Tail = Bring only new or updated rows • Dump = Bring entire table SQL Import
- 24. DB Connect: 3 Patterns 24 Database Lookup 1 Import Database Tables Search Commands 2 3
- 25. Splunk Search Language Extensions Execute database queries directly from the Splunk user interface with new Dbquery and Dbinfo Splunk search commands 25 *** DBoutput (BETA) - Create or Update database records on information Splunk searches
- 26. DB Connect – Search Commands 26 1. Use SQL-92 or Stored Procedures with DBQuery 2. Database Info user interface wraps DBinfo and DBquery commands SQL QUERY RESULTS
- 28. Summary 28 Splunk and Hadoop: – Splunk provides real-time analysis, visualization, and security – Hadoop provides parallel ETL or batch computation Splunk and Database: – Enrich search results with additional business context – Import data into Splunk for deeper analysis
Notas do Editor
- Splunk Hadoop Connect has three main functions:Export events to Hadoop – Collect and index massive streams of machine data in real-time using Splunk. Search, analyze and visualize your data and provide secure dashboards to multiple users across your organization. Then send events in a reliable, predictable way to HDFS for archiving, further processing or additional batch analytics. Users have several options when exporting events. They can pre-process data in Splunk before exporting the results into Hadoop, or they can simply export raw events. For example, when processing Apache® web logs, you can limit the data export to just the client IP, session ID and the URL fields. Explore Hadoop directories and files – Browse and navigate HDFS directories and files from the Splunk Hadoop Connect user interface, before deciding to import data into Splunk. View directories, files names, permissions, sizes and dates of the last modification, all without having to open the actual files. Drill down into a set of directories, examine files and with a click of a button import and index data in Splunk. Import and Index Hadoop data in Splunk – Address Hadoop limitations, such as the lack of visualizations, limited support for user access controls, and the need for data scientists and specialists to analyze data or to write MapReduce code. Splunk Hadoop Connect detects any updated or new file in the HDFS directory, for example as a result of Sqoop, Hbase or Hadoop commands, and collects and indexes the data. Once in Splunk, your data is immediately available for searching, reporting, analysis and visualizations and protected by role-based access controls.
- There are a whole host of ways Developerscan leverage Splunk to maximize enterprise technology investments.Accelerate Dev & Test: Use Splunk Enterprise out of the box. Splunk increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in production.Integrate with IT Infrastructure: Integrate Splunk data with other enterprise applications, using SDKs on top of our REST API. Build real-time data applications: Build applications that take the value of Splunk beyond IT. IT early-warning systems, security and fraud protection, clickstream analysis & other revenue enhancing analytics.
- There are a whole host of ways Developerscan leverage Splunk to maximize enterprise technology investments.Accelerate Dev & Test: Use Splunk Enterprise out of the box. Splunk increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in production.Integrate with IT Infrastructure: Integrate Splunk data with other enterprise applications, using SDKs on top of our REST API. Build real-time data applications: Build applications that take the value of Splunk beyond IT. IT early-warning systems, security and fraud protection, clickstream analysis & other revenue enhancing analytics.
- There are a whole host of ways Developerscan leverage Splunk to maximize enterprise technology investments.Accelerate Dev & Test: Use Splunk Enterprise out of the box. Splunk increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in production.Integrate with IT Infrastructure: Integrate Splunk data with other enterprise applications, using SDKs on top of our REST API. Build real-time data applications: Build applications that take the value of Splunk beyond IT. IT early-warning systems, security and fraud protection, clickstream analysis & other revenue enhancing analytics.
- Splunk DB Connect delivers reliable, scalable, real-time integration between Splunk Enterprise and traditional relational databases. With Splunk DB Connect, structured data from relational databases can be easily integrated into Splunk Enterprise, driving deeper levels of operational intelligence and richer business analytics across the organization.Organizations can drive more meaningful insights for IT operations, security and business users. For example, IT operations teams can track performance, outage and usage by department, location and business entities. Security professionals can correlate machine data with critical assets and watch-lists for: incident investigations, real-time correlations and advanced threat detection using the award-winning Splunk Enterprise. Business users can analyze service levels and user experience by customer in real-time to make more informed decisions.
- There are a whole host of ways Developerscan leverage Splunk to maximize enterprise technology investments.Accelerate Dev & Test: Use Splunk Enterprise out of the box. Splunk increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in production.Integrate with IT Infrastructure: Integrate Splunk data with other enterprise applications, using SDKs on top of our REST API. Build real-time data applications: Build applications that take the value of Splunk beyond IT. IT early-warning systems, security and fraud protection, clickstream analysis & other revenue enhancing analytics.
- None of the customer's identifying information is in the logs (which I will add to the dashboard); we simply identified a phone that has had problems downloading music, and enriched that information with data from our customers database.
- There are a whole host of ways Developerscan leverage Splunk to maximize enterprise technology investments.Accelerate Dev & Test: Use Splunk Enterprise out of the box. Splunk increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in production.Integrate with IT Infrastructure: Integrate Splunk data with other enterprise applications, using SDKs on top of our REST API. Build real-time data applications: Build applications that take the value of Splunk beyond IT. IT early-warning systems, security and fraud protection, clickstream analysis & other revenue enhancing analytics.
- This leading national telecommunications company in Canada, with $10.8 billion of annual revenue and 13.0 million customer connections including 7.6 million wireless subscribers, 3.4 million wireline network access lines, 1.3 million Internet subscribers and more than 635,000 TV customers.For their use case, they are collecting all network switch data into a single Oracle Database. Oracle tables stores: CPU utilization per switch / inbound pockets per switch / 30 different stats going to Oracle Database. In addition, they have an alerts system when there is a high utilization of network traffic.Splunk DB Connect is used to import the data from Oracle for more effective analysis:Detect high utilization patterns. Trending behavior in a specific time of the day per switch (is it normal or not). Visualizing outliers (high performance patterns).With Splunk the team can take proactive measures before the network is saturated, which reduce network utilization alerts. Volume = Monitoring over 10,000 switches, each with multiple interfaces. Splunk imports and indexesmillions of rows per day.It took the main Splunk user only 2 days from initial installation to graph network trends / outliers.
- There are a whole host of ways Developerscan leverage Splunk to maximize enterprise technology investments.Accelerate Dev & Test: Use Splunk Enterprise out of the box. Splunk increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in production.Integrate with IT Infrastructure: Integrate Splunk data with other enterprise applications, using SDKs on top of our REST API. Build real-time data applications: Build applications that take the value of Splunk beyond IT. IT early-warning systems, security and fraud protection, clickstream analysis & other revenue enhancing analytics.
- Splunk software lets users search and navigate their data from one place. Splunk DB Connect includes search language extensions that can be executed directly from the Splunk user interface. Dbquery and Dbinfo are Splunk search commands that enable you to execute database queries directly from the Splunk Enterprise user interface. Dbinfo fetches schema information from the database.
- Quick to set-up, scales to multiple concurrent databasesEnrich machine data with structured data from relational databasesExecute database queries directly from the Splunk user interface Browse and navigate database schemas and tablesCombine machine data with structured data from relational databases