O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Streaming real time data with Vibe Data Stream

2.183 visualizações

Publicada em

The process of streaming real-time data from a wide variety of machine data sources and entities can be very complex and unwieldy. Using an agent-based approach, Informatica has invented a new technique and open access product that makes this process much more user friendly and efficient, even when dealing with multiple environments such as Hadoop, Cassandra, Storm, Amazon Kinesis and Complex Event Processing.

Publicada em: Dados e análise
  • Seja o primeiro a comentar

  • Seja a primeira pessoa a gostar disto

Streaming real time data with Vibe Data Stream

  1. 1. A Practical Guide to Improving the Big Data Ingestion Process Presented by Alan Lundberg and Amrish Thakkar July 22, 2014
  2. 2. Safe Harbor The information being provided today is for informational purposes only. The development, release and timing of any Informatica product or functionality described today remain at the sole discretion of Informatica and should not be relied upon in making a purchasing decision. Statements made today are based on currently available information, which is subject to change. Such statements should not be relied upon as a representation, warranty or commitment to deliver specific products or functionality in the future.
  3. 3. Informatica Marketplace A Data Integration Ecosystem Partners Consumers Developers Informatica • Software, Services Vendors • Strengthen Partnership • Generate Awareness • Discover Solutions • Evaluate Products • Request Ideas • Administrators • Architects • Data Analysts • Contribute, Collaborate • Enable Customers • Engage & Interact • Identify Whitespace
  4. 4. Informatica Marketplace 1300+ Apps, Add-ons and Services to jump-start your productivity Data Integration Mappings, Utilities, Connectors, Code Testing and Deployment, Monitoring, Job Scheduling Data Quality Rules & Reference Data, Health Check, Accelerators, Services Cloud Connectors, Templates, Data Loaders, Plugins, Process Automation, Services
  5. 5. 6 6 Data / Sensor Diversity…
  6. 6. Architectural Implications Yesterday Today Data structured, homogenous High Volume and variety Centralized Database-centric Distributed Systems Client Server Systems Batch processing Prioritize Modeling events as enterprise objects / assets Real Time Events treated as 2nd class citizens
  7. 7. 8 How to make sense of it all… Transactions, OLTP, OLAP Documents and Emails Social Media, Web Logs Vibe Data Stream Vibe Data Stream Vibe Data Stream Machine Device, Scientific Event Processing Engine
  8. 8. Use Cases – Solving the Difficult Problems Detect Patterns Exception Monitoring Process Monitoring • 3 events within 5 milliseconds • A then B then C occurs • Geospatial processing • Deviations from norm (Monitoring, Fraud, Error) • Trending up/down to exceed a threshold • SLA monitoring • Are process workflows operating properly? • Are manual processes completed on time? • Detect Missing Work and Queued Work
  9. 9. Architectural Approach for Streaming Analytics Location Context (e.g. GIS) Event Based Applications Streaming Analytics RulePoint CEP Real Time Stream Transport / DeliveryUltra Messaging Operational Data Ultra Messaging PowerCenter CDC / Data Access (Field Devices, Applications, Clickstream, IoT, logs, etc.) Data Warehouse Hadoop / NoSQL Analytics CDC PWX Various Source Data Integration Applications / Technologies Streaming Collection Vibe Data Stream Stream Transformation B2B Data Transformation Power Exchange
  10. 10. Streaming Collection: Vibe Data Stream (VDS) • Distribute collection across one or thousands of endpoints • High performance/efficient streaming data collection over LAN/WAN • Available ecosystem of light weight agents (sources & targets) • Continuous ingestion of real-time generated data (sensors; logs; etc.) to multiple targets (batch/stream processing) • Perform filtering, transformation, etc. “close to the source” • Provide varying qualities of service • Streaming, guaranteed, etc. • Allow for dynamic configuration • Highly available and scalable
  11. 11. Low latency messaging is the foundation • The core of Informatica’s Vibe Data Stream is based on the Ultra Messaging platform • Stream transport is the core of any streaming analytics solution • Required for key streaming analytics capabilities, including: • Stream collection • Stream distribution • Load distribution and sharing • Remote connectivity and routing • Ultra Messaging has been proven in hundreds of low-latency, guaranteed delivery, and fault-tolerant deployments
  12. 12. Web Log Data Market Data 00:00:46: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up 00:00:47: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up 00:00:47: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed state to up 00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down 00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down 2 *Mar 1 18:46:11: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36) 18:47:02: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36) *Mar 1 18:48:50.483 UTC: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36) 00:00:46: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up 00:00:47: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up 00:00:47: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed state to up 00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down 00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down 2 *Mar 1 18:46:11: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36) 18:47:02: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36) Location *Mar 1 18:48:50.483 data UTC: %SYS-5-CONFIG_I: Configured from console by vty2 (10.34.195.36) Device Data
  13. 13. VDS Node VDS Node VDS Node VDS Node VDS Node VDS Node VDS Node VDS Node VDS Node VDS Node VDS Server ZooKeeper logserver1 logserver2 logserver3 logserver4 logserver5 logserver6 logserver7 logserver8
  14. 14. Ecosystem of Sources and Targets Sources Targets Transformations Power Center B2B Data TX RulePoint … and evolving
  15. 15. Vibe Data Stream vs Flume VDS Flume Architecture Broker-less Non-messaging Configuration Automatic Manual Failover Automatic Automatic Functionality Event Aggregation/ Messaging Log Aggregation Recommended QoS Guaranteed Guaranteed Primary Application Trades/CDRs/ logs/ etc. logs Monitoring Yes No Enterprise Product integration Informatica product line No
  16. 16. Vibe Data Stream performance vs Flume-ng 200 20.67 Flume Vibe Data Stream > 10x performance Test Setup Event Size: 300 bytes Source Type: Syslog Number of Sources: 16 Target Type: HDFS Hadoop Cluster: 9-node VDS/Flume Nodes: 1 MB/sec MB/sec
  17. 17. • Demo
  18. 18. Download Vibe Data Stream Free Today! • Vibe Data Stream Open Access Download: http://www.marketplace.informatica.com/vds
  19. 19. Thank You! Don’t build it. Find it. marketplace.informatica.com

×