O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Introduction to MANTL Data Platform

776 visualizações

Publicada em

A session in the DevNet Zone at Cisco Live, Berlin. Big data and the Internet of Things (IoT) are two of the hottest categories in information technology today, yet there are significant challenges when trying to create an end-to-end solution. The worlds of "IT" and “IoT" differ in terms of programming interfaces, protocols, security frameworks, and application lifecycle management. In this talk we will describe proven ways to overcome challenges when deploying a complete “device to datacenter” system, including how to stream IoT telemetry into big data repositories; how to perform real-time analytics on machine data; and how to close the loop with reliable, secure command and control back out to remote control systems and other devices.

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Introduction to MANTL Data Platform

  1. 1. Dmitri  Chtchourov,   MANTL  Data  Pla6orm,  Microservices  and  BigData  Services   Innova?on  Architect,  CIS  CTO  Group    
  2. 2. Agenda Problem  &  Opportunity     What  do  we  want  to  do?     What  is  in  it  for  us?     How  does  it  work?     What  have  we  done  so  far?     Anatomy  of  a  Service     Reference  Architectures  and  real  use  cases     PuMng  it  all  together      
  3. 3. Problem & Opportunity Rapid  innova?on  in  compu?ng  and  applica?on  development  services                                            No  single  service  is  op.mal  for  all  solu.ons      Customers  want  to  run  mul.ple  services  in  a  single  cluster  and                                    run  mul.ple  clusters  in  Intercloud  environment                              ...to  maximize  u,liza,on                            ...to  share  data  between  services        …Complex/BigData  and  Microservices  together    
  4. 4. Technologies matrix*  Service    Product    Cloud/Virtualiza?on    CIS/AWS/Metacloud/UCS…    Provisioning    Open  Stack/Terraform    Automa?on    Ansible    Clustering  &  Resource   Management    Mesos,  Marathon,  Docker    Load  Balancing    Avi  Networks    ETL  &  Data  Shaping    StreamSets    Log  Data  Gathering    Logstash    Metrics  Gathering    CollectD,  Avi  Networks    Messaging    KaUa,  Solace    Data  Storing  (Batch)    HDFS    Data  Storing  (OLTP/Real-­‐?me)    Cassandra    Data  Storing  (Indexing)    Elas?c  search    Data  Processing    Apache  Spark    Visualiza?on    Zoomdata   *Subset  example  
  5. 5. Cloud   Management   Data  Collect   Data   Storage   Data   Processing   Visualisa.on   Technologies stack
  6. 6. Datacenter and solution today VM7 or BM7 VM8 or BM8 VM4 or BM4 VM5 or BM5 VM6 or BM6 VM1 or BM1 Visualization Service Data Ingestion Service Analytics Service •  Configuration and management of 3 separate clusters •  Resources stay idle if service is not active •  Need to move data between clusters for each service VM2 or BM2 VM3 or BM3 VM1 or BM1 VM2 or BM2 VM3 or BM3
  7. 7. What do we want to do? Data  Inges?on  Service   Analy?cs  Service   Visualiza?on  Service   ….to  maximize  u,liza,on   ...to  share  data  between  services   Shared  cluster   Mul.ple  clusters  
  8. 8. Shared Cluster CIS/AWS/Metapod/UCS… VM1 or BM1 VM2 or BM2 VM3 or BM3 VM4 or BM4 VM5 or BM5
  9. 9. What is in it for us?                      Maximize  u.liza.on      Deliver  more  services  with  smaller  footprint                                        Shared  clusters  for  all  services                Easier  deployment  and  management  with  unified  service  plaCorm                          Shared  data  between  services                            Faster  and  more  compe..ve  services  and  solu.ons                        Combine  paradigms  for  flexibility  and  func.onality                  Run  complex  services  and  microservices  in  the  single  environment    
  10. 10. How does this work? Mesos  Slave   Spark  Task  Executor     Mesos  Executor   Mesos  Slave   Docker  Executor   Docker  Executor   Mesos  Master   Task  #1   Task  #2   ./python  XYZ   java  -­‐jar  XYZ.jar   ./xyz   Mesos  Master   Mesos  Master            Spark  Service  Scheduler            Marathon  Service  Scheduler   Zookeeper  quorum  
  11. 11. How does this work?  Mesos  provides  fine  grained  resource  isola+on   Mesos  Slave  Process   Spark  Task  Executor     Mesos  Executor   Task  #1   Task  #2   ./python  XYZ   Compute  Node   Executor   Container   (cgroups)  
  12. 12. How does this work?  Mesos  provides  scalability   Mesos  Slave  Process                                            Spark  Task  Executor     Task  #1   Task  #2   ./ruby  XYZ   Compute  Node   Python  executor  finished,   more  available  resources   more  Spark   Container   (cgroups)   Task  #3   Task  #4  
  13. 13. How does this work? Mesos has no single point of failure Mesos  Master  Mesos  Master   Mesos  Master   VM1 or BM1 VM2 or BM2 VM3 or BM3 VM4 or BM4 VM5 or BM5 Services keep running if VM fails!
  14. 14. How does this work? Master node can failover Mesos  Master  Mesos  Master   Mesos  Master   VM1 or BM1 VM2 or BM2 VM3 or BM3 VM4 or BM4 VM5 or BM5 Services keep running if Mesos Master fails!
  15. 15. How does this work?  Slave  process    can  failover    Tasks  keep  running  if  Mesos  Slave  Process  fails!   Mesos  Slave  Process                                            Spark  Task  Executor     Task  #1   Task  #2   ./ruby  XYZ   Compute  Node   Task  #3   Task  #4  
  16. 16. How does this work?  Can  deploy    in  many  environments    Get  orchestrated  by  Openstack,  Ansible  (scripts),  Cloudbreak    True  Hybrid  Cloud  deployment:  CIS,  AWS,  UCS,  vSphere,   other   CIS/                                                                                                                                            CIS/AWS/Metpod/vSphere/UCS…   Terraform                    REST  API                    REST  API                    Scripted  provisioning          Direct  provisioning   Policy,  Auto-­‐scaling   VM1 or BM1 VM2 or BM2 VM3 or BM3 VM4 or BM4 VM5 or BM5
  17. 17. How does this work? Microservices managed and scaled separately Microservices managed by Mesos in a single platform Microservices architecture for Mesos frameworks and other components CIS/AWS/Metacloud/vSphere/UCS… Terraform Spark Executor N Spark Executor 1 Spark Scheduler Kafka Broker N Kafka Broker 1 Kafka Scheduler Docker Docker TraefikMicroservices … REST API REST API Scripted provisioning Direct provisioning Policy, Auto-scaling VM1 or BM1 VM2 or BM2 VM3 or BM3 VM4 or BM4 VM5 or BM5
  18. 18. What have we done so far? Working  with  partners  on  adop.ng  and  co-­‐developing  services               Partners   Co-­‐development  Partners  
  19. 19. Anatomy of the service/framework   Riak  is  Basho  Technologies  distributed  highly  available  database     Op?mized  Mul?-­‐Datacenter  opera?on     We  are  working  together  with  Basho  Labs  on  developing  and  tes?ng  their     Mesos  Service  version  of  the  product                
  20. 20. Riak Use Cases Online / Commerce ●  Session Control ●  Shopping Cart ●  Product Ratings and Reviews Internet of Things ●  Connected Device Data ●  Sensor Data ●  Log Data Content Management ●  Storing Unstructured Data ●  Content Personalization ●  Advertising Data Gaming ●  Store Leaderboard Info ●  Store Bet Transactions ●  Online Chat Digital Communications ●  Online Community Chat ●  Notification and Alerting ●  Mobile Messaging
  21. 21. Development phases   Phase  0:  Package  applica?on  in  Docker  container  to  deploy  on  Mesos     Phase  1:  Convert  applica?on  to  Microservices  Architecture  to  deploy  as   Mesos  applica?on  with  mul?ple  components     Phase  2:  Create  an  intelligent  scalable  Mesos  service  based  on  the   applica?on              
  22. 22. Riak Service: Components
  23. 23. Riak Service: Architecture
  24. 24. Riak Service: Persistence
  25. 25. Riak Service: Operational Simplicity
  26. 26. Riak Service: Highly Scalable E-­‐commerce  Applica.on  with  Varying  Traffic  
  27. 27. Anatomy of the service/framework    Zoomdata  is  distributed  highly  available  large  scale  visualiza?on  pla6orm     Op?mized  very  big  data  set  micro-­‐query  analy?cs     We  are  working  together  with  Zoomdata  on  developing  and  tes?ng  their   Mesos  Service  version  of  the  product                
  28. 28. Zoomdata Service: Components
  29. 29. Zoomdata Service: Mesos + Kubernetes Mesos  Slave   Mesos  Master   Mesos  Slave   Mesos  Slave   Zoomdata  web  app   Mongodb   Spark  Worker   Spark  Executor   Spark  Executor   Proxy   (haproxy,  nginx)   Kubernetes   Mongo   Service/RC   Kubernetes   Spark-­‐Proxy   Service/RC   Spark-­‐Proxy   Zoomdata  web  app   Zoomdata  web  app   Kubernetes   Framework   Kubernetes   Zoomdata   Scheduler   Service/RC   Zoomdata  Scheduler   ProxyGen  Script   User   ●  Every  component  (Zoomdata   App,  MongoDB,  Spark-­‐Proxy,   Scheduler)  must  be  started  in   independent  K8s  Pod  and  there   must  be  exactly  one  MongoDB,   Spark-­‐Proxy  and  Scheduler  Pods   meanwhile  Zoomdata  App  can  be   scaled  with  help  Kubernetes   Replica?on  Controller.   ●  There  must  be  defined   Kubernetes  Service  for  MongoDB,   Spark-­‐Proxy,  Scheduler  as  they   will  be  used  in  Zoomdata’s  App   Pod.  Every  docker  container  will   have  env  variables  for  every   present  Service  injected   automa?cally.    
  30. 30. Anatomy of the service/framework StreamSets  is  an  open  source  con?nuous  big  data  ingest  infrastructure       Accelerates  ?me  to  analysis  with    unprecedented  transparency  and   processing  to  data  in  mo?on.   Cluster  deployments     JVM,  Docker,  Spark  Streaming  on  Mesos   Con?nuous  Opera?ons    to  Minimize  down?me      
  31. 31. Advantages of Streamsets Adaptable  Data  flow  -­‐  Design  and  execute  intent-­‐driven  data  flows  in  a  graphical   IDE   Instream  Sani?za?on  -­‐  transform  and  process  the  data  on  the  fly   Intelligent  Monitoring  -­‐  Get  early  warnings,  detect  anomalies  and  take  ac?on   Link  origins  to  des?na?ons  with in-stream data preparation
  32. 32. Streamsets Data Pipeline MESOS Streamsets Data Collector (SDC) Architecture Cluster  Streaming  mode   Data  Collector  runs  as  an  applica?on  within  Spark  Streaming,   Spark  Streaming  runs  on  Mesos  cluster  manager  to  process   data  from  a  KaUa  cluster.     The  Data  Collector  uses  a  cluster  manager  and  a  cluster   applica?on  to  spawn  workers  as  needed.     Cluster  Batch  Mode  :   Data  Collector  processes  all  available  data  from  HDFS  and  then   stops  the  pipeline.     MapReduce  generate  addi?onal  worker  nodes  as  needed.     Standalone  mode   Single  Data  Collector  process  runs  the  pipeline.  A  pipeline  runs   in  standalone  mode  by  default.  
  33. 33. MANTL Data Platform Overview A  modern,  baneries  included  pla6orm  for  rapidly  deploying  globally  distributed  services.   Mantl’s  goal  is  to  provide  a  fully  func?onal,  instrumented,  and  portable  container  based  PaaS  for  your   business  at  the  push  of  a  bunon   1)  Easy  deployment  and  configura?on  on  different   pla6orms   2)  High  availability  and  self-­‐healing   3)  Mul?-­‐datacenter  support   4)  Linear  scalability   5)  Smart  resource  management   6)  Wide  range  of  supported  frameworks  
  34. 34. MANTL nodes Consul  for  service  discovery   Mesos  cluster  manager   Marathon  for  cluster  management     Docker  container  run?me   Zookeeper  for  configura?on   management   Docker  containers   Any  Mesos-­‐based  workloads   Traefik  for  proxying  external  traffic     into  services  running  in  the  cluster  
  35. 35. Security  &   Opera.on   Frameworks   PlaCorm   Support   Mantl Components             Core   Components   ➢  Data  Storage  -­‐  Riak,  Cassandra,  HDFS   ➢  Data  processing  -­‐  Spark   ➢  Security  -­‐  Vault   ➢  Data  inges?on  –  KaYa     ➢  Metrics  collec?on  -­‐  Collectd   ➢  Logs  forwarding  -­‐  Logstash   ➢  Provisioning  -­‐  Terraform,  Ansible   ➢  Cluster  management  –  Mesos,  Marathon     ➢  Service  discovery  and  configura?on  management  -­‐  Consul,  Zookeeper,   Traefik   ➢  Container  run?me  -­‐  Docker   ➢  Cisco  Cloud  Services,  Cisco  MetaCloud   ➢  Amazon  Web  Services   ➢  Google  Compute  Engine   ➢  Openstack   ➢  DigitalOcean   ➢  Bare  Metal   ➢  Autoscaling  and  high  availability   ➢  Applica?on  load  balancer   ➢  Applica?on  dynamic  firewall   ➢  Manage  Linux  user  accounts   ➢  Authen?ca?on  and  authoriza?on   for  Consul,  Mesos,  Marathon  
  36. 36. Long  Running  Services   Big  Data  Processing   Batch  Scheduling   Supported Mesos Frameworks Data  Storage   Mesos  makes  it  easy  to  develop  distributed  systems  by  providing  high-­‐level  building  blocks.    
  37. 37. ANALYTICS  PLATFORM  MANAGEMENT   Data  Inges?on   • KaUa,  Streamsets  configurators   Data  Storage   Riak,  Cassandra,  HDFS   Model  DevOps  Machine  learning   MLLib,  Spark   Model  Deployment   • Model  loading,  versioning   Cluster  Management  &  Scheduling   Cluster  manager   Mesos   Cluster  Management    long  running   service   Marathon   Service  Discovery   Consul   Distributed  Virtual  network   Calico  ETCD   ADVANCED  ANALYTICS  APPS   Analy?cs  Accelerators  as  Apps   • Forecas?ng,  NLP,  op?miza?on,  enrichment  etc.   SPECIALIZED  ADVANCED  ANALYTICS  MODELS   Consul?ng  Services   Design,  Build,  Deploy   Maintain,  Manage  Performance   DASHBOARDS   ZoomData   Tableau,  Qlik,  Spo6ire,   Excel/BI  Cubes  …   BUSINESS  APPS   Custom  ZoomData  Visualiza?ons   (D3)   Custom  Applica?ons   Customer  System  Integra?on   CUSTOMIZATION  &   MANAGED  SERVICES   CISCO  INTERCLOUD   Customization MANTL Data Platform
  38. 38. Sample Architecture for Batch Data Processing Cassandra Elastic Search Spark Spark Mllib Riak Kibana Dashboard VisualisationStorage Stream Sets I/P in multiple formats Text, logs and json from various storage source. Spark application process data and store to elastic search or Cassandra or Riak storage for visualization else it stores in HDFS Machine learning algorithm for data science application Zoomdata Data Discovery D3 Web Application HDFS StreamSets Data Collector runs as an application in Spark Streaming to pull data from origin to spark CSV, Tab delimited etc. LOG file JSON TEXT
  39. 39. Sample Architecture for Data Streaming Kafka Cassandra Elastic Search Spark Spark Mllib Riak Kibana Dashboard VisualisationStorage Stream Sets Streaming network from different sources Kafka is used for collecting streaming data and data is consumed through consumer API by Streamset for further processing. Spark application process data and store to elastic search or Cassandra or Riak storage. Machine learning algorithm for data science application Zoomdata Data Discovery D3 Web Application
  40. 40. Use Case 1 - Shipped Analytics Collect log metric from cluster to analyze and drive Alert/Recommend engine •  Alert Engine - produces alert messages on a basis of some conditions. •  Trend Engine - produces trend messages related to data aggregation. •  Policy Engine – derives from Alert and Trend Engines produces policy messages which contain recommendations.
  41. 41. Use Case 1 - Shipped Analytics Architecture Central  Cluster   Probe   Probe   Probe  
  42. 42. DataCollector   DataCollector   DataCollector   node node node node node Use Case 1 - Shipped Analytics Data Flow
  43. 43. •  Iden?fy  the  top  technology  trends  by  analyzing  public  data  and  open   source  projects   •  Use  machine  learning  to  process  a  wide  range  of  public  data  available   on  the  world  wide  web  and  iden?fy  high  poten?al  emerging   technologies     •  Publish  results  to  a  web-­‐based  dashboards  and  refresh  results  regularly     Use case 2 - Emerging Top technology using Public data
  44. 44. Use Case 2 - Analysis Through Public data
  45. 45. Use Case 2 - Dataflow APIs   RSS  feeds   Scraping   Numeric   Network  Data   Text  data  (ar?cles,   blogs)   Staging   tables   Interac.ve  D3   Dashboards   Websites   Data Sources D a t a Extractors Data Storage Data Processing Machine Learning Visualization Below we used the framework to execute the project in CIS Data Platform
  46. 46. Lambda Reference Architecture Monitoring  /  Analy?cs  Cluster  (local,  Texas-­‐3)    Global  Monitoring  /  Analy?cs  Cluster  (global,  Texas-­‐1)   Monitoring  /  Analy?cs  Cluster  (local,  Ams.  -­‐1  )   Monitoring  /  Analy?cs  Cluster  (local,  Lon.-­‐1)   Local  components  and  deployment  is  the  same  as  global,  just  smaller     Real-­‐.me  and  batch  processing  (Lambda),  anomaly  detec.on,  visualiza.on       SSL   KaUa   SSL   SSL   MQTT  
  47. 47. MANTL Data Platform in Practice: putting it all together Working  on  advanced  enabling  technology  –  Mesos,  K8S,   Orchestra?on     Working  on  developing  individual  components  –  dev  &  co-­‐dev:   Zoomdata,  Riak,  Streamsets,  etc.     PuMng  together  reference  architectures  and  real  solu?ons  to   test  and  further  develop  the  technology     Provide  innova?on  and  advanced  services  to  customers     Pla6orm  to  develop  and  deliver  Microservices  and  Data   applica?ons      
  48. 48. Q/A
  49. 49. Next steps Con?nue  partnerships  and  co-­‐devlopment  efforts  with  industry   leaders  to  deliver  innova?on     Con?nue  applying  new  developed  technology  to  real  use  cases   and  PoC  with  customers  and  partners     Con?nue  working  closely  with  A&E  and  Product  teams  on   produc?za?on  roadmap     Work  with  A&E  team  closely  on  priori?za?on  of  our  R&D   ac?vi?es  to  stay  closely  aligned        
  50. 50. Anatomy of the service/framework Elas?csearch  is  a  highly  scalable  open-­‐source  full-­‐text  search  and  analy?cs   engine     Allows  to  store,  search,  and  analyse  big  volumes  of  data  quickly  and  in  near   real  ?me     Underlying  technology  in  applica?on  to  Op?mize  complex  search  in  Big  data     We  are  working  together  with  Elas?c  developing  and  tes?ng  their  UTILIZING   Mesos  cluster  to  run  Elas?csearch  
  51. 51. Elas?csearch  on  Mesos  Cluster        Elas?c  framework  scheduler   Marathon  framework  scheduler   Chronos  framework  scheduler       Zookeper Chronos  Executor   Marathon  Executor   HA  Proxy  node       Step  1:  Mesos  Cluster  with  Marathon    &   Chronos  running     Step  2:Elas?c  framework  installa?on  on   MESOS  Master  with  a  configured  #  of  mesos   slaves  to  be  launched     Step  3:  Deploys  the  ES  executore  in  MESOS   slaves     Step  4:    ES  nodes  discovery  and  Zookeper   pugin  in  ES  nodes     Step  5  Using  plugin  nodes  find  each  other   and  search  is  op?mized  at  cluster  level     Elas?csearch   executor  &   Zookeper  pugin  
  52. 52. MANTL Architecture – Datacenters Control  nodes  manage  the  cluster  and  resource  nodes.   Containers   automa?cally   register   themselves   into   DNS   so  that  other  services  can  locate  them.   Once  WAN  joining  is  configured,  each  cluster  can  locate   services  in  other  data  centres  via  DNS  or  Consul  API   Single  Datacentre   Mul?ple  Datacentre  
  53. 53. Client Client Client RPC over DNSmask RPC over DNSmask LAN gossip over DNSmask Server Server (Leader) Server replication replication Lead forwarding Internet Server Server (Leader) Server replication replication Lead forwarding Datacenter 1 Datacenter 2 Remote DC forwarding WAN gossip TCP&UDP Consul ➢  Service discovery ➢  Client health-checking ➢  Key-value store for configurations ➢  Multi-datacenter support
  54. 54. Mesos features Mesos  makes  it  easy  to  develop  distributed  systems  by  providing  high-­‐level  building  blocks.     ➢  Scalability ➢  Fault-tolerance and self-healing ➢  Resource isolation ➢  Fine Grained resource elasticity
  55. 55. Mesos architecture
  56. 56. Mesos setup for developing application
  57. 57. ZK ZK ZK Zookeeper quorum JN JN JN Shared edits DataNode DataNode Active NameNode Zookeeper Failover Controller Active NameNode Zookeeper Failover Controller DataNode Heartbeat Heartbeat Write Read Active NN state monitoring Standby NN state monitoring Monitor and maintain active lock Monitor and try to take active lock ➢  Used  to  store  and  distribute   data  accross  a  cluster   ➢  Is  a  base  for  batch  analy?c   processing   ➢  Is  highly  available  and  fault   tolerant   ➢  Automa?cally  scaled  and  self-­‐ healing  with  Mesos   HDFS framework
  58. 58. *  hnps://github.com/datastax/spark-­‐cassandra-­‐connector   Mesos Framework for Spark and Cassandra
  59. 59. * Smart broker.id assignment * Preservation of broker placement * Rolling restarts * Easy cluster scale-up Mesos framework for Kafka
  60. 60. Ø  Fault  tolerant  job  scheduler   Ø  handles  dependencies  and  ISO8601  based   schedules   Ø  Flexible  Job  Scheduling   Ø  Supports  arbitrarily  long  dependency  chain   Ø  supports  the  defini?on  of  jobs  triggered  by   the  comple?on  of  other  jobs   Mesos framework for Chronos
  61. 61. How MANTL Data Platform for business application •  Cisco Data Platform can be used to build custom applications or service for various analysis and Data analytics initiative. •  Companies can streamline Data ingestion, process, manipulate , analyse and visualize data all in single Infrastructure
  62. 62. Yali Load Testing Framework Yali Elas?csearch   KaUa   Cassandra   HDFS   Plugins Kafka Cassandra HDFS Storage Elasticsearch Generate data to load test storage
  63. 63. Elasticsearch Plugin Testing Results Job  Host   config     Elas.csearch   config   Execu.on   threads   Batches   Records/ batch/thread   Average   response   from  ES,  s   Records/s   Record  size,   b   Records   generated  *   10^6   Execu.on   .me,  min   Win7,  4   cpu,  16  ram   Cluster:  CentOs  6.7,   Elas?csearch  2.1.1,   VPN  network,  2   master(4  cpu,  16   ram),  15  worker   nodes(8  cpu,  32  ram)   12   60   50000   78   6804   280   36   84   Local:  CentOs  6.4,   Elas?csearch  2.1.1,   VMware  virtual   network,  single  node   (2  Core  cpu,  8  Gb   ram)     4   60   10000   1,6   14768   280   2,4   2,5   Records