Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and Evans Ye
âąTransferir como PPTX, PDFâą
9 gostaramâą5,911 visualizaçÔes
The SDACK architecture stands for Spark, Docker, Akka, Cassandra, and Kafka. At TrendMicro, we adopted the SDACK architecture to implement a security event inspection platform for APT attack analysis. In this talk, we will introduce SDACK stack with Spark lambda architecture, Akka and Kafka for streaming data pipeline, Cassandra for time series data, and Docker for microservices. Specifically, we will show you how we Dockerize each SDACK component to facilitate the RD team of algorithms development, help the QA team test the product easily, and use the Docker as a Service strategy to ship our products to customers. Next, we will show you how we monitor each Docker container and adjust the resource usage based on monitoring metrics. And then, we will share our Docker security policy which ensures our products are safety before shipping to customers. After that, we'll show you how we develop an all-in-one Docker based data product and scale it out to multi-host Docker cluster to solve the big data problem. Finally, we will share some challenges we faced during the product development and some lesson learned.
Recomendados
Recomendados
Mais conteĂșdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and Evans Ye
Semelhante a Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and Evans Ye (20)
Mais de Docker, Inc.
Mais de Docker, Inc. (20)
Ăltimo
Ăltimo (20)
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and Evans Ye
- 1. Using the SDACK Architecture on Security Event Inspection Darren Chen Evans Ye Sr. Software Engineer @ Trend Micro Sr. Software Engineer @ Trend Micro 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 2. About Darren âą Darren Chen (Yu-Lun Chen) âą Sr. Software Engineer @ Trend Micro âą Enthusiast in big data and cloud computing technologies âą Docker experience â 1.5 years 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 3. About Evans âą Evans Ye (Yu-Hsin Yeh) âą Sr. Software Engineer @ Trend Micro âą Apache Bigtop PMC member âą Develop big data apps & infra âą Docker experience â 2.5 years 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 4. How to make a software product ? 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 5. How to make a Dockerize software product ? 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 6. Before Motivation What is SDACK Agenda During Why Dockerize Security Monitor After Lessons Learned Conclusions Q&A 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 7. Motivation 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 8. Target Scenario 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 9. Problems ⹠Too many log to investigate ⹠Lack of actionable, prioritized recommendations 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 10. AD Windows Event DNS Proxy Web server âŠ.. Threat Analytic System 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 11. But we faced TwoproblemsâŠâŠ. 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 12. How to deal with Customersâ Private data ? 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Cloud On Premises
- 13. How to deal with Big Volume logs ? 2,000,000,000per day 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 14. We need to build an On-Premisesproduct which can deal with Big Data 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 15. How to deal with Big Data? 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 16. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Toolbox for building wide variety of big data product SDACK Architecture
- 17. What is SDACK 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 18. SDACK Source: http://www.slideshare.net/akirillov/data-processing-platforms-architectures-with-spark-mesos-akka-cassandra-and-kafka fast and general engine for large-scale data processing deployment and resource management toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications distributed, highly available database designed to handle large amounts of data across datacenters high-throughput, low-latency distributed pub-sub messaging system for real-time data feeds 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 19. Data Storage Data Analysis Data Preprocessing Data PipelinePackage 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 20. Threat Analytic System Architecture 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 21. Log API Server Web Server 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 22. 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 23. Medium-sized Enterprises 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 24. Large Enterprises 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 25. Fortune 500 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 26. With Docker ⹠Easy to scale ⹠Test once, run anywhere ⹠Widely supported by many platforms 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 27. Why Dockerize 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 28. Dockerize â Benefit 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Deploy Develop Test Scale
- 29. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Deploy Develop Test Scale Dockerize â Benefit 1
- 30. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. APIWeb Challenge ⹠Setup ⹠Operate ⹠Update
- 31. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Dockerize Software Technologies
- 32. Docker Compose for Operation 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Docker Compose kafka: build: . ports: - â9092:9092â spark: image: spark port: - â8080:8080â âŠâŠ
- 33. Docker Hub for Updating 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Docker Hub
- 34. Dockerize â Benefit 2 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Deploy Develop Test Scale
- 35. Benefit for Development âą Docker provides two benefits in our Spark jobs development â Reproducibility â Flexibility 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 36. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Reproducibility in Spark Streaming Job Development
- 37. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Dev Cluster Spark Streaming Job Development Data Streams
- 38. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Local Spark Streaming Job Development Data Streams Snapshot Data Set (Date : Jan. 04 ~ Jan. 08) Freq. : 1 min Batch size : 1000
- 39. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Local Spark Streaming Job Development Data Streams Snapshot Data Set (Date : Jan. 04 ~ Jan. 08) Freq. : 1 min Batch size : 1000 Freq. : 0.5 min Batch size : 5000 Freq. : 1 min Batch size : 50000 1 2 3
- 40. Quick Development Iteration Local LocalData Streams Snapshot Data Set 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Local Deploy Test Destroy Modify Job Job
- 41. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Flexibility in Hybrid Architecture
- 42. Data Research in Dev Cluster 2016 DockerCon | Copyright© 2016 Trend Micro Inc.2016 DockerCon | Copyright© 2016 Trend Micro Inc. Dev ClusterData scientists submit spark jobs Job
- 43. Data Research in Dev Cluster 2016 DockerCon | Copyright© 2016 Trend Micro Inc.2016 DockerCon | Copyright© 2016 Trend Micro Inc. Dev Cluster Job Result Data scientists submit spark jobs
- 44. Data Research in Dev Cluster 2016 DockerCon | Copyright© 2016 Trend Micro Inc.2016 DockerCon | Copyright© 2016 Trend Micro Inc. Dev ClusterData scientists submit spark jobs
- 45. Data Research in Dev Cluster 2016 DockerCon | Copyright© 2016 Trend Micro Inc.2016 DockerCon | Copyright© 2016 Trend Micro Inc. Dev Cluster Job Other members submit spark jobs
- 46. Data Research in Dev Cluster 2016 DockerCon | Copyright© 2016 Trend Micro Inc.2016 DockerCon | Copyright© 2016 Trend Micro Inc. Dev Cluster Job Wrong Result Other members submit spark jobs
- 47. Hybrid Architecture 2016 DockerCon | Copyright© 2016 Trend Micro Inc.2016 DockerCon | Copyright© 2016 Trend Micro Inc. Dev ClusterSubmit Spark Job Job Result Local
- 48. Whatâs More 2016 DockerCon | Copyright© 2016 Trend Micro Inc.2016 DockerCon | Copyright© 2016 Trend Micro Inc. Dev Cluster Web Service Development Local
- 49. Dockerize â Benefit 3 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Deploy Develop Test Scale
- 50. ⹠Test case 1 ⹠sub-test 1a ⹠sub-test 1b ⹠Test case 2 ⹠sub-test 2a ⹠sub-test 2b ⹠Test case n ⹠sub-test na ⹠sub-test nb 2016 DockerCon | Copyright© 2016 Trend Micro Inc. ⊠Clean & Consistent Environment
- 51. Dockerize â Benefit 4 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Deploy Develop Test Scale
- 52. Distributed Software Components 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 53. Akka ⹠High performance concurrency framework ⹠Clustering mechanism available ⹠Leverage on Akka, we build up our Akka cluster system 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 54. Our Akka Cluster System 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Client Master LDAP Server 1 2 3 4 Query account information Send the job Query LDAP ServerReturn the result LDAP Service
- 55. Our Akka Cluster System 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Master LDAP Host Name DB Data ProcessEndpoint JobJobJob
- 56. Dockerize for Each Micro-service 2016 DockerCon | Copyright© 2016 Trend Micro Inc. LDAP DB Data Process Endpoint Host Name Master
- 57. Dockerize for Scale Out 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Data Process Host Name DB LDAP Endpoint Data Process Data Process
- 58. Security 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 59. Docker Vulnerabilities since 1st release 2016 DockerCon | Copyright© 2016 Trend Micro Inc. The only high severity vulnerability was fixed within 2 days.
- 60. Misconfiguration 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Open it without ACL ?
- 61. Open Docker Registry 2016 DockerCon | Copyright© 2016 Trend Micro Inc. AU BE CA CN DE FI FR GB HK HR IE IR IT JP KR NL PL RU SE SG TW US ZA 0 10 20 30 40 50 60 70 80 90 Open Docker Registry w/o Access Control
- 62. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Some tools can make your Dockerize product more secure
- 63. Docker Bench for Security âą Check â Host configuration â Docker daemon configuration â Docker daemon configuration files â Container images and build files â Container runtime 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 64. CoreOS Clair âą Static analysis of vulnerabilities â Debian security bug tracker â Ubuntu CVE tracker â Red Hat security data 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 65. Docker Cloud 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 66. Monitor 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 67. Monitor stack 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Grafana CPU, Memory, Network Metrics
- 68. Monitor stack 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Grafana Metrics APP Metrics
- 69. Issue on cAdvisor âą cAdvisor can not send network usage correctly to InfuxDB â when the container use host network on a multiple network cards machine âą Use Telegraf to fix this problem 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 70. Before Motivation What is SDACK Agenda During Why Dockerize Security Monitor After Lessons Learned Conclusions Q&A 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 71. Lessons Learned 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 72. Lessons Learned âą Mount the stuff you may change it frequently to your Docker containers â For example, on PoC, mount your configuration files into Docker containers directly 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 73. On PoC 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Change Settings Re-build Images Deploy
- 74. Mount configuration files 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Host machine Conf Kafka container Conf Conf Spark container Conf Conf Conf Conf Conf Conf Kafka Configurations Conf Conf Conf Spark Configurations
- 75. Conclusions 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 76. Summary 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Dockerize ⹠Deploy ⹠Develop ⹠Test ⹠Scale Security ⹠Misconfiguration ⹠Docker Bench ⹠CoreOS Clair ⹠Docker Cloud Monitor ⹠Visibility ⹠cAdvisor ⹠InfluxDB ⹠Grafana for Security
- 77. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. We Need To build an On-Premises product which can deal with Big Data In the beginning âŠ
- 78. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. We Need To build an On-Premises product which can deal with Big Data Have Now Build Ship Run Conclusions
- 79. 2016 DockerCon | Copyright© 2016 Trend Micro Inc. Go ahead Dockerize your product
- 80. Thank you! 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 81. Q & A 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
- 82. Thank you! 2016 DockerCon | Copyright© 2016 Trend Micro Inc.
Notas do Editor
- Hi everyone, thank you all for being here I am very happy and thanks DockerCon gives me the chance to share you our ideas and what we have done Our topic today is using the SDACK architecture on security event inspection And We are from Trend Micro. Trend Micro is a IT security company, we develop innovative security solutions that make the world safe for businesses and consumers to exchange digital information
- Ok this is me. I am Darren chen, a software engineer at Trend Micro My experience in docker is more than one (about one and a half) years and I also interested in big data stack, such like spark kafka and hadoop
- And by my side is my colleague Evans Ye. He is also a software engineer at Trend Micro And now he is apache Bigtop project PMC member His docker experience is more than two years (two and a half years) Before we start the topic today, I would like to ask some questions.
- If you have experience on making a software product, please raise your hand ? [äșșć€æ] Great, looks like a lot of people have experience on making a software product. But thatâs not the topic we are going to talk about today. [äșșć°æ] Oh~, there are a few people know about how to make a software product. But thatâs OK. It doesnât matter. Because we will not discus it today
- Today we will focus on how to make a Dockerize software product We will share our experience on using Docker in our security inspection platform
- This is our agenda today. We separate the presentation into three parts. First, before we make a product, we must have a problem need to solve. So we will explain our motivation about our product and talk about what is S.D.A.C.K. SDACK and how we use SDACK to address the problem we faced. In the Second part, we will introduce how Docker makes our product development progress more efficient. And we also bring some knowledge about security and monitor of dockerize product. Finally, we will talk about our experience on using Docker and make a conclusion. Our presentation will take 35 minutes. And Iâll be happy to answer all of your questions in the last Q&A section. Ok. Iâd like to hand this presentation over to my colleague. Letâs welcome Evans Ye to talk about the background of our product.
- Whatâs the problem weâre trying to solve? To be more specifically, whatâs our target scenario? In most of the enterprises, they have InfoSec teams to oversight cyber security events happened inside the companies. And typically theyâll have Security Information and Event Management platform, which is so called SIEM, to collect large amount of logs for them to do further investigations. Spunk and ArcSight are two solutions for that. Whatâs the problem weâre trying to solve? To be more specifically, whatâs our target scenario? In most of the enterprises, they have InfoSec teams to oversight cyber security events happened inside the companies. And typically theyâll have Security Information and Event Management platform, which is so called SIEM, to collect large amount of logs for them to do further investigations.
- But, the problem is thereâre just too many log to investigate. And those SIEM platfroms is lack of actionable, prioritized recommendations
- So, weâd like to build a threat analytic system that has security intelligence built inside. It can collect large amount of log and do prioritization, filtering and anomaly detection. With that, we can only output valuable information to the infoSec team, which can significantly reduce the load for them. The systemâs goal is to ease infoSec peopleâs life and help them quickly respond to high priority threats.
- The first problem is how do we deal with customerâs private data? Apparently, customers many have concern if we put them in the cloud because thereâre too many PII data. So for this system, we design it to be an on-premises solution. The first problem is how do we deal with customerâs private data? Apparently, put them in the cloud is not a good idea because customers wonât like to see that happened. Thereâre too many PII in there logs. So for this system, we design it to be an on-premises solution.
- The second problem is how do we deal with big volume? For example, one of our customer has 2 billion of log per day that should be consumed and handled properly in our system.
- Therefore
- Here comes the SDACK architecture Itâs a toolbox for building wide variety of big data products.
- So what exactly is SDACK? Let me describe it with more details
- SDACK, S, D, A, C, K, stands for Spark, Docker, Akka, Cassandra, and Kafka. Spark is a fast and general purpose engine for large-scale data processing scenarios. It supports traditional batch processing as well as micro batch streaming. Docker, which you should be already familiar about it, is a great tool for shipping software, doing deployment, and it also has the ability to do resource management. Akka is a toolkit and runtime for building highly concurrent distributed and resilient message-driven applications. To me any business logic related code youâre going to write, you can use Akka to develop it. Cassandra is a distributed, highly available database designed to handle large amount of data across datacenters. The nice thing about it is because of the masterless clustering mechanism, Itâs very stable. We donât need to pay much attention on it when running in customerâs environment. Kafka is a high-throughput low-latency distributed pub-sub messaging system for dealing with real-time data feeds. Itâs a best choice to handle streaming data pipelines when building a big data product. S, D, A, C, K, SDACK, stands for Spark, Docker, Cassandra, and Kafka. Spark is a fast and general purpose engine for large-scale data processing scenarios. It supports micro batch streaming as well as traditional batch processing Docker, which you should be already familiar about it, is a great tool for shipping software, doing deployment, and it also has the ability to do resource management. Akka is a toolkit and runtime for building highly concurrent distributed and resilient message-driven applications. To me any business logic related code youâre going to write, you can use Akka. Cassandra is a distributed, highly available database designed to handle large amount of data across datacenters. The nice thing about it is because of the masterless clustering mechanism, Itâs very stable. We donât need to pay much attention on it when running in customerâs environment. Kafka is a high-throughput low-latency distributed pub-sub messaging system for real-time data feeds. Itâs a best choice to handle streaming data pipelines when building a big data product.
- Briefly speaking, we use docker for packaging.
- Now with the SDACK architecture, how exactly the system architecture will be looked like?
- This is our system architecture, starting from the top-left corner, we collect logs using some well-know tools and feeds them into Fluentd, which is an universal interface to receive arbitrary type of log. Fluentd then passes the log down to Kafka immediately to store the data on to the disk. Then weâll have Akka to do the log transformation. The processed log will be stored into Kafka and Cassandra. Spark them fetches data from Kafka or Cassandra and produce valuable insights into a traditional relational database. An API server works together with web portal and database to serve the investigation need to the infoSec team.
- Since weâve adopted the SDACK architecture, we can containerize every software technologies we used using the micro service strategy. The deployment and management of the system become super easy!
- Now, by adopting the SDACK architecture, every micro services can be scalable. we can build a product that is suitable for medium-sized enterprises
- As well as large enterprieses
- For those very large companies, our product can also be deployed on an existing IaaS or PaaS services such as Mesos, or kubernaties With that, Iâve finished my section talking about the background and the architecture of our system. Next Iâll hand over to Darren, whoâll describe you our system with more details.
- To conclude my part, with docker our system is easy to scale, can test once run anywhere, and is widely supported by many platforms.
- [éŁæ„evanséšćèźæŽé«æ¶æ§æŽćźæŽ] Ok, thanks Evans for talking about our system background And next we will move on why we need to Dockerize our system.
- In fact, we got a lot of advantages in our product development after we ado(a)pt docker So the following I will use four phases deploy, develop, test, and sale to explain how will Docker benefit our product
- Ok, letâs get started with the first one, deploy
- As mentioned before, our architecture is complex. There are many components in our system. If you want to setup this kind of environment. You will waste a lot of time on installing components such like kafka, spark, Cassandra and after installed you also need to configure each component. So the long process of installation makes developers and users suffered In addition, such a complex system is very hard to operate and update
- However, when we dockerize each component, previous problems are gone Each component can be properly installed and configured into an image It will reduce a lot of time on installation and letâs be easier to setup our threat analytic system
- We can illustrate our threat analytic system layout by a easy to read yaml file And use docker compose to operate our system in one command.
- And leverage on Docker Hub, we can quickly and easily update customer side threat analytic systemâs component For example , as trend micor (we) publish the new version of apiserver to Docker Hub(from trend micro), our customers can pull and update the latest apiserver directly (without complicated updating process)
- Since we dockerized our system, the deployment becomes easier And it will help our team members to develop their applications or algorithms in a more efficient way. So, next, letâs look at how will Docker benefit our development process
- Docker provides two key features to make our daily development become more efficient One is reproducibility Another is Flexibility In the following, I will introduce more details
- First, I will explain reproducibility in spark streaming job development
- Internal beta environment We have setup a dev cluster which provide our team members an environment to test their algorithms. In the dev environment, the data is streaming from our company data source and it never stop.[æȘæȘç] Now if you run your spark streaming job in the dev cluster and after a while you find your job failed(æȘæȘç), it will be hard to troubleshoot . Because the data is streaming, you can not reproduce the problem again.
- However, this problem could be solved by Docker. Because developer can easily setup the threat analytic system in their local machine. In addition, they can use (leverage on) the same snapshot data set, so they test their algorithms locally, then when their job fail, they will be able to reproduce the environment quickly and troubleshot their algorithms easily
- More over, they can control the speed and the amount of steaming in, so they will be able to simulate multiple scenarios in their local machine to improve the quality of their algorithms For example, we can increase our data input rate to test if our algorithms can deal with them in time or not
- So when we do the spark streaming job development, we can quick deploy the threat analytic system in our local environment(the new local environment) and test our job with the same snapshot data set. Once we found our jobs failed(some problems in our algorithms), we can quickly reconstruct another new local environment for testing after we fix the problem(issue). Therefore, leverage on Docker, we can reproduce the problem easily and speed up the development iteration.
- Next, I will explain how will Docker Flexibility help us achieve hybrid architecture and make the Data research more flexible.
- Let me talk about the background first. In our team, there are some Data Scientists doing data research on our dev cluster, because they need the real data But the result of their algorithms may not be stable enough during PoC stage So it might pollute our dev cluster database content, then affect other team members' job accuracy For example, when a data scientist submits a spark job into our dev cluster
- The result will be stored into our database
- Once the result is incorrect. It will pollute our dev cluster database
- After that, if someone submits another spark job and use the database content as data source
- He or she will base on polluted database content and generate the wrong result.
- Leverage on Docker, we can construct more flexible architecture to solve the previous problem. Our team member can setup their own threat analytic system locally. And then, they can still submit their spark jobs to dev cluster and utilize its computing power and real data, ăæȘæȘçă Finally the result can be wrote(send push) back to their local system [directly]. It means that Developers can use real data and system computing power on dev cluster, but no need to worry about the(li) incorrect result will mess up the dev environment.
- Whatâs more, we can construct any architecture based on the same concept. Which means developer can have different combination with dev cluster and local environment. For example, the front end engineer can do web development in their local machine with our dev cluster apiserver and database content So they can test with real world data without maintenance effort on other components
- I have mentioned about the deployment and development, Next, letâs move on to the Testing part (can add something)
- [For QA team, there are two important things, The first is clean environment for testing The second is consistent with production environment.] By using Docker, we can meet these two requirement easily. Becasue we dockerize our system, so QA team can setup a brand new testing environment for each integration test quickly. And no need to worry about other possible environmental factors may impact on the testing result In addition, all the dependencies are wrapped into docker images, so we make sure each integration test environment is the same as production environment
- The last part of why dockerize section is scale. I will talk about how will Docker benefit on scalability
- First, In our threat analytic system, there are many components which are distributed software (and designed to solve the big data problem) such like Kafka, Spark and Cassandra. [Using docker, we can quickly compose cluster with these components. Because all the settings are packaged into docker images.]
- Another case is we leverage on akka toolkits to achieve (other) micro-service scalability Akka is a high performance concurrency framework which help us to construct a distributed cluster easily.
- Letâs take a look on the work flow of our akka cluster system In our akka cluster system we adopt master slave architecture. //All queries will be sent to master and it will dispatch jobs to service For example, the client send a request to master for querying about account information, then master will dispatch this job to LDAP service and the LDAP service will query LDAP server to get the account information. After that, the LDAP service will send the result back to the client.
- In our akka cluster system, we have built up a lot of micro-service just like LDAP service. Each micro-service has its own tasks So when different kind of jobs are sent to the master, master will dispatch each job to corresponding service
- we dockerie each akk micro-service into a docker image. It will make us easy to setup and operate our akka cluster system
- In addition, when some services hit their capacity limit, we can scale it out easily (sca li out) For example, In our akka cluster system, we have a Data Process service to normalize our input logs Then, if there are too many logs steaming in to our system, the data process service will not be able to deal with them in time. We can scale out the data process service quickly to solve the problem
- I have introduced how Docker benefit our product development (threat analytic system), But donât forget when you ship your dockerize product to your customer, you need to take care about security. So, next I will show you our experience on Docker security
- As our survey, from docker first release to now, there are a few vulnerabilities in Docker. So far, the most dangerous vulnerability was fixed in 2 days. This means that there are not many security issues in docker itself. So what will make your dockerized product become insecure?
- It is misconfiguration This is an example, when you set up swam cluster in your environment, you may need to open a network port of your docker daemon. But do you know, this action may allow worldwide to access (let whole world be accessible) your docker engine?? Because a lot of people just forget about the network ACL setting.ăæȘæȘçă Actually in the Docker website, they already reminded you to set your network security rules when setup the docker swam.
- And as trend Micro study, we also found that there are a lot of docker registry in the world can be access directly without any authentication Itâs easy and happy to use Docker but when you get into misconfiguration, it will make you become unhappy So how to prevent this disaster happened ?
- The following, I will introduce a couple of useful tools
- First one is Docker Bench for Security. It is a shell script, very easy to use It can check your host, docker daemon and container setting For example, It can check whether your linux kernel version is greater than the minimum requirement or not It can also check your docker configuration file and directories ACL setting In container runtime, it can check if you use privileged container or not
- Second tool is CoreOS Clair. It can do further analysis on your Docker images to check if any vulnerability package you have installed The protection ability is based on Debian, Ubuntu and Red Hat vulnerability database
- Actually, Docker has announced the new security service recently if you already use Docker cloud service, they will do the vulnerability scan for your docker images to make sure your images are secure enough
- Finally letâs talk about how to do monitor on Docker
- We adopt cAdvisor, influxDB and Grafana to compose our monitor stack cAdvisor is responsible to retrieve usage of CPU, memeory and network from containers and then store these metrics into influxDB. So grafana gets the metrics from influxDB and the user can use grafana web portal to check each container system status [æȘæȘç]
- The monitor stack is not only used in container system status but also applications We send Spark, Kafka and our application metrics into influxDB, so that, we can monitor many kind of application metrics, such like Kafka data input throughput, on the grafana web portal
- However, when your container use host network and you also have multiple network cards on your physical machine, cadvisor wonât be able to send the correct network usage to influxDB. Therefore, we use telegraf as a solution to this problem. Telegraf is an agent to collect metrics data and write them to influxDB Using telegraf, we can get all network card information from cAdvisor and send them to influxDB, so that, we wonât miss the correct one(modify)
- We have finished the Dockerize, security and monitor At the end, we will share some experience[æȘæȘç] and make a conclusion for this talk.
- Let me talk about the lessons learned during dockerize the threat analytic system
- Mount the stuff which you may change it frequently to your Docker containers For example, In the beginning of dockerize our threat analytic system, we found that we need to change configurations frequently My suggestion is at the beginning of dockerize progress, mount the whole configuration files into Docker container, then you can change and apply new setting efficiently without re-build your docker images. (ćŸæçšenv variable)
- In our use case, we use many big data open source components in our threat analytic system, such like Spark, And, in the Spark website, you can find that a lot of configurations could be adjusted based on the workloads. However, in the beginning, we still donât have the proper setting , so we need to change configurations frequently. Once any configuration changed, we need to re-build our Spark docker image and deploy it again
- Therefore, we can mount configuration files into docker containers to solve the previous problem We directly modify our configuration files on host machine and apply the change to our dockerize components So we are able to deploy different setting rapidly to speed up our development progress
- At the end, let's make a conclusion of our talk today.
- Summarize the main points again First, we used four phases deploy, develop, test, and sale to explain docker can make our product progress more efficient and speed up the development iteration (And so far, we use Docker in our big data solution stack and everything is work well.) Second, The misconfiguration problem may cause the security issue in Docker. You can use exist tools like âDocker Bench for Securityâ and CoreOS Clair to make your dockerize product more secure. More over, you can leverage on Docker Cloud service to secure your docker images.[æȘæȘç] Finally, the monitor stack cAdvisor, InfuxDB and Grafana can enhance your productâs visibility
- In the beginning we mention we would like to solve the problem we faced. We want to build an on premises product which can deal with big data.
- By leverage on docker major benefits Build ship run, we have now built an on premises product which can deal with big data. Docker makes the problem of dealing with big data become simpler and also let the procedure of composing an on premises product become more efficient.
- And enjoy your dockerize journey
- So thanks everyone. Thatâs it today Are there any questions Thatâs it guys. Thanks for your attention Questions time. Does any one have questions? Iâm going to finish here and thank you for your attention. If you have any questions, Iâll be happy to answer them.
- Thank you all. Have a nice day !