We will have a deep view of the Elastic Stack - which is the next evolution of ELK Stack, learn how to build the powerful log analysis system with Elastic Stack and have an overview of specifications and comparison details between the self-managed cluster vs Elastic stack provided as SaaS from cloud providers.

1. DEEP DIVE INTO ELASTICSEARCH
Establish A Powerful Log Analysis System
With Elastic Stack.
On Premises vs SaaS Elastic Stack
Comparisons.
Tyler
DevOps Engineer
NFQ Asia Company

2. Agenda
• Intro.
• Overview: Elastic Stack.
• Establish a powerful log analysis system with Elastic Stack.
• Elastic stack options from cloud providers.
• Which one would be fit for us?
• Cost Reflections.
• In conclusion.

3. About Me
• In tech for 7+ years.
• Technical Project Coordinator @ AVASO Technology Solutions.
• Infrastructure Technical Lead @ Betfair Group PLC.
• DevOps Engineer @ NFQ Asia.
• Member of Vietnam Elasticsearch Community.
• Bash/PowerShell languages.
• A dog parent :D

4. About NFQ Asia
• Member of NFQ Company.
• 15+ years’ experience in e-business
strategy and software development
• 300+ professionals.
• 4 countries: offices in Lithuania,
Germany, Vietnam, Singapore.
• Founded in Vietnam since 2015.
• Having organized 5 community
events/hackathons in Vietnam.

5. DATA
Cost
Operations
Features
Platforms
Plugins
Capability
Mapping
Processors
Aggregations
APIs
Monitoring
Security
Encryption
Supports
Backup
Database
Searching
Analytics
ComplexityArchitecture
APIsFlexibility
Availability
Compatibility
Centralization

6. Elasticsearch is everywhere

7. What is Elastic Stack?
• Formerly known as ELK Stack.
• ELK - The acronym for three open source
projects: elasticsearch, logstash, and kibana.
• Distributed, scalable, and highly available
(both on premises or SaaS).
• The Elastic Stack is the next evolution of ELK.
• Supports the lightweight Beats data shippers
from ES v2.1.1.

8. Elasticsearch
• “You know, for Search”
• Free, Open Source.
• Search engine based on Lucene.
• Near real-time searching, analytics and
visualization capabilities.
• Sophisticated Restful API.

9. Logstash
• Open source data collection engine that unifies
data from disparate sources, normalizes it and
distributes it.
• The ingestion workhorse for elasticsearch and
more.
• Real-time capabilities and pluggable pipeline
architecture.
• Community-extensible and developer-friendly
plugin ecosystem.

10. Kibana
• Open source analytics and visualization
platform designed to work with elasticsearch.
• Specialized for large volumes of streaming and
real-time data.
• No code, no additional infrastructure required.
• Easily and quickly understandable through
graphic representation.

11. Beats Platform
• “Data shippers” that are installed on servers
as agents.
• Either elasticsearch directly or through
logstash.
• Library written based on Golang.
• Supports create your own beat for specific
use cases.

12. ESTABLISH A POWERFUL LOG ANALYSIS
SYSTEM WITH ELASTIC STACK

13. Rationale
• What is log?
• How do we solve the production issue as usual?
• How much time do you spend investigating the
production issue?
• Where are the archived log?
• Visualization and dashboards?

14. The Challenge
How do you satisfy the search needs of the application system’s over 2,000 docs
per second while simultaneously providing tactical operational insights that help
both Development Team and Operation Team iteratively improve the customer
experience?

15. The Simple Log Analysis Diagram

16. Demonstration

17. Scalability Rationale
• High availability.
• Petabyte-scale data is written and/or read frequently.
• High scalability.
• Sufficient data allocation.
• Costs.

18. The Elasticsearch Hot-Warm Architecture

19. The Elasticsearch Hot-Warm Architecture (cont.)

20. ON PREMISES VS SaaS ELASTIC STACK
COMPARISONS

21. WHAT IS AWS ELASTICSEARCH
SERVICE?
• Managed service in AWS Cloud.
• Introduced in Oct 2015.
• Fully managed; Zero admin.
• Highly available and reliable.
• Built-in Kibana support.
• Integrated with other services in AWS ecosystem.

22. The AWS Integration

23. What is Elastic Cloud?
• Launched in Oct 2015.
• Provided by Elastic.
• High provisioning and scaling.
• Hosted in the Cloud Providers.
• Service-oriented architecture.
• Containerization using Docker.
• Fully supports custom plugins and API.

24. Elastic Cloud Architecture

25. HOW DO I KNOW WHICH ONE IS FIT
WITH ME?

26. Specifications comparison sheet
Self-managed Elastic Stack AWS Elasticsearch Service Elastic Cloud Enterprise
Pros
More options and features.
Complete control settings and
capacity.
Access to other APIs
Comprehensive ES monitoring
solutions.
Lowest costs.
SaaS.
Simplify the operations via APIs.
Security by IAM.
Automated snapshots*.
Encryption at rest.
Monitoring included*.
Technical supported.
SaaS.
Fully control through APIs.
Technical Supported.
Uptime SLA.
Feature-rich and complete
monitoring product.
Available on Marketplace.
Cons
Self maintenance.
Infrastructure matters.
No technical supported.
X-Pack limit features.
Limited control.
Less capacity and scalability.
Backup once time per day.
No plugins, no logs.
Medium expensive.
Only support I2 series EC2
instances.
Most expensive.
Imperfect for AWS-hosted
solutions.

28. Costs Comparison Chart
8,400.38 10,678.56 11,512.51
75,303.17
81,375.17
11,316.98 14,500.26
25,201.1525,201.15
32,035.68 34,537.54
203,318.55
219,712.95
28,319.95
38,295.63
50,402.30
0
50,000
100,000
150,000
200,000
250,000
Elastic Stack (AWS) Elastic Stack (GCP) AWS Elasticsearch Services Elastic Cloud (GCP) Elastic Cloud (AWS)
Cost($)
Service Models
One Year One Year (All Upfront) Three Years Three Years (All Upfront)
*Costs calculated based on 3TB-data cluster in multi-AZ in
Frankfurt region

29. In Conclusion
• Elasticsearch leverage the power of analysis ability for both Dev/Ops teams.
• Easily operate/maintain the huge cluster of servers and microservices.
• Choose the proper architecture depend on application/system.
• Estimate the budget to meet the requirements.
• Optimize the aggregation to adopt the resources.
• High availability oriented system.

30. We are hiring…
• Java Senior/Lead Developer
• PHP Senior Developer
• PHP Technical Lead
• Front-end Senior Developer
• Front-end Technical Lead
• Technical Project Manager
Simply send us an email with your enclosed
updated CV to: career@nfq.asia

31. Contact Me
LinkedIn: linkedin.com/in/tylernguyen91
Email: tai.nguyen@nfq.asia
Telegram: @tylern91