Bio:
Jeff Moore is a Systems Engineer at Bandwidth with a focus on administering and extending Kubernetes, AWS, and the Elastic Stack to support internal customer use-cases. Originally a die-hard Kubernetes fan, his passions have now also extended to the Elastic Stack due to insights of the amazing things that can be done with well-architected data. He has lived in Raleigh for most of his life and currently has a love/hate relationship with the factory automation game Factorio. Jeff is a Certified Kubernetes Administrator with plans to complete the Elastic Certified Engineer exam.
Abstract:
In this talk, Jeff will talk about how he designed a command-line tool built on the go-elasticsearch project - making interactions with the elasticsearch APIs much easier. He will also go into the inspirations of the project, lessons learned, and future work.
Presentation on how to chat with PDF using ChatGPT code interpreter
Esctl in action elastic user group presentation aug 25 2020
1. 1
Jeff Moore
08/25/20, Systems Engineer, Bandwidth
esctl in Action
or How I Built a CLI tool
to Interact with
Elasticsearch
2. 2
About this presentation
It was fairly difficult deciding between catering towards Devs or Ops. In the
end, I decided to side with the Ops side of the house. Based on audience
request, I will go into the code in depth - or that can be the topic of a later talk.
Feel free to stop me at any point* so we can discuss this utility - however,
suggestions are encouraged to be held until the end of the slide deck
4. 4
Hopeful Takeaways
- An extensible CLI tool that may impact both
yourself and your Org
- Lessons learned from over a year of
becoming a stronger Golang developer
5. 5
Caveats
- This is not complete.
- Work has been focused on getting good boilerplate
- Not all endpoints have been implemented (templates, indices, etc)
- This is not intended to replace the SDKs
7. 77
We needed an easier way to
interact with Elasticsearch as
operators of the cluster
8. 8
Intent of esctl
1. Standard endpoints are mocked, with basic config options exposed as
flags
2. Extra commands are added based on functions that could be necessary
(i.e admin, util)
13. 13
escli vs esctl
Comparison of metrics
Metric escli esctl
Language Bash Golang
Function implementation Bash functions Subcommands
# of commands (Aug
2020)
117 44
New functions Easy to add Tedious
Extensibility Difficult Easy
Refactorability Difficult Not as difficult
Barrier to entry Easy Difficult
14. 14
Getting Started Demo
Overview of Demo
Environment
1 2 3 4
Overview of local
cluster init script
Overview of
Kubernetes
manifests
Demo of
commands and
features
17. 17
On Command Structure
- Elasticsearch api structure is very different to that of Kubernetes
- Format in v0.1.4 is esctl <endpoint/resource> <function>
- Format in Kubectl is kubectl <function> <endpoint/resource>
18. 18
Command Structure Comparison pt. 1
kubectl vs. esctl - Similarities
CLI Tool Command Function Resource
kubectl kubectl get nodes Get Nodes
esctl esctl cat nodes Get Nodes
19. 19
Command Structure Comparison pt. 2
kubectl vs. esctl - Differences
CLI Tool Command Function Resource
kubectl kubectl get ns Get Namespaces
esctl esctl index get Get Indices
(Yes, I am comparing namespaces to
shards. Feel free to discuss offline)
20. 20
Command Structure Comparison pt. 3
kubectl vs. esctl - the Justification
The endpoint behind commands like
esctl cat nodes is /_cat/nodes. By
mocking functions after their API
syntax, there is no mental translation
tax needed to move to/from esctl and
the REST API
21. 21
Proudly Built on go-elasticsearch SDK
Elasticsearch SDK, mostly written by Karel Minarik from Elastic
Defines api endpoint options as struct fields
Allows client initialization
Located at https://github.com/elastic/go-elasticsearch
23. 23
esctl Extensibility Examples - Showing Inactive Watchers
Problem: Watchers not currently active (read: disabled) need to be listed
Solution:
- Get all watchers (SearchRequest on .watches index)
- Filter to desired activity state
- Return output (esutil helper function)
24. 24
esctl Extensibility Examples - Getting Search Metrics
Problem: User queries need to be validated before a DeleteByQuery is run
Solution:
• Take a query
• Merge a known aggregation definition to that query
• Execute that query (SearchRequest)
• Return output (esutil helper function)
25. 25
esctl Extensibility Examples - Removing Empty Indices
Problem: Empty indices created by ILM policies cause disk utilization
inbalances
Solution:
- Get a list of all indices (IndicesGetRequest)
- Filter to indices containing 0 documents
- Delete those indices (IndicesDeleteRequest)
27. 27
Lessons Learned
- Writing a CLI tool isn’t easy
- Design should come before writing code to avoid a large number of
refactors
- Reading documentation thoroughly before using imported code is
extremely valuable
- This CLI tool continues to be a great learning opportunity for me
28. 28
Overall, building this CLI tool was
a painful experience; but, if it was
painless, I would be a Java dev
instead of a Golang dev
Me, just now
