Strategies for Landing an Oracle DBA Job as a Fresher
How our ISP cost us a full day of the entire R&D team - Lior Redlus - DevOpsDays Tel Aviv 2017
1. How our ISP cost us a full day of
the entire R&D team
Lior Redlus
Co-founder and Chief Data Scientist
Coralogix
2. About Myself
• 32yr. Scientist at heart
• B.Sc and M.Sc in Neuroscience and Information Processing (BIU)
• Co-founder and Chief Data Scientist @ Coralogix
3. About Coralogix
• A Machine Learning powered scalable Log Analysis solution
• Log Management already included: indexing, querying, filtering,
alerting etc.
• Coralogix Analytics:
• Turns your data into patterns and flows
• Gives you deep insights on your system
• Automatically detects production problems
• Finds system behavior changes between code deployments
4. Interacting with your logging data
• Coralogix provides 3 ways to get insights from your logs:
1. Coralogix Dashbaord – a simple and powerful dashboard with machine
learning capabilities
2. Elastic’s Kibana – with a rich query language and flexible visualizations
3. Elasticsearch API – for deep technical querying and aggregations
5. Good product, happy customers!
• Everything worked smoothly for months
• Until we got a call from a customer (0.5TB / day)
• Some of his heavier dashboards could not be loaded
• He was not happy
• And neither were we
6. Well, of course. This makes no one happy!
• The error message was replicated in our offices as well
7. Kibana – technical overview
Port 5601
Node.js
server
Angular.js
client
localhost
Docker container
Docker container
Docker container
Our proprietary Kibana proxy:
• Emulates elasticsearch for Kibana
• Confines customers to only
access their data
• Parses queries for various SLA
restrictions
Port
9200
Port
9200
Customer
Publicdomain
8. So what could have gone wrong..?
• We looked into everything we could think of:
• Was the customer’s dashboard defined properly?
• It was.
• Was any indexed elasticsearch data corrupted?
• No.
• Was a large Kibana dashboard overloading our Kibana Proxy?
• Not according to the CPU and memory monitoring.
• Was there a hidden bug in our Kibana Proxy for certain queries?
• Replies seemed to be correct for every query we researched.
• Was any Docker container replaced recently, possibly with different settings?
• Yes, but new settings were not introduced.
• Was any Docker networking bug (and there are many…) interacting here?
• Not any that we could find.
9. Everything looked perfect!
• However, we did have one odd finding:
• When we were connected to our VPN, all the problems disappeared!
• Late at night and disappointed, we decided to call it a day:
10. Connecting the dots…
• Returning home, we each loaded the dashboard, and to our surprise
– everything worked!
• The same ISP served us and the customer, but not our homes.
• The new suspect was our Internet Service Provider!
11. Results – 1
• The next day, confident, we experimented:
• SSL vs. no SSL
• Kibana’s standard port 5601 vs. 443 https port
• Adding our Kibana CNAME to Cloudflare
• The results were staggering!
12. Results – 2
• Loading the dashboard without SSL through port 5601:
• Loading the same dashboard with SSL through port 443 and Cloudflare:
13. Results, solution and conclusion
• The ISP was throttling our requests, causing timeouts and packet
losses – eventually crashing heavy-loaded dashboards
• Adding our Kibana to Cloudflare under port 443 solved our problems
• (aside from wasting a whole day of our R&D team!)
• Conclusion: trust no-one!
14. Questions?
• Please feel free to contact me directly:
Lior Redlus, Chief Data Scientist, lior@coralogix.com
One month free trial @ http://www.coralogix.com
Notas do Editor
ןב 32
רגוב ראות ןושאר ינשו ב-
Neuroscience and Information Processing
וק-רדנואפ ו-
Chief Data Scientist
גולארוק'סקי