Poland Meetup June

1. Tomasz Gintowt
Elastic Stack
upgrade
03-06-2020

2. 2
Architekt, DevOps, SysOps oraz
DBA. Jestem skupiony na
dostarczaniu rozwiązań
składowania i przetwarzania
danych. Nie są mi obce wszelkiej
maści bazy danych, systemy real
time data i streamingu.
Doradzam, wdrażam, projektuje
jak efektywnie wykorzystywać
technologie.
Tomasz Gintowt
https://www.linkedin.com/in/tomasz-gintowt/

3. Kibana
Beats
Version 6.4.6 Filebeat, metricbeat
ElasticSearch
Version 6.4.6 - 3 x master, 3 x
hot, 5 x cold, 1 x ingest
3
Version 6.4.6 +
ReadOnlyRest plugin
Patient

4. Elastic Stack cluster
4
21870 1T
2501583
5707
Active Shards
Shards per Hot node CPU
RAM
Shards per Cold node

5. Game plan
01 UPGRADE FROM 6.4 TO 6.8
02 UPGRADE FROM 6.8 TO 7.X

6. 6
Check
Breaking
Changes

7. 7
Back up
your
DATA!
Warning
Back up
your
DATA!
Back up
your
DATA!

8. Kibana Beats
ElasticSearch Logstash
8
1 2 3 4
Order

9. Rolling
Upgrade
9
Full
Cluster
Restart
Upgrade

10. curl -XPUT
localhost:9200/_cluster/settings -H
'Content-Type: application/json' -d
'{
"transient" :{
"cluster.routing.allocation.exclude.
_ip" : "111.111.111.111"
}
}'
10
Testing on DEV

11. ● elasticsearch-plugin list
● elasticsearch-plugin remove
XXX --purge
● elasticsearch-plugin install
file:////tmp/
11
ES plugins

12. Change allocation to
"primaries"
Start ES node
/_flush/synced
Install new
plugins
Cluster Status
GREEN
12
Ansible
Stop ES node
Remove old
plugins
Upgrade ES
package
Change allocation to
"all"

13. - name: Enable allocation
uri:
url: http://{{ es.host }}:{{ es.port }}/_cluster/settings
method: PUT
body_format: json
body: "{{ es.enable_allocation }}"
delegate_to: "{{ es.master }}"
register: enable_allocation
until: enable_allocation.status == 200
13
Ansible

14. - name: Wait till relocation finish
uri:
url: http://{{ es.host }}:{{ es.port }}/_cluster/health
method: GET
register: response_shards
until: response_shards.json.unassigned_shards
== 0 and response_shards.json.initializing_shards ==
0
retries: 200
delay: 20
delegate_to: "{{ es.master }}"
14
Ansible

15. https://github.com/elastic/ansible-elasticsearch
15
Ansible

16. 16
Back up
.kibana index
Check
configuration
file
Stop all
kibana
instances
Kibana upgrade

17. 17
Problems

18. ● AUTHOR (YEAR). Title of the publication. Publisher
● AUTHOR (YEAR). Title of the publication. Publisher
● AUTHOR (YEAR). Title of the publication. Publisher
● AUTHOR (YEAR). Title of the publication. Publisher
● AUTHOR (YEAR). Title of the publication. Publisher
● AUTHOR (YEAR). Title of the publication. Publisher
18
Upgrade Assistant

19. {"type":"log","@timestamp":"2020-06-01T20:33:17Z","tags":["info","mig
rations"],"pid":11550,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2020-06-01T20:33:24Z","tags":["info","mig
rations"],"pid":11550,"message":"Migrating .kibana_1 saved objects to
.kibana_2"}
{"type":"error","@timestamp":"2020-06-01T20:33:27Z","tags":["warning"
,"stats-collection"],"pid":11550,"level":"error","error":{"message":"
mapping set to strict, dynamic introduction of [managed] within
[index-pattern] is not allowed","name":"Error","stack":"Error:
mapping set to strict, dynamic introduction of [managed] within
[index-pattern] is not allowedn at Object.write
(/usr/share/kibana/src/server/saved_objects/migrations/core/elastic_i
ndex.js:110:23)"},"message":"mapping set to strict, dynamic
introduction of [managed] within [index-pattern] is not allowed"} 19
Errors

20. {"type":"log","@timestamp":"2020-06-01T20:22:18Z","tags":["fatal","r
oot"],"pid":9805,"message":"{ Error: listen EACCES: permission
denied 127.0.0.1:443n at Server.setupListenHandle [as _listen2]
(net.js:1260:19)n at listenInCluster (net.js:1325:12)n at
GetAddrInfoReqWrap.doListen (net.js:1458:7)n at
GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:62:10)n code:
'EACCES',n errno: 'EACCES',n syscall: 'listen',n address:
'127.0.0.1',n port: 443 }"}
20
Errors

21. https://www.elastic.co/guide/en/logstash/current/upgrading-logstash.html
If any Logstash
plugin is not
compatible with
ver 7.0, then wait
until it’s ready.
Elasticsearch 7.x does not
require Logstash 7.x.
Indexing fail if you
attempt to send the
7.x template to a
cluster ES 6.x
21
Logstash upgrade

22. Upgrading between
non-consecutive major
versions (e.g. 5.x to 7.x) is
not supported.
If you’re on a previous 6.x
version of Beats, upgrade
to version 6.8 first.
22
Upgrading to 6.8 is
required because the
Elasticsearch index
template was modified to
be compatible with
Elasticsearch 7.0 (the _type
setting changed from doc
to _doc).
Starting with 7.0, the fields
exported by Beats conform to
the Elastic Common Schema
(ECS).
migration.6_to_7.enabled: true
Beats

23. 23
?
Questions
?
?

24. 24
THANKS
Tomasz Gintowt
https://www.linkedin.com/in/tomasz-gintowt/