Fernet tokens: newton summit

1. Matt Fischer (mfisch) Dolph Mathews (dolphm) Lance Bragstad (lbragstad) Fernet tokens Improving keystone’s scalability

2. Fernet tokens What is a token? Why do we need another token format? Why does token non-persistence matter? What's inside a Fernet token? How does key rotation work? How does key rotation work in a cluster? What is the state of OpenStack at Time Warner Cable? What prerequisites does Fernet require? What did it take to migrate to Fernet? What did we learn operating Fernet tokens?

4. What is a token? GET /v2/b5a951/servers HTTP/1.1 Host: servers.api.openstack.org Accept: application/json X-Auth-Token: $TOKEN

6. Why not use UUID tokens? They must be persisted. 779810523fb24886b67a23f4f823b685

7. Why not use PKI tokens? They are huge. MIIE-gYJKoZIhvcNAQcCoIIE7zCCBOsCAQExDTALBglghkgBZQMEAgEwggNMBgkqhkiG9w0BBwGgggM9BIIDO XsidG9rZW4iOnsibWV0aG9kcyI6WyJwYXNzd29yZCJdLCJyb2xlcyI6W3siaWQiOiIzNjBiMTc3ZDhjMjM0 N2ZmOTVlMGFjMTYxNWJhOGZiNiIsIm5hbWUiOiJhZG1pbiJ9XSwiZXhwaXJlc19hdCI6IjIwMTUtMDItMjZ UMDU6NDg6MjYuMDk0MDk4WiIsInByb2plY3QiOnsiZG9tYWluIjp7ImlkIjoiZGVmYXVsdCIsIm5hbWUiOi JEZWZhdWx0In0sImlkIjoiNTkwMDJjZTczOWYxNDNiYjhiMmNjMzNjYWY5OGZjZjkiLCJuYW1lIjoiYWRta W4ifSwiY2F0YWxvZyI6W3siZW5kcG9pbnRzIjpbeyJyZWdpb25faWQiOm51bGwsInVybCI6Imh0dHA6Ly8x MDQuMjM5LjE2My4yMTU6MzUzNTcvdjMiLCJyZWdpb24iOm51bGwsImludGVyZmFjZSI6ImFkbWluIiwiaWQ iOiI5YTI5ZWFmMjBmNzk0MmI2YjljOTZjZmIwYWEwMmEzZSJ9LHsicmVnaW9uX2lkIjpudWxsLCJ1cmwiOi JodHRwOi8vMTA0LjIzOS4xNjMuMjE1OjM1MzU3L3YzIiwicmVnaW9uIjpudWxsLCJpbnRlcmZhY2UiOiJwd WJsaWMiLCJpZCI6ImQzMjMzYWZkMmI2MDQxZDRhMzlmOGFjMTIzMzc1N2ZkIn1dLCJ0eXBlIjoiaWRlbnRp dHkiLCJpZCI6IjFiNzk2ZTIxNGY4MTQwMTE4MTA4YTdlNGU0Y2E2ZTE2IiwibmFtZSI6IktleXN0b25lIn1 dLCJleHRyYXMiOnt9LCJ1c2VyIjp7ImRvbWFpbiI6eyJpZCI6ImRlZmF1bHQiLCJuYW1lIjoiRGVmYXVsdC J9LCJpZCI6Ijg1YTlhZjE0NWRkYjRkMTlhOTU0NGRmYmVhYzVkMWYwIiwibmFtZSI6ImFkbWluIn0sImF1Z Gl0X2lkcyI6WyJZeW9iU2FIY1ROQ3U3c2V1c2RUdHBRIl0sImlzc3VlZF9hdCI6IjIwMTUtMDItMjZUMDU6 MzM6MjYuMDk0MTI3WiJ9fTGCAYUwggGBAgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVVuc2V0MQ4 wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxGDAWBgNVBAMMD3d3dy5leGFtcGxlLmNvbQIBATALBg lghkgBZQMEAgEwDQYJKoZIhvcNAQEBBQAEggEAYJR+ETbjA4RpgToeRm0qh-zxRWyBL4RdN99hLHV6foIpc r6uXMN-DaUJvGygPDi1wi-HAbpErJAe9iRHk4+8BUnX--jQRTaYhkg237eyjpYHU8Hgt8Ydn7Wdnn0hriXK t+RZBG-ZEnnP-MZ9V9GGJz-BoAMHx42uF5j6mlfVvUxtJGSaZ2wPROkLIHAjrX-8zEo8YhtGQHi-rFvXOoP +w8TVb907R2WNsGs3LbFKRmDv-yev6pMnz+gQu8uImf2idd18hyEYdw8M9bgZc2YsGBiPSeIm-VhzH9qTX0 e7fK-chhAE+saIEbl5Mw0PzybhTyKHRzqtsW4HWFOlbE0yOA==

8. Why not use PKIZ tokens? They are still huge. PKIZ_eJxtVcmSozgUvPMVc6_oKMBgm0Mf2IzBCIpVlm4sNiAEtssLy9eP7K6Jqo4YboCUysyX7-nXL_ZopmV7 _-gger784oBtm-8VcnYnbNePwlODQj-xb6tZ1zX_qquBORqx6moVreq20nAATLUyh6rygFa1F65uG0sZeE0 brKqqgKLZtuHvr01pKZ8YSo3fX5scpnxmKW0x2Us4OQPae3MpKhPWnZJzdWfKxZG-fi6uTQaDxm9s2TPAgE gwe10i-9DkPWLOfkwpIJWMYq32LId4c7LgfN2-2p1c5zBhG50aW8I5bxxlHw0N3tdDtndoISh1qdtLm9gDi JMbMOwbIDgBBlpyIEZLQII7mNuJnTrDhgH2GmN1pmgRvCRgS7khSO82Oa_sjrY2ObFvaYf26ZUr_2ZgYojr Eo683fPX78WmhOaw82MgITHtPCvhgWjzvpW2HLBwh4nX-kYgYENtmCd3BAX63IhgeMuYkUcmB4kbHsHxgb- 8wlBuC0s5c3kfzoxafpicCcPynIvy8WVkJwu5NTA56ZQ_9Xc1X27VpTutR2AwyQTILjFFDkzSxIxZgjmZvb h4lAQ8WXyBSd9AHb2XVjrhbkNw9ATctDnzhbOb4at0Tu2RkIC4HX3DHDFBPIYhRXG1AHNKEUEy6hAPIJhw5 Cju9toUXdpzGVTue_Fp1vnOzLuy04WiG56Ap3IbDn6zfoBY5V1iz34kjR4BjL4p-AQI4JkDd4HmJ4sn2hPs B9CZ-UOLDtdIfFVoKKFzzeBL4hm_fAELDhgVQy07TwwpjkMmg9a-0cqsTIJnPdPXDqBDC7sXSraRP-y1V4U yJo8dcObKbfuNSBIex7YErISFqlpgI-CxUdYotmcQOy0mxeiJKYuwR5-s825z416Otjd62Hs8KyH9Ooketu GE9oAl8aa8fBHT6U8Sw0cONyzu9pKV_sz90cLodxsh3wZ_BSn8imupO8o3S6_GsSkxhjyaW55jNAVECtm37 AUmlQQgK6eFJCAC-T-aP-v-J-IbAVuUf1aP--rxNklGMekrIRM290g8NxnFt6yjJOmd3qavvpiLRUrx5u_O 5H62JjDMH52JJMja-hhbuooSNoEsjU0iDWyGIZ1NF6itpQqJyWk10NMUjAZR2YjyUrYKaGl6Z6bxIJAGQ0V GGgRbQ03TvPdoaZg-UIfXZr0aNlwK5Rnvg9EyVPgHAABjUS7KSaYHa3MrrJG6nffIA1tT_2c2ckbwc6Camh aoZlWZ6s5fHiM7FSN_F4LPwIZ62eK-Ck7bCCpG5gpWk55VZuJb-wZ30-Uwfh6c4_0Srgp12Ak0si9usTwdm uUcuHlIuqUjXarRXcN-_THIn6tdAN-nPSg57PGwD4Wt2Avm6qpmghnW1w0ZrGUX7cQ3MprKmr7nWFmkufam ysNiZfWSqNPDabMl54Q7ykPw2Gzxx1G8gzcNvGvRvTCjTLAqtQ1dZ7xM-zxbbam8Vha3SgGNhxL8-bESItc 8SiF3PhHSXD4Mfztp16N2Em_F8CYqviBlaj917zPUwf2h-1nsiVSIpWGKeu-Gdtc6rtfD2eRWEbn5VNhNU- wivHb8i14U1yo6RNH7qf0Y4ValpVTG9nR4NMHv39zrQjM94_ty-xc2_Erg

9. What does a Fernet token look like? small enough, non-persistent, setup, online validation gAAAAABU7roWGiCuOvgFcckec- 0ytpGnMZDBLG9hA7Hr9qfvdZDHjsak39YN98HXxoYLIqVm19Egku5YR3wyI7heVrOmPNEtmr- fIM1rtahudEdEAPM4HCiMrBmiA1Lw6SU8jc2rPLC7FK7nBCia_BGhG17NVHuQu0S7waA306jyKNhHwUnp sBQ=

11. What do we mean by non-persistent? > SELECT * from `token`; Empty set (0.00 sec)

12. What is the impact of non-persistence? Fewer backend writes. Lowers data replication footprint. Zero replication lag for validation. No database cleanup cronjobs. Token validation is hard. Token revocation is hard.

14. What's inside a Fernet token? user identity + scope of authorization

15. How are Fernet tokens composed? payload = msgpack(user_id + project_id + expiration)

16. How are Fernet tokens composed? payload = msgpack(user_id + project_id + expiration) decoded = HMAC(version + timestamp + AES(payload))

17. How are Fernet tokens composed? payload = msgpack(user_id + project_id + expiration) decoded = HMAC(version + timestamp + AES(payload)) token = base64(decoded)

18. Where can we learn more about Fernet? https://github.com/fernet/spec HMAC(version + timestamp + IV + AES(plaintext))

19. What is a Fernet key? SHA-256 HMAC + 128-bit AES 128-bit key + 128-bit key BfDqM5lrTVhZ2bVzmlhKy0xNbbFg9owTz2hZeIG7rQc=

21. How do we configure Fernet tokens? [token] provider = fernet [fernet_tokens] key_repository = /etc/keystone/fernet-keys/ max_active_keys = 3

22. How often should we rotate keys? How fast can your adversary crack both a 128-bit AES-CBC encryption key and a 128-bit SHA256 HMAC signing key?

23. How often should we rotate keys? How fast can your adversary crack both a 128-bit AES-CBC encryption key and a 128-bit SHA256 HMAC signing key? Rotate more frequently than that.

24. What are the variables of a key rotation strategy? Rotation frequency Token lifespan: [token] expiration Number of keys: [fernet_tokens] max_active_keys

25. How often should we rotate keys? keystone.conf [tokens] expiration = 86400 # 1 day keystone.conf [fernet_tokens] max_active_keys = 25 hourly rotation frequency

26. How often should we rotate keys? keystone.conf [tokens] expiration = 604800 # 1 week keystone.conf [fernet_tokens] max_active_keys = 8 daily rotation frequency

27. How often should we rotate keys? keystone.conf [tokens] expiration = 7776000 # 90 days keystone.conf [fernet_tokens] max_active_keys = 5 monthly rotation frequency

28. What should be considered for key rotation? How many max_active_keys should you have? 3 or more. Do you have to rotate all your nodes synchronously? No. Why not store Fernet keys in the database? (see stack overflow)

29. What does keystone provide to manage keys? $ keystone-manage fernet_setup $ keystone-manage fernet_rotate

30. What does fernet_setup do? max_active_keys = 4 ■ (staged: the next primary key) ■ (primary: token generation & validation)

31. What does our first fernet_rotate do? max_active_keys = 4 ■ (staged: the next primary key) ■ (secondary: token validation) ■ (primary: token generation & validation)

32. What does our second fernet_rotate do? max_active_keys = 4 ■ (staged: the next primary key) ■ (secondary: token validation) ■ (secondary: token validation) ■ (primary: token generation & validation)

33. What does our third fernet_rotate do? max_active_keys = 4 ■ (staged: the next primary key) ■ (deleted) ■ (secondary: token validation) ■ (secondary: token validation) ■ (primary: token generation & validation)

34. What does our fourth fernet_rotate do? max_active_keys = 4 ■ (staged: the next primary key) ■ (deleted) ■ (deleted) ■ (secondary: token validation) ■ (secondary: token validation) ■ (primary: token generation & validation)

36. How do we use fernet_setup with a cluster? max_active_keys = 3 node1 node2 node3 ■ fernet setup ■ key rotation cluster sync staged keys differ across nodes all nodes have all primary & secondary keys

37. How do we use fernet_setup with a cluster? max_active_keys = 3 node1 node2 node3 ■ ----> ■ ----> ■ fernet setup ■ ----> ■ ----> ■ key rotation cluster sync staged keys differ across nodes all nodes have all primary & secondary keys

38. How do we use fernet_setup with a cluster? max_active_keys = 3 node1 node2 node3 ■ ■ ■ fernet setup ■ ■ ■ key rotation cluster sync staged keys differ across nodes all nodes have all primary & secondary keys

39. How do we use fernet_rotate with a cluster? max_active_keys = 3 node1 node2 node3 ■ ■ ■ fernet setup ■ ■ ■ key rotation ■ cluster sync staged keys differ across nodes all nodes have all primary & secondary keys

40. How do we use fernet_rotate with a cluster? max_active_keys = 3 node1 node2 node3 ■ ----> ■ ----> ■ fernet setup ■ ■ ■ key rotation ■ ----> ■ ----> ■ cluster sync staged keys differ across nodes all nodes have all primary & secondary keys

41. How do we use fernet_rotate with a cluster? max_active_keys = 3 node1 node2 node3 ■ ■ ■ fernet setup ■ ■ ■ key rotation ■ ■ ■ cluster sync staged keys differ across nodes all nodes have all primary & secondary keys

42. How do we use fernet_rotate with a cluster? max_active_keys = 3 node1 node2 node3 ■ ■ ■ fernet setup ■ ■ ■ key rotation ■ ■ ■ cluster sync ■ staged keys differ across nodes all nodes have all primary & secondary keys

43. How do we use fernet_rotate with a cluster? max_active_keys = 3 node1 node2 node3 ■ ----> ■ ----> ■ fernet setup ■ ----> ■ ----> ■ key rotation ■ ■ ■ cluster sync ■ ----> ■ ----> ■ staged keys differ across nodes all nodes have all primary & secondary keys

44. How do we use fernet_rotate with a cluster? max_active_keys = 3 node1 node2 node3 ■ ■ ■ fernet setup key rotation ■ ■ ■ cluster sync ■ ■ ■ staged keys differ across nodes all nodes have all primary & secondary keys

45. How do we use fernet_rotate with a cluster? max_active_keys = 3

46. What breaks if we rotate twice without syncing? max_active_keys = 3 node1 node2 node3 ■ ■ ■ fernet setup ■ ■ ■ key rotation cluster sync staged keys differ across nodes all nodes have all primary & secondary keys

47. What breaks if we rotate twice without syncing? max_active_keys = 3 node1 node2 node3 ■ ■ ■ fernet setup ■ ■ ■ key rotation ■ cluster sync staged keys differ across nodes all nodes have all primary & secondary keys

48. What breaks if we rotate twice without syncing? max_active_keys = 3 node1 node2 node3 ■ ■ ■ fernet setup ■ ■ ■ key rotation ■ cluster sync ■ staged keys differ across nodes node2 and node3 are now missing node1’s primary key

50. What’s the state of OpenStack at TWC? Multiple regions Liberty Keystone + mixed versions of other services Fernet tokens since July 2015 Keystone Keystone Keystone Keystone Keystone Keystone MySQL Region 1 Region 2

51. What’s the state of OpenStack at TWC? Multiple regions Liberty Keystone + mixed versions of other services Fernet tokens since July 2015

52. What’s the state of OpenStack at TWC? Multiple regions Liberty Keystone + mixed versions of other services Fernet tokens since July 2015

54. What prerequisites does Fernet require? Keystone Kilo or newer, Liberty recommended Any Django OpenStack Auth release Kilo or newer Any KeystoneMiddleware release Kilo or newer Enable caching in Keystone

59. What did it take to migrate to Fernet? Why migrate? How did you orchestrate the upgrade? How long was your outage? How does TWC rotate tokens?

60. What did it take to migrate to Fernet? Why migrate? How did you orchestrate the upgrade? https://github.com/matthewfischer/ansible/tree/master/keystone-upgrade How long was your outage? How does TWC rotate tokens?

63. How does Fernet key rotation work at TWC? Does not use keystone-manage Keys stored in EYAML and deployed by Puppet Key changes are reviewed using Gerrit Rolling rotations across multiple regions and nodes Frequency

69. What did we learn operating Fernet tokens? Empty token table No messy cronjobs No replication lag Performance

73. Where can we learn more about Fernet? http://docs.openstack.org/developer/keystone/configuration.html#token-provider http://docs.openstack.org/admin-guide/keystone_fernet_token_faq.html http://www.mattfischer.com/blog/?p=648 http://dolphm.com/openstack-keystone-fernet-tokens http://lbragstad.com/fernet-tokens-and-key-rotation http://lbragstad.com/fernet-tokens-and-key-distribution https://github.com/fernet/spec

Fernet tokens: newton summit

Recomendados

Recomendados

Mais conteúdo relacionado

Mais de Lance Bragstad

Mais de Lance Bragstad (8)

Último

Último (20)

Fernet tokens: newton summit