stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
âą
0 gostouâą201 visualizaçÔes
Encrypting data at rest is a must-have for any modern SaaS company. And if you run your software stack on Linux, LUKS/dm-crypt [1] is the usual go-to solution. However, as the storage becomes faster, the IO latency, introduced by dm-crypt becomes rather noticeable, especially on IO intensive workloads. At first glance it may seem natural, because data encryption is considered an expensive operation. But most modern hardware (specifically x86 and arm64) platforms have hardware optimisations to make encryption fast and less CPU intensive. Nevertheless, even on such hardware transparent disk encryption performs quite poorly.
Recomendados
Mais conteĂșdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
Semelhante a stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin (20)
Ăltimo
Ăltimo (20)
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
- 1. Speeding Up Linux Disk Encryption Ignat Korchagin @ignatkn
- 2. @ignatkn $ whoami â Performance and security at CloudïŹare â Passionate about security and crypto â Enjoy low level programming
- 3. Encrypting data at rest
- 6. @ignatkn The storage stack block subsystem filesystems applications
- 7. @ignatkn The storage stack storage hardware block subsystem filesystems applications
- 8. @ignatkn Encryption at rest layers storage hardware block subsystem filesystems applications SED, OPAL
- 9. @ignatkn Encryption at rest layers storage hardware block subsystem filesystems applications SED, OPAL LUKS/dm-crypt, BitLocker, FileVault
- 10. @ignatkn Encryption at rest layers storage hardware block subsystem filesystems applications SED, OPAL LUKS/dm-crypt, BitLocker, FileVault ecryptfs, ext4 encryption or fscrypt
- 11. @ignatkn Encryption at rest layers storage hardware block subsystem filesystems applications SED, OPAL LUKS/dm-crypt, BitLocker, FileVault ecryptfs, ext4 encryption or fscrypt DBMS, PGP, OpenSSL, Themis
- 12. @ignatkn Storage hardware encryption Pros: â itâs there â little conïŹguration needed â fully transparent to applications â usually faster than other layers
- 13. @ignatkn Storage hardware encryption Pros: â itâs there â little conïŹguration needed â fully transparent to applications â usually faster than other layers Cons: â no visibility into the implementation â no auditability â sometimes poor security https://support.microsoft.com/en-us/help/4516071/windows-10-update-kb4516071
- 14. @ignatkn Block layer encryption Pros: â little conïŹguration needed â fully transparent to applications â open, auditable
- 15. @ignatkn Block layer encryption Pros: â little conïŹguration needed â fully transparent to applications â open, auditable Cons: â requires somewhat specialised crypto â performance issues â encryption keys in RAM
- 16. @ignatkn Filesystem layer encryption Pros: â somewhat transparent to applications â open, auditable â more ïŹne-grained â more choice of crypto + potential integrity support
- 17. @ignatkn Filesystem layer encryption Pros: â somewhat transparent to applications â open, auditable â more ïŹne-grained â more choice of crypto + potential integrity support Cons: â performance issues â encryption keys in RAM â complex conïŹguration â unencrypted metadata
- 18. @ignatkn Application layer encryption Pros: â open, auditable â ïŹne-grained â full crypto ïŹexibility
- 19. @ignatkn Application layer encryption Pros: â open, auditable â ïŹne-grained â full crypto ïŹexibility Cons: â encryption keys in RAM â requires explicit support in code and conïŹguration â unencrypted metadata â full crypto ïŹexibility
- 20. LUKS/dm-crypt
- 21. @ignatkn Device mapper in Linux applications
- 22. @ignatkn Device mapper in Linux applications filesystems xfs vfatext4 ïŹle I/O
- 23. @ignatkn Device mapper in Linux applications block device drivers nvme brdscsi filesystems xfs vfatext4 ïŹle I/O block I/O
- 24. @ignatkn Device mapper in Linux applications block device drivers nvme brdscsi filesystems xfs vfatext4 device mapper dm-crypt dm-mirrordm-raid ïŹle I/O block I/O block I/O
- 25. @ignatkn Device mapper in Linux applications block device drivers nvme brdscsi filesystems xfs vfatext4 device mapper dm-crypt dm-mirrordm-raid ïŹle I/O block I/O block I/O
- 28. @ignatkn dm-crypt (idealized) filesystem dm-crypt block device drivers encrypt write BIO encrypted BIO
- 29. @ignatkn dm-crypt (idealized) filesystem dm-crypt block device drivers encrypt decrypt write BIO read BIO encrypted BIOs
- 30. @ignatkn dm-crypt (idealized) filesystem dm-crypt block device drivers encrypt decrypt Linux Crypto API write BIO read BIO encrypted BIOs
- 32. @ignatkn Test setup: RAM-based encrypted disk $ sudo modprobe brd rd_nr=1 rd_size=4194304
- 33. @ignatkn Test setup: RAM-based encrypted disk $ sudo modprobe brd rd_nr=1 rd_size=4194304 $ fallocate -l 2M crypthdr.img
- 34. @ignatkn Test setup: RAM-based encrypted disk $ sudo modprobe brd rd_nr=1 rd_size=4194304 $ fallocate -l 2M crypthdr.img $ sudo cryptsetup luksFormat /dev/ram0 --header crypthdr.img
- 35. @ignatkn Test setup: RAM-based encrypted disk $ sudo modprobe brd rd_nr=1 rd_size=4194304 $ fallocate -l 2M crypthdr.img $ sudo cryptsetup luksFormat /dev/ram0 --header crypthdr.img $ sudo cryptsetup open --header crypthdr.img /dev/ram0 encrypted-ram0
- 36. @ignatkn test storage stack Test storage stack /dev/ram0
- 37. @ignatkn test storage stack Test storage stack /dev/ram0 /dev/mapper/encrypted-ram0
- 38. @ignatkn test storage stack Test storage stack /dev/ram0 /dev/mapper/encrypted-ram0 no filesystem
- 39. @ignatkn test storage stack Test storage stack /dev/ram0 /dev/mapper/encrypted-ram0 no filesystemEncrypted R/W
- 40. @ignatkn test storage stack Test storage stack /dev/ram0 /dev/mapper/encrypted-ram0 no filesystemEncrypted R/W Unencrypted R/W
- 41. @ignatkn Test setup: sequential reads $ sudo fio --filename=/dev/ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=plain
- 42. @ignatkn Test setup: sequential reads $ sudo fio --filename=/dev/ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=plain ... READ: io=21013MB, aggrb=1126.5MB/s WRITE: io=21023MB, aggrb=1126.1MB/s
- 43. @ignatkn Test setup: sequential reads $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt
- 44. @ignatkn Test setup: sequential reads $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt ... READ: io=1693.7MB, aggrb=150874KB/s WRITE: io=1696.4MB, aggrb=151170KB/s
- 45. @ignatkn Test setup: sequential reads $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt ... READ: io=1693.7MB, aggrb=150874KB/s WRITE: io=1696.4MB, aggrb=151170KB/s ~7x slower!
- 46. @ignatkn Test setup: sequential reads $ sudo cryptsetup benchmark -c aes-xts # Tests are approximate using memory only (no storage IO). # Algorithm | Key | Encryption | Decryption aes-xts 256b 1823.1 MiB/s 1900.3 MiB/s
- 47. @ignatkn Test setup: sequential reads $ sudo cryptsetup benchmark -c aes-xts # Tests are approximate using memory only (no storage IO). # Algorithm | Key | Encryption | Decryption aes-xts 256b 1823.1 MiB/s 1900.3 MiB/s desired: ~696 MB/s, actual: ~294 MB/s
- 48. @ignatkn We tried... â switching to diïŹerent cryptographic algorithms â aes-xts seems to be the fastest (at least on x86)
- 49. @ignatkn We tried... â switching to diïŹerent cryptographic algorithms â aes-xts seems to be the fastest (at least on x86) â experimenting with dm-crypt optional ïŹags â âsame_cpu_cryptâ and âsubmit_from_crypt_cpusâ
- 50. @ignatkn We tried... â switching to diïŹerent cryptographic algorithms â aes-xts seems to be the fastest (at least on x86) â experimenting with dm-crypt optional ïŹags â âsame_cpu_cryptâ and âsubmit_from_crypt_cpusâ â trying ïŹlesystem-level encryption â much slower and potentially less secure
- 51. @ignatkn Despair
- 52. @ignatkn Ask the community âIf the numbers disturb you, then this is from lack of understanding on your side. You are probably unaware that encryption is a heavy-weight operation...â https://www.spinics.net/lists/dm-crypt/msg07516.html
- 53. @ignatkn But actually... âUsing TLS is very cheap, even at the scale of CloudïŹare. Modern crypto is very fast, with AES-GCM and P256 being great examples.â https://blog.cloudflare.com/how-expensive-is-crypto-anyway/
- 54. @ignatkn dm-crypt: life of an encrypted BIO request block device drivers filesystem
- 55. @ignatkn Crypto APIdm-crypt dm-crypt: life of an encrypted BIO request block device drivers filesystem
- 56. @ignatkn Crypto APIdm-crypt block device drivers filesystem kcryptd write dm-crypt: life of an encrypted BIO request
- 57. @ignatkn Crypto APIdm-crypt block device drivers filesystem cryptd kcryptd write dm-crypt: life of an encrypted BIO request
- 58. @ignatkn Crypto APIdm-crypt block device drivers filesystem cryptd kcryptd write write_tree dm-crypt: life of an encrypted BIO request
- 59. @ignatkn Crypto APIdm-crypt block device drivers filesystem cryptd kcryptd dmcrypt_write write write_tree dm-crypt: life of an encrypted BIO request
- 60. @ignatkn Crypto APIdm-crypt block device drivers filesystem cryptd kcryptd dmcrypt_write write write_tree dm-crypt: life of an encrypted BIO request
- 61. @ignatkn Crypto APIdm-crypt block device drivers filesystem cryptd kcryptd_io kcryptd dmcrypt_write write read write_tree dm-crypt: life of an encrypted BIO request
- 62. @ignatkn Crypto APIdm-crypt block device drivers filesystem cryptd kcryptd_io kcryptd dmcrypt_write write read write_tree dm-crypt: life of an encrypted BIO request
- 63. @ignatkn Crypto APIdm-crypt block device drivers filesystem cryptd kcryptd_io kcryptd dmcrypt_write write read write_tree dm-crypt: life of an encrypted BIO request
- 64. @ignatkn Crypto APIdm-crypt block device drivers filesystem cryptd kcryptd_io kcryptd dmcrypt_write write read write_tree dm-crypt: life of an encrypted BIO request
- 65. @ignatkn Crypto APIdm-crypt block device drivers filesystem cryptd kcryptd_io kcryptd dmcrypt_write write read write_tree dm-crypt: life of an encrypted BIO request
- 66. @ignatkn queues vs latency âA signiïŹcant amount of tail latency is due to queueing eïŹectsâ https://www.usenix.org/conference/srecon19asia/presentation/plenz
- 67. @ignatkn Crypto APIdm-crypt block device drivers filesystem cryptd kcryptd_io kcryptd dmcrypt_write write read write_tree dm-crypt: life of an encrypted BIO request
- 68. @ignatkn Crypto APIdm-crypt block device drivers filesystem cryptd kcryptd_io kcryptd dmcrypt_write write read write_tree dm-crypt: life of an encrypted BIO request
- 69. @ignatkn dm-crypt: git archeology â kcryptd was there from the beginning (2005) â only for reads: âit would be very unwise to do decryption in an interrupt contextâ
- 70. @ignatkn dm-crypt: git archeology â kcryptd was there from the beginning (2005) â only for reads: âit would be very unwise to do decryption in an interrupt contextâ â some queuing was added to reduce kernel stack usage (2006)
- 71. @ignatkn dm-crypt: git archeology â kcryptd was there from the beginning (2005) â only for reads: âit would be very unwise to do decryption in an interrupt contextâ â some queuing was added to reduce kernel stack usage (2006) â oïŹoad writes to thread and IO sorting (2015) â for spinning disks, but âmay improve SSDsâ â mentions CFQ scheduler, which is deprecated
- 72. @ignatkn dm-crypt: git archeology â kcryptd was there from the beginning (2005) â only for reads: âit would be very unwise to do decryption in an interrupt contextâ â some queuing was added to reduce kernel stack usage (2006) â oïŹoad writes to thread and IO sorting (2015) â for spinning disks, but âmay improve SSDsâ â mentions CFQ scheduler, which is deprecated â commits to optionally revert some queuing â âsame_cpu_cryptâ and âsubmit_from_crypt_cpusâ option ïŹags
- 73. @ignatkn dm-crypt: things to reconsider â most code was added with spinning disks in mind â disk IO latency >> scheduling latency
- 74. @ignatkn dm-crypt: things to reconsider â most code was added with spinning disks in mind â disk IO latency >> scheduling latency â sorting BIOs in dm-crypt probably violates âdo one thing and do it wellâ Unix principle â the task for the IO scheduler
- 75. @ignatkn dm-crypt: things to reconsider â most code was added with spinning disks in mind â disk IO latency >> scheduling latency â sorting BIOs in dm-crypt probably violates âdo one thing and do it wellâ Unix principle â the task for the IO scheduler â kcryptd may be redundant as modern Linux Crypto API is asynchronous by itself â remove oïŹoading the oïŹoad
- 77. @ignatkn Crypto APIdm-crypt block device drivers filesystem cryptd kcryptd_io kcryptd dmcrypt_write write read write_tree dm-crypt: life of an encrypted BIO request
- 78. @ignatkn dm-crypt (synchronous) filesystem dm-crypt block device drivers encrypt decrypt write BIO read BIO encrypted BIOs
- 79. @ignatkn dm-crypt (synchronous) filesystem dm-crypt block device drivers encrypt decrypt Sync Linux Crypto API write BIO read BIO encrypted BIOs
- 80. @ignatkn dm-crypt: removing queues â dm-crypt module: a simple patch, which bypasses all queues/async threads based on a new runtime ïŹag
- 81. @ignatkn dm-crypt: removing queues â dm-crypt module: a simple patch, which bypasses all queues/async threads based on a new runtime ïŹag â Linux Crypto API is a bit more complicated â by default speciïŹc implementation is selected dynamically based on priority â aes-ni synchronous implementation is marked as âinternalâ â aes-ni (FPU) is not usable in some interrupt contexts
- 82. @ignatkn dm-crypt: removing queues â dm-crypt module: a simple patch, which bypasses all queues/async threads based on a new runtime ïŹag â Linux Crypto API is a bit more complicated â by default speciïŹc implementation is selected dynamically based on priority â aes-ni synchronous implementation is marked as âinternalâ â aes-ni (FPU) is not usable in some interrupt contexts â xtsproxy: a dedicated synchronous aes-xts module
- 83. @ignatkn xtsproxy crypto API module aes-xts proxy
- 84. @ignatkn xtsproxy crypto API module aes-xts proxy Is FPU available?
- 85. @ignatkn xtsproxy crypto API module aes-xts proxy Is FPU available? __xts-aes-aesni yes
- 86. @ignatkn xtsproxy crypto API module aes-xts proxy Is FPU available? __xts-aes-aesni xts(ecb(aes-generic)) yes no
- 87. @ignatkn xtsproxy crypto API module aes-xts proxy Is FPU available? __xts-aes-aesni xts(ecb(aes-generic)) yes no
- 88. @ignatkn Test setup: sequential IO $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt
- 89. @ignatkn Test setup: sequential IO $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt $ sudo modprobe xtsproxy
- 90. @ignatkn Test setup: sequential IO $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt $ sudo modprobe xtsproxy $ sudo dmsetup table encrypted-ram0 --showkeys | sed 's/aes-xts-plain64/capi:xts-aes-xtsproxy-plain64/' | sed 's/$/ 1 force_inline/' | sudo dmsetup reload encrypted-ram0
- 91. @ignatkn Test setup: sequential IO $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt $ sudo modprobe xtsproxy $ sudo dmsetup table encrypted-ram0 --showkeys | sed 's/aes-xts-plain64/capi:xts-aes-xtsproxy-plain64/' | sed 's/$/ 1 force_inline/' | sudo dmsetup reload encrypted-ram0
- 92. @ignatkn Test setup: sequential IO $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt $ sudo modprobe xtsproxy $ sudo dmsetup table encrypted-ram0 --showkeys | sed 's/aes-xts-plain64/capi:xts-aes-xtsproxy-plain64/' | sed 's/$/ 1 force_inline/' | sudo dmsetup reload encrypted-ram0
- 93. @ignatkn Test setup: sequential IO $ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt $ sudo modprobe xtsproxy $ sudo dmsetup table encrypted-ram0 --showkeys | sed 's/aes-xts-plain64/capi:xts-aes-xtsproxy-plain64/' | sed 's/$/ 1 force_inline/' | sudo dmsetup reload encrypted-ram0 $ sudo dmsetup suspend encrypted-ram0 && sudo dmsetup resume encrypted-ram0
- 96. @ignatkn ssd: IO latency (iowait) â ssd disk â dm-crypt device
- 97. @ignatkn ssd: IO latency (iowait) inline enabled â ssd disk â dm-crypt device
- 99. @ignatkn CloudïŹare cache impact â unencrypted â encrypted â encrypted with patched dm-crypt
- 100. Conclusions
- 101. @ignatkn Conclusions â a simple patch which may improve dm-crypt performance by 200%-300% â fully compatible with stock Linux dm-crypt â can be enabled/disabled in runtime without service disruption
- 102. @ignatkn Conclusions â a simple patch which may improve dm-crypt performance by 200%-300% â fully compatible with stock Linux dm-crypt â can be enabled/disabled in runtime without service disruption â modern crypto is fast and cheap â performance degradation is likely elsewhere
- 103. @ignatkn Conclusions â a simple patch which may improve dm-crypt performance by 200%-300% â fully compatible with stock Linux dm-crypt â can be enabled/disabled in runtime without service disruption â modern crypto is fast and cheap â performance degradation is likely elsewhere â extra queuing may be harmful on modern low latency storage
- 104. @ignatkn Caveats and future work â the patch improves performance on small block size/high IOPS workloads â >2MB block size shows worse performance
- 105. @ignatkn Caveats and future work â the patch improves performance on small block size/high IOPS workloads â >2MB block size shows worse performance â the whole setup assumes hardware-accelerated crypto â xtsproxy supports x86 only
- 106. @ignatkn Caveats and future work â the patch improves performance on small block size/high IOPS workloads â >2MB block size shows worse performance â the whole setup assumes hardware-accelerated crypto â xtsproxy supports x86 only â your mileage may vary â always measure and compare before deployment â let us know the results
- 107. @ignatkn Links â https://gitlab.com/cryptsetup/cryptsetup â http://man7.org/linux/man-pages/man8/dmsetup.8.html â https://blog.cloudïŹare.com/speeding-up-linux-disk-encryption â https://github.com/cloudïŹare/linux
- 108. Questions?