Trusted Execution Environments (TEEs) are gaining traction in academia and industry as a fulcrum to build trustworthy systems.
Built as dedicated hardware components in mobile or server-grade processors, and available in infrastructure-as-a-service cloud providers,
TEEs allow applications with high privacy and confidentiality demands to be deployed and executed over untrusted environments,
shielding data and code from compromised systems or powerful attackers.
After a quick introduction to basic concepts for TEEs, I will survey some of our most recent contributions exploiting TEEs,
including as defensive tools in the context of Federated Learning, as support to build secure cache systems for edge networks,
shielding novel runtime environments (ie, WebAssembly) within Intel SGX enclaves, and more.
For each of the systems built, I will highlight some of the lessons learned, hopefully useful to future
researchers and practitioners entering this exciting area of research.
1. Lessons Learned
in Building Trustworthy Systems wit
h
Trusted Execution Environments
Invited Talk - LaBR
I
26 October 202
1
Dr Valerio Schiavon
i
University of Neuchâtel, Switzerland
2. /41 valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•B.Sc. and M.Sc. in Software Engineering, Rome, I
T
•University start-up (web extraction), Rome, I
T
•Research Engineer, INRIA Rhône-Alpes, F
R
•Ph.D. in Computer Science, UniNE, C
H
•Postdoc and various coordination positions
•Lecturer (Maître-Assistant) at UniN
E
•Co-founded one start-up (SafeCloud Tech sàrl
)
•Co-founded ARM HPC User Group (AHUG)
Career Path
2
2007-2009
2010-2014
2014-2018
2003-2005
2018-today
2017-today
2020-today
2005-2007
4. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
Agenda
4
1.A short but required introduction to TEE
s
2.Some systems we built
3.Lessons learned
if you attended
my talk @ Journees
Securité last week,
you are all set
(repetita juvant)
Let’s make this as interactive as possibl
e
interrupts welcom
e
5. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
Motivating Scenario
5
Intel SGX AMD SEV
•Suppose you want to develop an online service to handle
very sensitive dat
a
•E.g., ECG log
s
•Data privacy is paramoun
t
•Only for allowed stakeholder
s
•Data integrity is paramoun
t
•If data integrity is compromised, risks of false alert
s
•The code being executed must also be con
fi
dentia
l
•E.g., algorithms to compute HR variations and detect
health anomalies
Source: my heart
6. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
Single-host deployment
6
Intel SGX AMD SEV
off-chi
p
hardware
host-os
CPU
hardware attack
s
(cold boot,…)
OS attack
s
(rootkits,..)
in-process attack
s
(memory corruption, ROP)
code
data
Untrusted
Trusted
Lots of bad things!
7. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
7
Intel SGX AMD SEV
off-chi
p
hardware
host-os
CPU
hardware attack
s
(cold boot,…)
OS attack
s
(rootkits,..)
in-process attack
s
(memory corruption, ROP)
enclave code
enclave data
Untrusted
Trusted
TEE
Enclav
e
creation
Single-host deployment
Lots of bad things!
fewer
Enclave
8. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Hardware protected area against powerful attack
s
•The content of the enclaves is shielded from:
•Compromised operating system, compromised system
libraries, attackers with physical access to a machin
e
What is a TEE ?
8
off-chi
p
host-os
CPU
enclave code
enclave data
Enclav
e
creation
Attestatio
fi
dentiality
Integrity
9. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Code and data in the enclave never leave the
CPU package unencrypte
d
➡Outside the CPU, everything is encrypted
Con
fi
dentiality
9
enclave code
enclave data
•When memory is read back into cache lines, the
CPU decrypts
Enclave Page Cach
e
(SGX term)
10. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Code and data in the enclave never leave the
CPU package unencrypte
d
➡Outside the CPU, everything is encrypted
Con
fi
dentiality
10
enclave code
enclave data
•When memory is read back into cache lines, the
CPU decrypts
Enclave Page Cach
e
(SGX term)
CPU
DRAM
11. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Code and data in the enclave never leave the
CPU package unencrypte
d
➡Outside the CPU, everything is encrypted
Con
fi
dentiality
11
enclave code
enclave data
•When memory is read back into cache lines, the
CPU decrypts (with the help of the MME)
Enclave Page Cach
e
(SGX term)
CPU
DRAM MEE
Memory Encryptio
n
Engine (Intel SGX)
Untrusted
encrypted traf
fi
c
12. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•The CPU verify the integrity of cache line
s
•The CPU verify the integrity of virtual-to-
physical addresse
s
•Intel SGX: MME maintains the root of a Merkle
tre
e
•Arm TrustZone: vendor-speci
fi
c.
•Example: Samsung’s Knox uses passive and
active counter-measure
s
•In the case of AMD SEV: no integrity
Integrity
12
CPU vendor-dependant by de
fi
nition (see next)
13. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
Intel SGX
13
Intel SGX AMD SEV
Enclave
Create enclave
Call trusted
function
…
Execute
Return
Call
gate
Trusted function
Untrusted Trusted
➊
➋
➏
➎
➍
➌
➐
Intel SGX
Operating System
•Available since 2015, SkyLak
e
•Hardware-protected area on di
e
•Support strong adversarial model
s
•Split the program in two parts
:
•Untrusted vs. trusted, enclaves
•Code integrity, genuine hardware
•Intel Attestation Servic
e
•Memory limits, EPC, up to 512 MB in recent server-grade
processors, up to 128 MB until recentl
y
•Intel SDK, C/C++, Rust SDK, frameworks for legacy systems
(Scone, SGX-LKL, graphene-sgx, etc.)
14. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Secure Encrypted Virtualizatio
n
•Secure Memory Encryptio
n
•Designed for virtualized systems (VMs
)
•Lack of integrity protectio
n
•SEV-SNP
fi
xing thi
s
•Attestatio
n
•Requires in-silicon mitigation
?
•To be checked against SEV-SN
P
AMD SEV
14
Call function
…
Trusted
j
AMD SEV
Guest Operating System (VM)
Enclave
Create enclave
Call trusted
function
…
Execute
Return
Call
gate
Trusted function
Untrusted Trusted
➊
➋
➏
➎
➍
➌
➐
Intel SGX
Operating System
Execute
Return
k
l
Operating System
m
n
➀
➁
➂
➃
➄
Intel SGX AMD SEV
EuroSec’18
CCS’19
15. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
• Two-world separation, one TA at the tim
e
• Lack of built-in attestation servic
e
•2~5Mb per TA
TrustZone
15
Normal world Secure world
Host
application
OP-TEE
client
OP-TEE
Linux driver
GP TEE
client API
User
space
Privileged
space
Secure
monitor
Trusted
application (TA)
GP TEE
internal API
OP-TEE
OS
TEE
REE
16. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
• Risc-V
:
• MultiZon
e
• KeySton
e
• Pengla
i
•Since 2017, Google’s Titan M on Android Pixel (since v3
)
•IBM SecureBlue & SecureBlue+
+
•Upcoming new ARM Con
fi
dential Compute Architecture (CCA)
Other TEEs
16
Take-away message
:
TEEs are not a silver bullet !
17. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Operations inside TEEs run at bare-metal spee
d
•Strong adversarial models (i.e., compromised OS
)
•Orders of magnitude faster than SotA homomorphic encryption
The Good
17
10
0
10
1
10
2
10
3
10
4
10
5
ADD SUB MUL EXP(k)
Ratio
8−bit 16−bit 24−bit
536ms 544ms
548ms
44ms
HElib
•Microsoft SEAL
•Google Private Join and Compute?
(see SRDS’18)
18. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
• At least in the current incarnations
:
1. Requires some craft from programmer
s
2. Might lack fundamental properties
3. Performances can be poor (goto 1)
4. Requires good knowledge of system issue
s
5. Continuous stream of side-channel attack
s
• Followed by a stream of mitigations, patches.
.
The Bad
18
Intel won’t
fi
x
(outside threa
t
model of SGX)
Can target several TEEs
19. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
Agenda
19
1.A short but required introduction to TEE
s
2.Some systems we built
3.Lessons learned
End of Part 1
not so Ugly, hopefully
20. V. Schiavoni - Invited Talk - 23.09.21
•Untrustworthy cloud provider
s
•Processing data over the clou
d
•Privacy-preserving real time cardiac data analysis
Secure Stream Processing
of Medical Data
20
joint work with CSEM (Centre suisse d’électronique et microtechnique, Neuchâtel)
and Imperial College London, UK
Fig: Carlos Segarra
21. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
Secure MedTec
h
21
PUB
PUB
PUB
MQTT PubSub
Smart Building
broker
broker
broker
KNX
ZigB
…
PUB
PUB
PUB
MedTech
…
…
…
Subscribers
SUB
SUB
SUB
SUB
notify
notify
TZ
TZ
TZ
pub-su
b
middlewar
e
22. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
Secure MedTec
h
22
PUB
PUB
PUB
MQTT PubSub
Smart Building
broker
broker
broker
KNX
ZigB
…
PUB
PUB
PUB
MedTech
…
…
…
Subscribers
SUB
SUB
SUB
SUB
notify
notify
TZ
TZ
TZ
23. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Smart-building sensor
s
•Med-tech scenarios
Secure MedTec
h
23
PUB
PUB
PUB
MQTT PubSub
Smart Building
broker
broker
broker
KNX
ZigB
…
PUB
PUB
PUB
MedTech
…
…
…
Subscribers
SUB
SUB
SUB
SUB
notify
notify
TZ
TZ
TZ
24. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
• Pub/Sub brokers
• Interact with TZ trusted ap
p
• Clients are IoT things, MQTT known standard
KevlarTZ: Brokers
24
untrusted trusted
REE TEE
Secure Monitor
Mode
TEE Cache
TA Heap Mem.
Tamper Proof Secure Storage
TLS
Endpoint inside TrustZone
init
put
get
del
API
base64 AES cache per.stor.
in-TEE
client
in-REE
clients
KEVLAR-TZ
25. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
• Secure persistent storag
e
•Tamper-proof over REE
fi
le-syste
m
•Alternatively, use Replay Protected Memory Block,
requires hardware suppor
t
• Fast volatile cach
e
•Write-through, additional policies easy to ad
d
•Internal and external API for TA
KevlarTZ: Architecture
25
REE TEE
Secure Monitor
Mode
TEE Cache
TA Heap Mem.
Tamper Proof Secure Storage
TLS
Endpoint inside TrustZone
init
put
get
del
API
base64 AES cache per.stor.
in-TEE
client
in-REE
clients
KEVLAR-TZ
26. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
• Op-TEE, host-app and trusted-app, 791 Lo
C
•Modular implementatio
n
•Persistent storag
e
•Cach
e
•AE
S
•Encoding (base64)
Implementation
26
•Open-source: https://github.com/mqttz/kevlar-tz
27. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Emulation vs. hardwar
e
•QEM
U
•Micro-benchmark
s
•encoding/decoding throughpu
t
•encrypt/decrypt throughpu
t
•Network throughput over TCP
•Macro-benchmark
s
•wrist-sensors for ECG data
Evaluation
27
28. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Process 1 minute of ECG data (5-sec sample on the left
)
•Increasing number of client
s
•Simulate hospital
fl
oo
r
•Not designed for very-large workload
s
•Saturates at 15 client
s
•Cause: lack of true multi-threading in TAs
Processing Input Stream
28
29. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Get random ke
y
•Highlight performance di
ff
erence between volatile and
persistent memory
•miss: go fetch data on persistent tamper-proof storag
e
•hit : fetch from secure memory (2Mb
)
Volatile vs. Persistent
29
30. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
Secure MedTec
h
30
PUB
PUB
PUB
MQTT PubSub
Smart Building
broker
broker
broker
KNX
ZigB
…
PUB
PUB
PUB
MedTech
…
…
…
Subscribers
SUB
SUB
SUB
SUB
notify
notify
TZ
TZ
TZ
31. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
Secure MedTec
h
31
•SGX-Spark, developed at IM
P
•Deployment of Spark jobs inside SGX
enclave
s
•Con
fi
dentiality and integrity of existing
spark jobs
•No need to modify existing job cod
e
Fig: Carlos Segarra
32. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
Secure MedTec
h
32
•Cardiac activity monitoring, EC
G
•Intervals between the R peak
s
•Timestamps to compute the Heart Rate Variability (HRV
)
•HRV algorithms running inside SGX enclave
s
•In our case, developed internally at CSEM
Fig: Carlos Segarra
Source: my heart
Source: my heart
34. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
Secure MedTec
h
34
and Imperial College London, UK
joint work with CSEM (Centre Suisse Electronique et Microtecnique, Neuchâtel)
•End-to-end secure medical data processing platfor
m
•Client-side and shielded MQTT brokers via ARM TrustZone
•Server-side with Intel SG
X
•Took 3 years (2019-2021), involved 8 people (students and
seniors), with very limited budget (in-kind
)
•Lead to several scienti
fi
c peer-reviewed publications
•Computer Science but also Medical Journal
s
•CSEM considered it for production (under discussion
)
35. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
1.Pick the proper TE
E
•SGX on the server-sid
e
•TrustZone on the client sid
e
2.Tech (research proto) was immatur
e
•Spark-SGX did not work in streaming-mode, had to
settle on batc
h
•Drawbacks on the throughpu
t
3.Pick the system name carefully …
MedTech: Lessons Learned
35
The choice could be force
d
but what if not ?
36. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
• SGX-FS:
fi
le-system storage with SGX, sealing
Secure Storage with TEE
36
Ram-FS
RAM EPC
fuse
SgxRam-FS
RAM EPC
fuse
➊
➋
➌
Sgx-FS
RAM EPC
fuse
Write/Read
file
Write/Read
file
•Open-source: https://github.com/dburihabwa/sgx-fs
(CloudCom’18)
•TEE client-side, sealing on the cloud ?
38. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Building user-space
fi
le-systems leveraging SGX is possibl
e
•Manageable overhead adding security features, but
:
•Limit cross-enclave boundaries
•Limit secure memory (EPC) usag
e
•We should have looked more carefully into Intel Protected FS
SGX-FS: Lessons Learned
38
39. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Optimised interface with
fi
le-syste
m
•Legacy apps
•Sqlite, Polybench, ratio to native
WebAssembly in SGX
39
(IEEE ICDE’21) WASM in SGX
•Open-source: https://github.com/JamesMenetrey/unine-twine
40. valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•Optimised also means to extend the standard API
s
•If you go that way, di
ffi
cult (but not impossible) to push
upstream your contribution
s
•Modifying the APIs might require strong standardisation
e
ff
orts, too much for our resource
s
•We did not foresee immediately the future application
s
•Users from the crypto-market world contacted us
Twine: Lessons Learned
40
41. /41 valerio.schiavoni@unine.ch - Lessons using TEEs - 25.10.21
•TEEs becoming increasingly popula
r
• Available on cheap devices on the marke
t
• Cloud providers
•One must trust the hardware provide
r
•Pros/cons (performance, side-channels
)
•Can be used to build a large variety of system
s
•Support for heterogeneous TEEs more future-proof
One Slide to Remember
41
Thanks for your attention
!
valerio.schiavoni@unine.ch