The boom of artificial intelligence brought to the market a set of impressive solutions both on hardware and software sides. On the other hand, massive implementation of AI in various areas brings about problems, and security is one of the greatest concerns. The speaker will present results of hands-on vulnerability research of different components of AI infrastructure, including NVIDIA DGX GPU servers, ML frameworks, such as PyTorch, Keras, and TensorFlow, data processing pipelines and specific applications, including medical imaging and face recognition–powered CCTV. Updated Internet Census toolkit based on the Grinder framework will be introduced.
3. Disclaimer
Please note, that this talk is by Sergey and Hacking Odyssey group.
We don't speak for our employers.
All the opinions and information here are of our responsibility. So, mistakes and bad
jokes are all OUR responsibilities.
3https://github.com/sdnewhophttps://scada.sl/
Hacking Odyssey Group
Sergey Gordeychik
Anton Nikolaev
Denis Kolegov
Maria Nedyak
Roman Palkin
Hacking Odyssey Projects
Grinder Framewrok
AISec
DICOM Sec
SD-WAN New Hop
19. 19
James Mickens, Harvard University, USENIX Security '18-Q: Why
Do Keynote Speakers Keep Suggesting That Improving Security Is
Possible?
https://www.youtube.com/watch?v=ajGX7odA87k
20. 20
Mission-centric Cybersecurity
Gapanovich, Rozenberg, Gordeychik, Signalling cyber security: the need for a mission-centric approach
https://www.railjournal.com/in_depth/signalling-cyber-security-the-need-for-a-mission-centric-approach
a process that ensures
control object operation with
no dangerous failures or
damage, but with a set
economic efficiency and
reliability under adversarial
anthropogenic information
influence
30. AIFinger Project
The goals of the project is to provide tools and results of passive and active fingerprinting of
Machine Learning Frameworks and Applications using a common Threat Intelligence
approach and to answer the following questions:
How to detect ML backend systems on the Internet and Enterprise network?
Are ML apps secure at Internet scale?
What is ML apps security level in a general sense at the present time?
How long does it take to patch vulnerabilities, apply security updates to the ML
backend systems deployed on the Internet?
sdnewhop.github.io/AISec/
github.com/sdnewhop/AISec
Contributors:
● Sergey Gordeychik
● Anton Nikolaev
● Denis Kolegov
● Maria Nedyak
31. AIFinger Project Coverage
Frameworks
○ TensorFlow
○ NVIDIA DIGITS
○ Caffe
○ TensorBoard
○ Tensorflow.js
○ brain.js
○ Predict.js
○ ml5.js
○ Keras.js
○ Figue.js
○ Natural.js
○ neataptic.js
○ ml.js
○ Clusterfck.js
○ Neuro.js
○ Deeplearn.js
○ Convnet.js
○ Synaptic.js
○ Apache mxnet
Databases with ML Content
○ Elasticsearch with ML data
○ MongoDB with ML data
○ Docker API with ML data
Databases
○ Elasticsearch
○ Kibana (Elasticsearch
Visualization Plugin)
○ Gitlab
○ Samba
○ Rsync
○ Riak
○ Redis
○ Redmon (Redis Web UI)
○ Cassandra
○ Memcached
○ MongoDB
○ PostgreSQL
○ MySQL
○ Docker API
○ CouchDB
Job and Message Queues
○ Alibaba Group Holding AI Inference
○ Apache Kafka Consumer Offset Monitor
○ Apache Kafka Manager
○ Apache Kafka Message Broker
○ RabbitMQ Message Broker
○ Celery Distributed Task Queue
○ Gearman Job Queue Monitor
Interactive Voice Response (IVR)
○ ResponsiveVoice.JS
○ Inference Solutions
Speech Recognition
○ Speech.js
○ dictate.js
○ p5.speech.js
○ artyom.js
○ SpeechKITT
○ annyang
Measuring Artificial Intelligence and Machine Learning Implementation Security on the Internet
https://www.researchgate.net/publication/337771481_Measuring_Artificial_Intelligence_and_Machine_Learning_Implementation_Security_on_the_Internet
36. 36
Tensorboard
…
Everything
+ vulns
The TensorFlow server is meant
for internal communication only.
It is not built for use in an
untrusted network.
Totally more than 120
results
45. 45
Ok, let’s scan!
Nmap scan report for X.X.X.X
Host is up (0.010s latency).
Not shown: 991 closed ports
PORT STATE SERVICE VERSION
22/tcp open ssh OpenSSH 6.0p1 Debian 4 (protocol 2.0)
80/tcp open http lighttpd
427/tcp open svrloc?
443/tcp open ssl/http lighttpd
623/udp open ipmi
554/tcp filtered rtsp
1723/tcp filtered pptp
5120/tcp open barracuda-bbs?
5988/tcp open wbem-http?
5989/tcp open ssl/wbem-https?
48. 48
I have only one question!
http://www.demotivation.us/i-have-only-one-question-1267735.html
Why it
still
enabled
by default
in 2020?
What do
you
need a
helmet
for?
How the complex password will help?!!
49. 49
Strange certificate
Issued by Quanta Computers Inc?
128 bytes (1024) RSA key?..
Issued 17 of April 2017…
Same serial over the Internet!!!
50. 51
Find and decode firmware
Google for Quanta Computers BMC firmware
binwalk
7-zip
Voilà
51. 52
Grep the cert and keys
TLS services on BMC uses RSA 1024
with weak cyphers, default Diffie-
Hellman primitives.
The private/public keys are hardcoded
in firmware and are the same for many
instances of
Quanta Computers BMC, including
NVIDIA DGX-1.
Public and private keys can be found
unencrypted in
Firmware.
This allow passively decrypt network
communications without MITM
conditions.
52. 53
Other greps?
NetNTLMv2: 28912.2 MH/s
MD5: 450.0 GH/s
SHA-256: 59971.8 MH/s
MS Office 2013: 163.5 kH/s
bcrypt $2*$, Blowfish (Unix): 434.2 kH/s
Can we use DGX to bruteforce DGX password hash?!
57. 58
Lesson learned
• Please don’t use one way hashing with salt. Use plaintext or reversible
encryption.
• Password encryption key should be hardcoded and stored in same folder as a
user database.
• It is important to keep it like the product name.
• Store it in several places across the filesystem for resilience.
58. 59
Hardcoded RC4 Key in JViewer-SOC
• JViewer-SOC (KVM and IPMI applet) use RC4 cipher with a hardcoded key for traffic
encryption.
• In the JViewer-SOC java applet com.ami.kvm.jviewer.soc.video package contains Decoder
• class.
• This class defines DecodeKeys constant which is equal to “fedcba9876543210”.
• Constant is used to initialize RC4 key scheduling (expansion) algorithm.
This allows an attacker to bypass security features, decrypt traffic and extract sensitive
information.
59. 60
Insecure random number generator in RAKP/AES
• JSOL.jar/com/ami/jsol/common/Util.java defines functions random4ByteArray
and random16ByteArray.
• The Random function from java.util.Random class is used.
• These functions are used within RAKP crypto protocol implementation.
• According to the specification of the RAKP it is based on Bellare-Rogaway
protocols .
• The issue is that the 1 protocols require random numbers in cryptographically
sense.
The same function is used to generate IV for AES encryption in the processEncryption function
of IPMISession class.
60. 61
CSRF is not an issue….
A vulnerability to Cross-Site Request Forgery (CSRF) attack was found in the Nvidia BMC
Web Service. It allows an attacker to force an authenticated user to execute the API
endpoints within the web application.
There is a list of internal queries which require active session authentication and don’t
require CSRF token.
/rpc/ getsessiontoken .asp
/rpc/ getrole.asp
/rpc/ getadvisercfg.asp
/rpc/ getvmediacfg.asp
/rpc/ flash_browserclosed.asp
/rpc/ getvideoinfo.asp
/rpc/ getsessiontoken.asp
/rpc/ getrole.asp
/rpc/ downloadvideo.asp
/rpc/ restarthttps.asp
/rpc/ getvmediacfg.asp
/rpc/ getadvisercfg.asp
61. 62
Unrestricted SingImage key upload
SingImage upload feature in DGX-1 BMC accept any correct RSA 1024 public key without any verification.
This key is used to verify firmware signature.
SignImage upload routine, implemented in libifc.so.2.42.0 WebValidateSignImageKey function accept any
correct RSA 1024 public key without any verification of authenticity of the key and store it in the
/conf/public.pem.
CheckImageSign function implemented in libipmimsghndlr.so use public.pem to verify firmware signature.
62. 63
Unrestricted File Upload through CSRF
Web-server handler libmodhapi.so defines stripped function at 0x8BE0
address. This function is being called when an authorized user sends POST request to
/page/file_upload.html .
If a POST request is multipart/form-data this function checks for file argument and if its name
doesn’t end with a ‘/’ symbol¨ looks up for a file path in the hardcoded fille-argument-name-to-
file-path mapping.
However if the argument name ends with ‘/’¨ file is being saved at the file system defined as file
argument name filename.
Thus it is possible to upload custom files and overwrite existing ones with user-defined
absolute path.
Example attack vector - overwrite ./shadow or ./passwd file in the “/conf/” folder to create/modify
users and/or replace default shell to get remote root access via ssh.
Vulnerability can be exploited via CSRF.
67. 68
Disclosure timeline
Tue, 3 Sep 2019, 16:42 – Initial submission
Thu, 19 Sep 2019, 00:40– List of internet-faced DGXs collected by Grinder
Sun, 22 Sep 2019, 23:05 – Ack and workaround discussion
Sat, 5 Oct 2019, 19:50 – Remote root submission
Tue 17 Dec 2019, 21:00 – Call with Alex Matrosov to discuss soooo responsible
disclosure
Feb 2020 – COVID 19 outbreak, cancellation of PHDays and OFFZONE
April – Aug 2020 – GradeZero Rock’n’roll
Tue, 25 Aug, 21:10 – Failed fix (QA issues)
Now – Fixes, Initial disclosure @CodeBlue 2020
Kudos to Alex, Shawn, NVIDIA PSIRT
68. 69
Supply chain is a pain
Megarac SP (DGX-1)
Quanta Computer Inc.
IBM (BMC Advanced System Management)
Lenovo (ThinkServer Management Module)
Hewlett Packard Enterprise Megarac
Mikrobits (Mikrotik)
Megarac SP-X (DGX-2)
Netapp
ASRockRack IPMI
ASUS ASMB9-iKVM
DEPO Computers
TYAN Motherboard
Gigabyte IPMI Motherboards
Gooxi BMC
69. 70
Takeaways
• Big Thing doesn’t mean good security
• Good AI researches are bad cybersec pro
• All vulnerabilities are important
• Supply chain is a pain
• Things are better with Grinder
70. 71
Infection of the AI models
http://www.scada.sl/2019/11/malign-machine-learning-models-and-bad.html
72. Pre-trained model workflow
1. Model
interface (some
wrapper, cli,
etc.)
.py / .sh /
etc
2. Download the
weights in some
form
3. Run the
model
.pb / .h5 / .pth
.json / .yml
/.csv
78. Step 2. Infect it!
Overwrite
the magic
number
`Classic` Pickle
payload
Python code to
execute on load
Shell code
to run on
load
79
79. Python Pickle Injection
Pickle is a python package used to 'serialize' an
object to string format and store them to or
load from a file.
Pickle is a simple stack language, which means
pickle has a variable stack.
• Every time it finished 'deserializing' an object it
stores it on the stack.
• Every time it reaches a '.' while 'deserializing', it
pop a variable from the stack.
Besides, pickle has a temporary memo, like a
clipboard.
'p0', 'p1' means put the top obj on the stack to
memo and refer it as '0' or '1'
'g0', 'g1' act as get obj '0' or '1'
Pickle has two packages: pickle and cPickle,
they have some specific differences like
different methods, but most of the case they
act in the same way.
http://xhyumiracle.com/python-pickle-injection/
85. Custom serialization
•Protobuf format (.pb)
•~1300 operations (math, conditionals, statistics, etc.)
•Only TWO of them were found dangerous
•WriteFile (any text, any file)
•ReadFile (any file)
18
Looks like Google
is aware of them
94. Timeo Danaos et dona ferentes
https://github.com/pytorch/pytorch/issues/31875
`torch.load()` uses ``pickle`` module implicitly, which is known to be
insecure. It is possible to construct malicious pickle data which will
execute arbitrary code during unpickling. Never load data that could have
come from an untrusted source, or that could have been tampered with.
**Only load data you trust**.
97. Face recognition
170 000 cameras across the city
Face recognition system based
on FindFace technology
The current face recognition
system operates on the "black
lists" (criminals, missed people)
The system does not compare
all people caught in the camera
with all residents of Moscow!
98. Let’s check it out!
• Segmentation dons not works
• Or works, but with poor accuracy
• Questions
• The presence of a biometric DB
• The relevance of the biometric DB
• Biometric attacks
• Use of masks, etc.
• False positive handling
https://www.betafaceapi.com/
99. Biometric DB
White List (anyone you can)
• Upload photos via the app
Blacklist (not allowed)
• Register when a COVID is
detected
• Other citizens ???
Where to get?
How to compare with the
person?
103. 104
What can we do?
For Researchers
AI Cybersecurity is Green Field
From SDN to Model Privacy, from Secure SDL to Adversarial
Robustness
For Enterprises
Don’t trust AI if adversarial “input” is possible
AI IS NOT spherical model traveling in a vacuum!
For Governments
Centralize data and annotation
Force vendors to follow security best practices from the beginning
Detect and control AI-based abuses
