x86_64 Hardware Deep dive 06th May, 2020 SAKURA Internet Research Center. Senior Researcher / Naoto MATSUMOTO

1. x86_64 Hardware Deep dive
6th May, 2020 SAKURA Internet, Inc. Research Center SR / Naoto MATSUMOTO
(C) Copyright 1996-2020 SAKURA Internet Inc

2. x86_64 Hardware Deep dive cheatsheet
2
# uname -sr; tail -1 /etc/lsb-release
Linux 4.15.0-72-generic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
Setup commands
# apt update
# apt upgrade -y
# apt install -y hwloc hwloc-nox
# tar xzvf ./mlc_v3.7.tgz (Download from Intel.com)
# update-pciids
Usage commands
# lscpu
# lshw
# lstopo
# lspci –tv
# Linux/mlc
SOURCE: SAKURA Internet Research Center (2020/05)

3. GCC 9.2 Install TIPS
3
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc4/linux-headers-5.3.0-050300rc4_...201908111734_all.deb
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc4/linux-headers-5.3.0-050300rc4-...201908111734_amd64.deb
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc4/linux-modules-5.3.0-050300rc4-...201908111734_amd64.deb
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc4/linux-image-unsigned-5.3.0-050300rc4-...11734_amd64.deb
# dpkg -i linux-headers-5.3.0-050300rc4_5.3.0-050300rc4.201908111734_all.deb
# dpkg -i linux-headers-5.3.0-050300rc4-generic_5.3.0-050300rc4.201908111734_amd64.deb
# dpkg -i linux-modules-5.3.0-050300rc4-generic_5.3.0-050300rc4.201908111734_amd64.deb
# dpkg -i linux-image-unsigned-5.3.0-050300rc4-generic_5.3.0-050300rc4.201908111734_amd64.deb
# sync; sync; sync; reboot
# uname -sr; tail -1 /etc/lsb-release
Linux 5.3.0-050300rc4-generic
DISTRIB_DESCRIPTION="Ubuntu 19.04"
# apt install build-essential libgmp-dev libmpfr-dev libmpc-dev zlib1g-dev numactl -y
# wget http://ftp.tsukuba.wide.ad.jp/software/gcc/releases/gcc-9.2.0/gcc-9.2.0.tar.gz
# tar xzvf ./gcc-9.2.0.tar.gz
# ../configure --prefix=/usr --disable-multilib --with-system-zlib --enable-languages=c,c++,d,fortran,go,objc,obj-c++
# make -j12
# make install
# gcc --version
gcc (GCC) 9.2.0
SOURCE: SAKURA Internet Research Center (2020/05)

4. PCIe P2P DMA Install TIPS
4
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2-rc6/linux-headers-5.2.0-050200rc6...201906222033_all.deb
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2-rc6/linux-headers-5.2.0-050200rc6...201906222033_amd64.deb
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2-rc6/linux-modules-5.2.0-050200rc6...201906222033_amd64.deb
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2-rc6/linux-image-unsigned-5.2.0-050200rc6...22033_amd64.deb
# dpkg -i linux-headers-5.2.0-050200rc6_5.2.0-050200rc6.201906222033_all.deb
# dpkg -i linux-headers-5.2.0-050200rc6-generic_5.2.0-050200rc6.201906222033_amd64.deb
# dpkg -i linux-modules-5.2.0-050200rc6-generic_5.2.0-050200rc6.201906222033_amd64.deb
# dpkg -i linux-image-unsigned-5.2.0-050200rc6-generic_5.2.0-050200rc6.201906222033_amd64.deb
# sync; sync; sync; reboot
# apt install -y nvme-cli libbobcat-dev
# git clone http://www.github.com/sbates130272/p2pmem-test
# cd p2pmem-test/
# ./p2pmem-test /dev/nvme0n1 /dev/nvme1n1 /dev/p2pmem0 -s 1 -4k --check
Copy one 4KB chunk from /dev/nvme0n1 to /dev/nvme0n1 via the memory
exposed by /dev/p2pmem0. Perform a check on the data (i.e. write know
data to /dev/nvme0n1 and validate that by reading /dev/nvme1n1 after
the p2pmem based transfer).
https://github.com/sbates130272/donard (DRAFT)
# cd /nvme_donard
# cd /libdonard
# ./waf
# cd /libdonard/build/speed
# dd if=/dev/zero of=/<nvme_drive>/test1.dat bs=1K count=100K
# ./nvme2gpu_read -b 128M -D /<nvme_drive>/test1.dat
# ./nvme2gpu_read -b 128M =/<nvme_drive>/test1.dat
SOURCE: SAKURA Internet Research Center (2020/05)

5. PCIe 4.0 NVMe SSD(M.2) RAID0 TIPS
5
# lscpu
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 5 3600 6-Core Processor
# apt install nvme-cli fio -y
# update-pciids
# lspci -tv
# lspci -vv
# nvme list
Node Model
------------ ---------------------------
/dev/nvme0n1 GIGABYTE GP-ASM2NE6100TTTD
/dev/nvme1n1 GIGABYTE GP-ASM2NE6100TTTD
:
/dev/nvme3n1 Force MP600
# mdadm --create /dev/md0 --raid-devices=3 --level=0 /dev/nvme[013]n1
# mdadm --detail /dev/md0
Number Major Minor RaidDevice State
0 259 0 0 active sync /dev/nvme0n1
1 259 1 1 active sync /dev/nvme1n1
2 259 5 2 active sync /dev/nvme3n1
# mkfs.ext4 /dev/md0
# mkdir /mnt/gen4
# mount /dev/md0 /mnt/gen4/
# fio --directory=/mnt/gen4/ --bs=4k --rw=read --size=20G --numjobs=16 --group_reporting --name=test
read: IOPS=2553k, BW=9973MiB/s (10.5GB/s) ...
# lsmem
Total online memory: 16G
SOURCE: SAKURA Internet Research Center (2020/05)

6. PCIe 4.0 NVMe SSD(M.2) SMART-LOG TIPS
6
# uname -sr; tail -1 /etc/lsb-release
Linux 4.15.0-58-generic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
# nvme list
Node SN Model Format FW Rev
---------------- -------------------- ---------------------------------------- ---------------- --------
/dev/nvme0n1 SN192608967864 GIGABYTE GP-ASM2NE6100TTTD 512 B + 0 B EGFM11.0
/dev/nvme1n1 SN192608967963 GIGABYTE GP-ASM2NE6100TTTD 512 B + 0 B EGFM11.0
/dev/nvme2n1 1924822900012855002D Force MP600 512 B + 0 B EGFM11.0
/dev/nvme3n1 PHBT7160022A032E INTEL MEMPEK1W032GA 512 B + 0 B K3110300
# nvme smart-log /dev/nvme2n1
Smart Log for NVME device:nvme2n1 namespace-id:ffffffff
critical_warning : 0
temperature : 36 C
available_spare : 100%
available_spare_threshold : 5%
percentage_used : 0%
data_units_read : 4,493,212
data_units_written : 1,336,869
host_read_commands : 41,019,727
host_write_commands : 4,432,398
controller_busy_time : 22
power_cycles : 78
power_on_hours : 44
unsafe_shutdowns : 45
media_errors : 0
num_err_log_entries : 98
Warning Temperature Time : 0
SOURCE: SAKURA Internet Research Center (2020/05)

7. Null block device High Performance I/O TIPS
7
# apt update
# apt upgrade; sync; sync; sync; reboot
# uname -sr; tail -1 /etc/lsb-release
Linux 4.15.0-58-generic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
# lscpu
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 5 3600 6-Core Processor
# rmmod null_blk
# modprobe null_blk nr_devices=8
# fio --bs=4k --size=128G --rw=read --numjobs=8 --runtime=40 --group_reporting=1
--name=nullb0 --filename=/dev/nullb0
--name=nullb1 --filename=/dev/nullb1
--name=nullb2 --filename=/dev/nullb2
--name=nullb3 --filename=/dev/nullb3
--name=nullb4 --filename=/dev/nullb4
--name=nullb5 --filename=/dev/nullb5
--name=nullb6 --filename=/dev/nullb6
--name=nullb7 --filename=/dev/nullb7
:
read: IOPS=7201k, BW=27.5GiB/s (29.5GB/s)(384GiB/13980msec)
# mount -t tmpfs -o size=8g tmpfs /mnt/ram
# fio --bs=4k --size=500m --rw=read --numjobs=8 --runtime=40 --group_reporting=1
--name=tmpfs --directory=/mnt/ram/
:
read: IOPS=6169k, BW=23.5GiB/s (25.3GB/s)(6000MiB/249msec)
SOURCE: SAKURA Internet Research Center (2020/05)