1. Software Defined Memory for Oracle
Databases (Deep Dive)
Efstathios Efstathiou
Master Database Engineer
Federal Office of Information Technology, Systems and Telecommunication FOITT
4. What is Software Defined Memory?
In an ideal world
Cost wouldn’t be problem
We would have infinite memory ressources
Memory would be persistent
All data would equally treated (no cold and hot data)
So no need for:
Caching and buffering
No fancy algorhythms
No tiering
Less complicated statistics structure
5. What is Software Defined Memory?
In the past
We only had a very slow and a very fast
tier
HDD
RAM
There was a huge gap in performance
Throughput could be cured by adding
lots of spindles by some extent
But what about latency?
8. What is Software Defined Memory?
Storage and Memory Hierarchy Today
9. What is Software Defined Memory?
Storage and Memory Hierarchy Tomorrow
10. What is Software Defined Memory?
Back from the the grave: Virtual Memory (1/4)
11. What is Software Defined Memory?
Back from the the grave: Virtual Memory (2/4)
First form of Tiering and overprovisioning
Brought us virtual address space
Couldn’t handle alot of load with spinning disk
Memory pages are small and there are alot of them
Access time was big issue
12. What is Software Defined Memory?
Back from the the grave: Virtual Memory (3/4)
SCSI protocol Based SSD made it more attractive, but:
The higher speed on an old / emulated proctol (SCSI) ate CPU and
added latency
SATA/SAS SSD NVMe
13. What is Software Defined Memory?
Back from the the grave: Virtual Memory (4/4)
3D Xpoint DIMMs and PCIe SSD with lower latencies can
futher close the gap, making virtual memory more
appealing than ever.
14. What is Software Defined Memory?
Making Memory persistent
To use all these new technologies
there is only one thing to add:
Make RAM presistent
Non-Volatile RAM to the rescue !
Used small quantities on the Storage
Industy for years (e.g. your Netapp’s
NVRAM)
Battery Backed
Copies your Data in RAM to a Flash
Chip, so it can be restored after
power loss
15. What is Software Defined Memory?
Now we got all the hardware, but what about
API’s?
NVMe Drives are available today for all operating systems
Persistent Memory (e.g. NVDIMM) devices came with a
propiertary driver in the past
There an new standard API for Linux for persistent
memory devices => PMEM.IO
Standartization
Easier Programming, but still alot of effort
Are there any commercial/production ready solutions?
16. SDM Vendors
Currently there two major SDM Vendors
Plexistor
Complete SDM Solution
ScaleMP
FLX Flash exentsion for ScaleMP vSMP Foundation
Very different concepts/visions of SDM
17. SDM Vendors
Plexistor
Redesigned the block-based filesystem to suit the
memory-based storage by extending the VFS data
structures
Uses the persistent memory library (PMEM.IO)
Supports Tiering between NVDIMM and Backend, e.g. 3D Xpoint
Flash Dimm or NVMe PCIe SSD
Supports Remote Addressing Memory over RDMA (PM Brick)
Exposes SDM as Filesystem (M1FS) without buffering
Very focussed on Latency
19. SDM Vendors
ScaleMP
Orginally designed to provide a Virtual SMP System
Spans multiple server together a big machine
Accesses their cpu/memory ressources over Infiniband / RDMA
Extends the virtual memory (VM) subsystem, instead of
the VFS
Requires first booting into ScaleMP Hypervisor (like Xen/OVM
Dom0)
Abstraction treats Flash as virtual NUMA node providing more
RAM to your vSMP System (like DomU/VM)
21. Use Cases for Oracle
We identified the following use cases
Optimizing/Boosting Oracle Database I/O Performance
without In-Memory Option
Giving Oracle faster and leaner I/O Stack that provides better
troughput at lower latency
Provide more memory for Oracle In-Memory Option
SIMD processing on SDM with Flash Tier as backend
22. Use Cases for Oracle
Optimizing/Boosting Oracle Database I/O
Performance without In-Memory Option
Requires SDM to be exposed as filesystem or block device
Plexistor exposes as filesystem /m1fs
Simple (just copy database to /m1fs) ☺
Reduce Buffer Cache to minimum
Bypasses page cache ☺
I/O is done trough kernel threads => fast ☺
ScaleMP exposes as RAM
Ramdisk as block device with e.g. Rapiddisk or tmpfs
Not persistent
23. Use Cases for Oracle
Provide more memory for Oracle In-Memory
Option
Requires SDM to be exposed as RAM or Virtual RAM
Plexistor exposes as filesystem /m1fs
Create a Swap File or Memory File on M1FS on the physical host
Create a VM with overprovisioned ressources or virtual dimm with
memory file (currently only KVM)
ScaleMP exposes as RAM per se
Boot ScaleMP (like Dom0)
Startup vSMP Virtual System / VM which is like a DomU/VM with
virtual dimms added ☺
24. Use Cases for Oracle
Provide more memory for Oracle In-Memory
Option
How does Oracle do it on their systems?
25. Use Cases for Oracle
Decisions made
We decided to first optimzing database I/O
Applies to more database installations
Saves most licenses
We choose Plexistor and M1FS for this task
Providing more memory for In-Memory Option is currently
tested in our lab
Comparing ScaleMP and Plexistor approaches
26. Benchmarks (Live Demo)
Purpose
Determine Performance
using M1FS
Advantages of byte
addressable memory
Using kernel threads for I/O
Find the break even / slow
down factors when tiering
What is the best byte address
able to block addressable
device ratio?
27. Benchmarks (Live Demo)
Plexistor Hardware Setup
HP ML 350 G9
2x NVDIMM Modules (Intel/Micron)
Single Node, no cluster
Centos 7.2
Plexistor SDM 1.9
SDM/M1FS with 216 GB
Oracle 12.1.0.2
Calibrate I/O random read
28. Benchmarks (Live Demo)
Reference benchmark with FlashGrid
Setup
2x HP ML 350 G9
Intel NVMe SSDs
Oracle Linux 7.2
FlashGrid 16.7
Oracle 12.1.0.2
Oracle RAC Option
Calibrate I/O random read
30. Now Plexistor
Benchmarks (Live Demo)
Limited by number of NVDIMMs / memory channels per dimm
Single Server (superior performance to 2 Node RAC)
31. Benchmarks (Live Demo)
Analysis
Plexistor is very fast
Bypassing traditional I/O stack
Giving quite a boost due to using kernel threads
Your 2nd tier should be as fast as possible, too
3D Xpoint DIMMs (Tier2)
Up to a cache miss ratio of 10%:
System runs at max possible IOPS
Corresponds 2x the size of Tier 1 => good caching algorhythm
33. Conclusion
Software Defined Memory
Is possible thanks to new hardware technologies reducing
latency
Can massively improve your Oracle Database performance
without buying additional extra cost options
Can provide an high performant alternative Oracle’s
Engineered systems solutions
