4. Private & Confidential 4
Storage Status—HDFS
4
Metric Value
Number of Nodes Thousands
Storage Capacity Hundreds PB
Num of Files Billions
Max QPS Hundreds of thousands
5. Private & Confidential 5
Storage Status—Presto
5
Metric Value
Number of Nodes Thousands of instances
TP90 About 2 min
Input Dozens of PB per day
Number of Queries
Hundreds of thousands per
day
12. Private & Confidential 12
Storage Acceleration—Architecture
Kafka HDFS
Audit
HMS
Computing
Application
Operator Hot Table
Cache Manager
Alluxio
HDFS
Load/Unload/m
ount
Path
delete/create
event
Set/Clear
partition
property
Load data
Load Table
Update Policy
Input
Output
Get tag
13. Private & Confidential 13
Presto Query
Log
Hot Table
(Hive Table)
• Partition by date
• Calculate the number of visits of table
every day
Storage Acceleration—Hot Table
14. Private & Confidential 14
• Scheduled every day
Hot Table
Most frequently
visited weighted
tables in the last
seven days
Recent m
partitions
every table
Load from
HDFS to
Alluxio
Persist
relationship
Set tag in
HMS
Note: It is an alpha version and the
subsequent iterations will be optimized
continuously
Storage Acceleration—Update Policy
15. Private & Confidential 15
HDFS
Alluxio
HMS
Presto
On Alluxio No tag
• key:cache,
value:${DC}/Alluxio/ebj@${Alluxio_nameservice}
• If partition exists, set property in partition
property
• Else, set property in table property
Storage Acceleration—HMS Tag
18. Private & Confidential 18
PATH
mount
unmount
load
query
HIVE TABLE
mount
unmount
load
load all the
recorded paths
load recent n
partitions
query
ADMIN
monitor and operator
Storage Acceleration—REST API
20. Private & Confidential 20
• 6 merged, 2 WIP, 1 fixed by Alluxio.
TYPE PR STATUS
Hadoop 2.10
Fix HdfsVersion miss hadoop 2.10 config merged
Fix integration/yarn/pom.xml enforcer-plugin miss hadoop
2.10.x config
merged
Fix common.go miss hadoop 2.10 configuration merged
Command Line
Improve shell command support ebj nameservice merged
Fix for Alluxio.logs.dir
fixed by
Alluxio
Web Page
Fix isMounted should not invoke ufs, if not /metrics page very
slowly
merged
Fix FormatUtils.getSizeFromBytes method should supports EB merged
NameServices
Fix unescape the ufs url of Alluxio fsadmin report metrics
result
WIP
Metrics Fix cache radio total not include cacheMisses WIP
Storage Acceleration—Community Contribution
21. Private & Confidential
Private & Confidential 21
1 Storage Situation
2
3
4
Storage Acceleration
Storage Servitization
Future Plan
Storage Acceleration and Servitization at Shopee
22. Private & Confidential
Private & Confidential 22
Storage Servitization—Status
▪ Most of data is stored in HDFS
▪ Various development languages are used
▪ HDFS has insufficient support for non Java clients
23. Private & Confidential
Private & Confidential 23
Fuse for HDFS
S3 for HDFS
▪ Deploy alluxio fuse service on physical machine
▪ Deploy alluxio fuse service on kubernetes cluster
▪ Using S3 API to access alluxio proxy service
Storage Servitization—Solutions
24. Private & Confidential
Private & Confidential 24
▪ Kernel
▪ User-level daemon
High-Level Architecture
Storage Servitization—Fuse
WHAT IS IT
▪ FileSystem in Userspace
25. Private & Confidential
Private & Confidential 25
▪ libfuse
▪ JNR-Fuse
▪ JNI-Fuse
Requirements
Implementation
Storage Servitization—Alluxio Fuse
▪ Standalone Fuse
▪ Fuse on Workers
Deployment
▪ Not support random writes
Limitations
26. Private & Confidential
Private & Confidential 26
Store Servitization—Alluxio CSI
▪ On nodeserver pod
▪ On separate pod(new feature)
Fuse Deployment mode
WHAT IS IT
▪ Standard storage interface for
containers
27. Private & Confidential
Private & Confidential 27
▪ Fuse sidecar container in a Pod to mount the
Alluxio directory
▪ Independent configuration of pods, high flexibility
▪ Each Pod runs a Fuse container without affecting
each other
▪ Each Fuse process occupies a container, so the
solution consumes more resources
Futures
Store Servitization—k8s sidecar for Alluxio
WHAT IS IT
28. Private & Confidential
Private & Confidential 28
Store Servitization—Summarize
Fuse on
K8s-csi
K8s-sidecar
Fuse on nodeserver
pod
Fuse on separate pod
maintenance
cost
high low higher higher
resource usage low lower high high
independence high low high high
29. Private & Confidential
Private & Confidential 29
▪ Bucket: A bucket is a container for objects stored in Amazon S3
▪ Object: Objects are the fundamental entities stored in Amazon S3
▪ Key: An object key (or key name) is the unique identifier for an
object within a bucket.
▪ Region: You can choose a region to store the created buckets
Store Servitization—S3
Buckets
Objects
Keys Regions
Amazon
S3
Concepts
Conception
30. Private & Confidential
Private & Confidential 30
▪ Alluxio can mount HDFS data
▪ Alluxio provides Proxy service
▪ Proxy is compatible with the basic operations of the S3 API
▪ S3 SDK supports many development languages
Store Servitization—S3 for HDFS
Access HDFS data via Alluxio using S3 protocol
31. Private & Confidential
Private & Confidential 31
▪ 1-level directory as bucket
▪ Subdirectories and file paths as key
Store Servitization—Alluxio Proxy for S3 mapping
34. Private & Confidential
Private & Confidential 34
Store Servitization—Community contribution
TYPE PR STATUS
proxy
Fix wrong format of s3 bucket creationDate merged
Support parse authorization headers for s3 proxy WIP
fuse
Fix wrong method call to get username and wrong
parameter assignment
merged
csi Replace invalid env with args in nodeserver merged
doc
Fix bug case of S3 REST API merged
Fix wrong file name in k8s doc merged
Fix ambiguous description for impersonation in CN
doc
merged
ozone Update ozone from 1.1.0 to 1.2.1 closed
▪ 6 merged, 1 WIP, 1 closed.
35. Private & Confidential
Private & Confidential 35
1 Storage Situation
2
3
4
Storage Acceleration
Storage Servitization
Future Plan
5 Future Plan
Storage Acceleration and Servitization at Shopee
36. Private & Confidential
Private & Confidential 36
▪ Speed up Spark and Hive
▪ Implement adaptive cache policy on CacheManager
▪ Support more POSIX APIs
▪ Optimize CSI
Storage Service
Future Plan
Storage speed up