https://delex-conf.com/talks/2020-talks/deep-diving-into-the-dynamic-provisioning-of-glusterfs-volumes-in-k8s-with-heketi
In this talk Artem will make a deep dive into the dynamic provisioning of GlusterFS volumes in kubernetes with Heketi. With overview of mentioned stack and detailed description of solution architecture, build on top of mentioned stack.
Be ready to look into the source code with Artem, and see how Heketi works with GlusterFS. Is it issue-free? Nope for sure. So Artem will go through most common troubleshooting. Describe possible improvements and develop it from the ground-up.
The outcome of the talk: common mistakes in the architecture, conclusions and recommendations.
3. Did you work with GlusterFS and Heketi?
Most popular answers
А что это такое?
Немножко
БГ миловал!
Борис Гребенщиков
Использую в проде
Крутая штука!
4. GlusterFS
GlusterFS is a scalable network filesystem suitable for
data-intensive tasks such as cloud storage and media
streaming.
* - Distributed Glusterfs Volume
- Replicated Glusterfs Volume
- Distributed Replicated Glusterfs Volume
- Striped Glusterfs Volume
- Distributed Striped Glusterfs Volume
5. Heketi
Heketi provides a RESTful management interface which can be used to
manage the life cycle of GlusterFS volumes. With Heketi, cloud services
like OpenStack Manila, Kubernetes, and OpenShift can dynamically
provision GlusterFS volumes.
Kubernetes
Heketi
GlusterFS
Management
Mount volumes
13. Not all information we can find in the docs
And here: https://docs.openshift.com/container-platform/3.11/install_config/persistent_storage/dynamically_provisioning_pvs.html
https://kubernetes.io/docs/concepts/storage/storage-classes/#glusterfs
type provisionerConfig struct {
…
url string
user string
volumeType gapi.VolumeDurabilityInfo
volumeNamePrefix string
thinPoolSnapFactor float32
customEpNamePrefix string
.... }
There is no information here: https://kubernetes.io/docs/concepts/storage/storage-classes/#glusterfs
Fortunatelly, we can see a pretty good description here:
https://github.com/kubernetes/examples/blob/master/staging/persistent-volume-provisioning/README.md
customepnameprefix: By default dynamically provisioned volumes has an endpoint and service created with the naming schema of glusterfs-dynamic-<PVC UUID format. With this
option present in storageclass, an admin can now prefix the desired endpoint from storageclass. If customepnameprefix storageclass parameter is set, the dynamically provisioned
volumes will have an endpoint and service created in the following format where - is the field separator/delimiter: customepnameprefix-<PVC UUID>
Task: custom gluster volumes names
14. External projects as part of Heketi
gorilla/mux
is a powerful URL
router and dispatcher
Library for creating powerful
modern CLI applications as well
as a program to generate
applications and command files
An embedded
key/value database for
Go
BoltDB
apps/glusterfs/dbcommon.go
heketi-cli vloume list
GET HTTP://localhost:8080/volumes/list
JSON
/var/lib/heketi.db
17. How Heketi manage Gluster
executor mock ssh kubernetes
How
manage
Does not send
any commands
out to servers.
Sends
commands to
real systems
over ssh
Send
commands to
k8s api
Purpose Development
and tests
GlusterFS as
separate
servers
Manage
GlusterFS
k8s pods
/etc/heketi/heketi.json
… "glusterfs": {
"executor": "kubernetes",
"db": "/var/lib/heketi/heketi.db"
"kubeexec": {
"host":
"https://kubernetes.default.svc.cluster.local",
"fstab": "/var/lib/heketi/fstab” ….
Custom fstab
for GlusterFS
PODs
18. Possible Heketi database problems
Problem Impact Solution
We have a volume in
GlusterFS, but have no in
Heketi
Lost control
Add information
about the lost volume to db
We have a volume in Heketi,
but have no in GlusterFS
Inconsistent DB
Delete volume from Heketi DB
use heketi-cli or API
Volume settings not same
as in the Heketi DB
Inconsistent DB Fix values in Heketi DB
We replaced the physical
device with GlusterFS bricks
Storage isn’t health
Recreate all volumes
19. If we replace a physical device
Restore LV from the lvm archive. Ex.:
/etc/lvm/archive/vg_0888196c04e9e5f7ad346ba7ec173c01_00141-488962832.vg
Replacing a broken GlusterFS POD – it’s realy easy! Example:
device=$(grep /dev/ $backup | sed -e s/'t'//g -e s/'#'//g | cut -d " " -f3)
device_id=$(grep /dev/ $backup -B1 | head -n1 | cut -d " " -f3)
pvcreate --uuid $device_id $device –norestorefile
vgname=$(grep -E 'vg_.*{' $backup | cut -d " " -f1)
vgcreate $vgname $device
bricks=$(grep brick_ $backup | sed s/'t'//g | cut -d " " -f1)
for brick in $bricks; do
chunksize=256K; tp_name=…; poolmetadatasize=…; size=…
lvcreate -qq --autobackup=y --poolmetadatasize $poolmetadatasize --chunksize
$chunksize --size $size --thin $vgname/$tp_name --virtualsize $size --name $brick"
mkfs.xfs -i size=512 -n size=8192 /dev/mapper/$vgname-$brick
done
Reset all bricks:
gluster v status $volume | grep ${brickpath: -25} | grep " N " &&
gluster volume reset-brick $volume $brickpath start &&
gluster volume reset-brick $volume $brickpath $brickpath commit force
But replacing only a broken device isn’t possible from Heketi. We need to do this manually:
heketi-cli node add --zone=1 --cluster=$heketiClusterID --management-host-name=$newNodeAddress --storage-host-name=$newNodeAddress
heketi-cli device add --name $deviceName --node $newNodeID
heketi-cli node disable $oldNodeID
heketi-cli node remove $oldNodeID
heketi-cli device delete $oldDeviceID
heketi-cli node delete $oldNodeID
20. Good advices
- OpenShift docs (Gluster storage)
- RedHat knowledge base (KB)
No-Cost RHEL Developer Subscription:
https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscription-now-available/
https://github.com/TargetProcess/heketi_tools
3. You can use our scripts for checking and fixing Heketi and Gluster
https://play.google.com/books/reader?id=D5q1DwAAQBAJ&authuser=2&hl=ru
2. Implementation Guide for IBM Blockchain
Platform for Multicloud
1. Feel free to use RedHat resources:
4. RTFM : https://github.com/heketi/heketi
