Abstract: This talk will introduce you to the concept of Kubernetes Volume plugins. We will not only help you understand the basic concepts, but more importantly, using practical examples, we will show how you can develop your own volume plugins and contribute them back to the community of the OSS project as large as Kubernetes. We will conclude the talk by discussing various challenges one can come across when contributing to a high velocity OSS project of Kubernetes' size which can help you avoid the pain and enjoy the path. Sched Link: http://sched.co/6BYB

1. Custom Volume
Plugins
@agonzalezro

2. Just one thing

3. It's easy!

4. Easier than you might think

5. What's a
volume?

6. Persistent
vs
Non persistent

7. User POV

8. spec:
containers:
- name: web
image: nginx
ports:
- name: web
containerPort: 80
volumeMounts:
- name: www-root
mountPath: "/usr/share/nginx/html"
volumes:
- name: www-root
flocker:
datasetName: my-flocker-vol

9. ...
volumeMounts:
- name: www-root
mountPath: "/usr/share/nginx/html"
volumes:
- name: www-root
flocker:
datasetName: my-flocker-vol

10. Available
4 AWS EBS, GCE PD, Azure File
4 Git Repo
4 NFS
4 GlusterFS, Cinder
4 Flocker
4 Secrets

11. Is this
new?

12. HTTP API

13. /VolumeDriver.Create
.Mount
.Path
.Unmount
.Remove

14. Go interface{}

16. 1. Add it to the API (pkg/api/{,v1/}types.go):
type VolumeSource struct {
EmptyDir *EmptyDirVolumeSource `json:"emptyDir,omitempty"`
Flocker *FlockerVolumeSource `json:"flocker,omitempty"`
...

17. 2. Add it to the Kubelet (cmd/kubelet/app/plugins.go):
func ProbeVolumePlugins(pluginDir string) []volume.VolumePlugin {
allPlugins := []volume.VolumePlugin{}
allPlugins = append(allPlugins, empty_dir.ProbeVolumePlugins()...)
allPlugins = append(allPlugins, git_repo.ProbeVolumePlugins()...)
...

18. 3. Implement the volume plugin (pkg/volume/...):
type VolumePlugin interface {
Init(host VolumeHost) error
Name() string
CanSupport(spec *Spec) bool
NewBuilder(spec *Spec, podRef *api.Pod, opts VolumeOptions)
(Builder, error)
NewCleaner(name string, podUID types.UID)
(Cleaner, error)
}

19. type VolumePlugin interface {
Init(host VolumeHost) error
Name() string
CanSupport(spec *Spec) bool
...

20. ...
NewBuilder(
spec *Spec, podRef *api.Pod, opts VolumeOptions,
) (Builder, error)
NewCleaner(
name string, podUID types.UID,
) (Cleaner, error)
}

21. Simplified example of NewBuilder
func (p *plg) NewBuilder(spec, pod, opts) (volume.Builder, error) {
source, _ := p.getFlockerVolumeSource(spec)
builder := flockerBuilder{
flocker: &flocker{
datasetName: source.DatasetName,
pod: pod,
...
},
...
}
return &builder, nil
}

22. 4. Implement the builder
type Builder interface {
Volume
SetUp(fsGroup *int64) error
SetUpAt(dir string, fsGroup *int64) error
GetAttributes() Attributes
}

23. Simplified example of SetUpAt for git repo

24. I lied

26. func (b *gitRepoVolumeBuilder) SetUpAt(dir string, fsGroup *int64) error {
if volumeutil.IsReady(b.getMetaDir()) {
return nil
}
wrapped, err := b.plugin.host.NewWrapperBuilder(b.volName, wrappedVolumeSpec, &b.pod, b.opts)
if err != nil {
return err
}
if err := wrapped.SetUpAt(dir, fsGroup); err != nil {
return err
}
args := []string{"clone", b.source}
if len(b.target) != 0 {
args = append(args, b.target)
}
if output, err := b.execCommand("git", args, dir); err != nil {
return fmt.Errorf("failed to exec 'git %s': %s: %v",
strings.Join(args, " "), output, err)
}
files, err := ioutil.ReadDir(dir)
if err != nil {
return err
}
if len(b.revision) == 0 {
// Done!
volumeutil.SetReady(b.getMetaDir())
return nil
}
var subdir string
switch {
case b.target == ".":
// if target dir is '.', use the current dir
subdir = path.Join(dir)
case len(files) == 1:
// if target is not '.', use the generated folder
subdir = path.Join(dir, files[0].Name())
default:
// if target is not '.', but generated many files, it's wrong
return fmt.Errorf("unexpected directory contents: %v", files)
}
if output, err := b.execCommand("git", []string{"checkout", b.revision}, subdir); err != nil {
return fmt.Errorf("failed to exec 'git checkout %s': %s: %v", b.revision, output, err)
}
if output, err := b.execCommand("git", []string{"reset", "--hard"}, subdir); err != nil {
return fmt.Errorf("failed to exec 'git reset --hard': %s: %v", output, err)
}
volumeutil.SetReady(b.getMetaDir())
return nil
}

27. 1. Check meta: was it called before? Is it ready?
2. Use empty dir to prepare the path
3. git clone it
4. Checkout the revision sent on the pod definition
5. Job done? Mark as ready

28. Important bit of empty_dir
switch ed.medium {
case api.StorageMediumDefault:
err = ed.setupDir(dir)
case api.StorageMediumMemory:
err = ed.setupTmpfs(dir, securityContext)
default:
err = fmt.Errorf("unknown storage medium %q", ed.medium)
}
volume.SetVolumeOwnership(ed, fsGroup)

29. Another simplified example for Flocker

30. I lied
again

32. func (b flockerBuilder) SetUpAt(dir string, fsGroup *int64) error {
if volumeutil.IsReady(b.getMetaDir()) {
return nil
}
if b.client == nil {
c, err := b.newFlockerClient()
if err != nil {
return err
}
b.client = c
}
datasetID, err := b.client.GetDatasetID(dir)
if err != nil {
return err
}
s, err := b.client.GetDatasetState(datasetID)
if err != nil {
return fmt.Errorf("The volume '%s' is not available in Flocker. You need to create this manually with Flocker CLI before using it.", dir)
}
primaryUUID, err := b.client.GetPrimaryUUID()
if err != nil {
return err
}
if s.Primary != primaryUUID {
if err := b.updateDatasetPrimary(datasetID, primaryUUID); err != nil {
return err
}
}
b.flocker.path = s.Path
volumeutil.SetReady(b.getMetaDir())
return nil
}

33. 1. Was it called? Is it ready?
2. Is the dataset ready? If not, not my problem mate
3. If primary UUIDs doesn't match (rescheduled pod),
update them
4. Wait until ready
5. Mark as ready

34. 5. Implement the Persistent Volume Plugin
type PersistentVolumePlugin interface {
VolumePlugin
GetAccessModes() []api.PersistentVolumeAccessMode
}

35. func (p *plg) GetAccessModes() []api.PersistentVolumeAccessMode {
return []api.PersistentVolumeAccessMode{
api.ReadWriteOnce,
}
}

36. http://bit.ly/kubecon

37. "Problems"

38. 4 CLAs
4 Shippable & Jenkins
4 hack/ scripts
4 PR

39. Summary
1. Add API
2. Enable the plugin on the Kubelet
3. Implement VolumePlugin interface: NewBuilder &
NewCleaner
4. Implement the builder itself: SetUpAt
5. Persistent?
6. ❤

41. Just one (more) thing

43. We are hiring at

44. Thanks!
@agonzalezro