1. Chmouel Boudjnah!
OpenStack Summit 2014 Atlanta
OpenStack Swift as a Backend for Git
Chmouel Boudjnah - @chmouel - eNovance

2. Chmouel Boudjnah - @chmouel - eNovance
The mandatory about me

3. Chmouel Boudjnah - @chmouel - eNovance
About me
I am not the author, Fabien Boucher
aka: @bzho
Chmouel Boudjnah - @chmouel - eNovance

4. Chmouel Boudjnah - @chmouel - eNovance
About me
We both work for eNovance

5. Chmouel Boudjnah - @chmouel - eNovance
About me
OpenStacker for quite some time now

6. Chmouel Boudjnah - @chmouel - eNovance
So what is Swift?

7. Chmouel Boudjnah - @chmouel - eNovance
• Object Storage
• Not a file system
• Eventually Consistent
• HTTP REST API
• Very Scalable
• Software defined Storage
What is Swift

8. Chmouel Boudjnah - @chmouel - eNovance
Why would you want to use GIT with Swift?

9. Chmouel Boudjnah - @chmouel - eNovance
Why Swift with GIT
Why not?

10. Chmouel Boudjnah - @chmouel - eNovance
Why Swift with GIT
When you use a file system you have to :
Perform backups of your repositories regularly
Perform complex and risky operations to extend the storage capabilities
for you repositories.
You have mostly have to manage it yourself
It’s an another piece in your infrastructure
You have to manage the downtime
Not fun

11. Chmouel Boudjnah - @chmouel - eNovance
So what are the benefits?

12. Chmouel Boudjnah - @chmouel - eNovance
Why Swift with GIT
The advantages of using Swift
Safety: The inherent replication of a data store.
Capability: easily extend storage capacity.
High Availability: no single point of failure.
Maintainability: allow operations teams to maintain the cluster without
any downtime.
In an OpenStack environment it’s usually already there.

13. Chmouel Boudjnah - @chmouel - eNovance
Let’s see how git internal works

14. Chmouel Boudjnah - @chmouel - eNovance

15. Chmouel Boudjnah - @chmouel - eNovance

16. Chmouel Boudjnah - @chmouel - eNovance
So how did we plug that to Swift?

17. Chmouel Boudjnah - @chmouel - eNovance
We did not want to modify the GIT client binaries.

18. Chmouel Boudjnah - @chmouel - eNovance
We used a pure python GIT library called Dulwich

19. Chmouel Boudjnah - @chmouel - eNovance
• Create, read, manage loose objects (blob, tree,
commit, tag).
• Create, read, manage pack files, pack indexes.
• Implement the Git smart protocol through git-
upload-pack and git-receive-pack.
• Implement the Git, HTTP, SSH listeners to start
Dulwich as a Git server.
• Provide a well designed interface to the storage
backend. (Repo, MemoryRepo).
Dulwich

20. Chmouel Boudjnah - @chmouel - eNovance
How do we use Dulwhich
We implemented a custom repo called SwiftRepo as a backend to the git
listener of Dulwich which communicate to OpenStack Swift

21. Chmouel Boudjnah - @chmouel - eNovance
So about that SwiftRepo backend how does it
work?

22. Chmouel Boudjnah - @chmouel - eNovance
SwiftRepos backend to Dulwich
It doesn’t use the standard way to store Git
references (one file by reference).
It doesn’t store loose objects but instead uses the pack format.
In addition to the .index that relate to each pack file we added .info object
It makes the Request to Swift concurrently.

23. Chmouel Boudjnah - @chmouel - eNovance
How’s the benchmark then?

24. Chmouel Boudjnah - @chmouel - eNovance
Benchmark Setup
Benchmark performed on two architectures :
Github :
● Git client from eNovance network <-> GitHub
!
Dulwich/Swift :
●· Git client from eNovance network <-> Dulwich (Rax compute) <->
Rax Cloudfiles
!
Three different repos :
!
●· Swiftsync (873 objects with 293 commits)
●· Edeploy (2246 objects with 618 commits)
●· Swift (23480 objects with 3212 commit)

25. Chmouel Boudjnah - @chmouel - eNovance
Full repository push

26. Chmouel Boudjnah - @chmouel - eNovance
Clone

27. Chmouel Boudjnah - @chmouel - eNovance
That’s it, hope you
enjoyed.

28. Chmouel Boudjnah - @chmouel - eNovance
Eventual Consistentie
Why Eventual Consistentie window is not an issue