In November 2012 I helped the giant JBoss Tools repo containing 400+ eclipse plugins, 2+ millions line of code with 7+ year of history migrate from SVN into a multitude of Git repositories. Most existing large migrations I could find info about was about the size as one of our 30+ modules in JBoss tools and it turned out the advice in most of these just did not cope with a migration as large as this. During this tale I cover what tools I examined, rejected and in the end selected to do the migration. I will outline the migration process we did as a team, what things Git enabled immediately and which things we are moving towards now that Git have opened up for a more flexible build and development model and structure. Before this move a 6+ months long preparation and even longer thinking process was started to ensure all data would be migrated properly in an efficient and repeatable manner with as little possible impact on developer productivity as possible. After this talk you should be well prepared to do both small and large migrations to git for Eclipse based plugin projects and know what dangers to look out for and what tools to consider. If you want to take a sneakpeak you can see the scripts and instructions for the migration at https://github.com/maxandersen/jbosstools-gitmigration

1. A tale about a Big
SVN to Git migration
Max rydahl Andersen
@maxandersen
Senior Principal Engineer, red Hat
Wednesday 27 March 13

2. • Do not use svn2git
• Learn about fast-filter.py
• Scripts/Resources at:
• https://github.com/maxandersen/jbosstools-gitmigration
Wednesday 27 March 13

3. goals for git migration
• Split up Big Repo to smaller ones
• Keep the history
• Get to use GitHub
• Make it fast and fun again!
Wednesday 27 March 13

4. Keeping it simple
common plugins .core
.ui
...
features .feature
.source.fe
ature
...
tests .tests
...
site
Wednesday 27 March 13

5. Keeping it simple
common plugins .core
.ui
...
features .feature
.source.fe
ature
...
tests .tests
Svn
...
site
git svn clone -s
git svn --authors-file=authors.txt
http://svn.example.com/svn-repository/ svn-mirror
svn-mirror.git
Wednesday 27 March 13

6. ...a real project
common plugins .core
.ui
...
features .feature
.source.fe
ature
...
tests .tests
...
site
Wednesday 27 March 13

7. ...a real project
JSF
... plugins .core ... plugins .core
common plugins .core
hibernate plugins .core CDI plugins .core plugins .core
seam plugins .core
.ui
.ui .ui .ui .ui
.ui .ui
...
... ... ... ...
features .feature
... ...
.source.fe features .feature features .feature
ature
features .feature features .feature features .feature
features .feature
... .source.fe
ature .source.fe
ature
.source.fe .source.fe .source.fe
tests .tests .source.fe ature ature ature
... ature
...
...
... ... ...
tests .tests
site ...
tests .tests
... tests .tests
tests .tests tests .tests
tests .tests ...
site
...
site ... ...
...
site
site site
site
... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core
.ui .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui
... ... ... ... ... ... ... ... ... ... ... ...
features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature
.source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe
ature ature ature ature ature ature ature ature ature ature ature ature
... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core
... ... ... ... ... ... ... ... ... ... ... ...
.ui .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui
tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests
... ... ... ... ... ... ... ... ... ... ... ...
... ... ... ... ... ... ... ... ... ... ... ...
features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature
site site site site site site site site site site site site
.source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe
ature ature ature ature ature ature ature ature ature ature ature ature
... ... ... ... ... ... ... ... ... ... ... ...
tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests
... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core ... plugins .core
... ... ... ... ... ... ... ... ... ... ... ...
.ui .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui .ui
site site site site site site site site site site site site
... ... ... ... ... ... ... ... ... ... ... ...
features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature features .feature
.source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe .source.fe
ature ature ature ature ature ature ature ature ature ature ature ature
... ... ... ... ... ... ... ... ... ... ... ...
tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests tests .tests
... ... ... ... ... ... ... ... ... ... ... ...
site site site site site site site site site site site site
Wednesday 27 March 13

8. How hard can it be ?
Svn
git svn clone --prefix=svn/ -s
git svn --authors-file=authors.txt
http://svn.example.com/svn-repository/ svn-mirror
svn-mirror.git
Wednesday 27 March 13

9. Check list for migration
1.Create svn-mirror.git from mirrored svn
2.Cleanup Mirror
1. rename master to trunk, Delete dead branches,
Checkout all branches, tag branches to real tags
3.Create individual Repositories
1. Use fast-filter.py to process content, Remove empty
tags/branches, Garbage Collect
4.Clean up Garbage
1. Prune big files, Fix line endings, Garbage Collect
5. ? (Publish to Github)
6.Profit!
Wednesday 27 March 13

10. SVN Mirroring
svn sync Svn
Svn
Mirror
svnsync init file://svn-mirror http://svn.example.com/repository
while(true)
svnsync sync file://svn-mirror http://svn.example.com/repository
Wednesday 27 March 13

11. SVN Mirroring
svn sync Svn
Svn
Mirror
svnsync init file://svn-mirror http://svn.example.com/repository
while(true) git svn
svnsync sync file://svn-mirror http://svn.example.com/repository
cd svn-mirror
git svn rebase --fetch-all
svn-mirror.git
Wednesday 27 March 13

12. SVN Mirroring
svn sync Svn
Svn
Mirror
svnsync init file://svn-mirror http://svn.example.com/repository
while(true) git svn
svnsync sync file://svn-mirror http://svn.example.com/repository
cd svn-mirror
git svn rebase --fetch-all
svn-mirror.git
Svn and Git now isolated and
always in sync !!!
Wednesday 27 March 13

13. svn mirroring
• Atlassian blogs has excellent material, but...
• Use --fetch-all to get full history!
• Do not remove metadata - it will help you in case
of failures
Wednesday 27 March 13

14. Cleanup Branches/Tags
svn-mirror.git
Wednesday 27 March 13

15. Cleanup Branches/Tags
svn-mirror.git
• Branches:
• trunk
• deadbranch
• tags/jbt-4.0.1
• tags/jbt-4.0.2
• jbt-4.0.x
• jbt-3.03
Wednesday 27 March 13

16. Cleanup Branches/Tags
svn-mirror.git
• Branches: • Branches:
• trunk • master
• jbt-4.0.x
• deadbranch
• jbt-3.03
• tags/jbt-4.0.1
• Tags:
• tags/jbt-4.0.2 • jbt-4.0.1
• jbt-4.0.x • jbt-4.0.2
• jbt-3.03
Wednesday 27 March 13

17. Cleanup branches/tags
svn-mirror.git
# Rename trunk to master
$ git branch -m trunk master
$ git branch -rd origin/trunk
# Delete dead branches
$ git branch -a -r |
grep -E "dead|broken|weird|I_WAS_DRUNK" |
xargs git branch -r -D
Wednesday 27 March 13

18. Cleanup branches/tags
svn-mirror.git
# Convert svn tag-branches into git-tags
$ git branch | grep tags |
sed -e "s/tags///g" |
xargs -n 1 -I {}
git tag -m "svn branch tag" {} tags/{}
# delete svn tag-branches
$ git branch | grep tags |
xargs -n 1 git branch -d
Wednesday 27 March 13

19. To split or not to split ?
svn-mirror.git
Wednesday 27 March 13

20. To split or not to split ?
svn-mirror.git
Wednesday 27 March 13

21. Wednesday 27 March 13

22. split/filter process
svn-mirror.git
• /modulex
• plugina
• pluginb
• /moduley
• plugina
Wednesday 27 March 13

23. split/filter process
svn-mirror.git fastfilter
• /modulex
• /modulex
• plugina
• plugina
• pluginb
• pluginb
• /moduley
• plugina
Wednesday 27 March 13

24. split/filter process
svn-mirror.git fastfilter filter-branch
• /modulex
• /modulex
• plugina
• plugina
• plugina
• pluginb
• pluginb
• pluginb
• /moduley
• plugina
Wednesday 27 March 13

25. fast-filter+filter-branch
$ python filter_repo.py
jbosstools-svn-mirror jbosstools-openshift "^openshift.*"
Wednesday 27 March 13

26. fast-filter+filter-branch
$ python filter_repo.py
jbosstools-svn-mirror jbosstools-openshift "^openshift.*"
$ git filter-branch --tag-name-filter cat --prune-empty
--subdirectory-filter openshift -f -- --all
Wednesday 27 March 13

27. Clean up branches
(again)
• $ deleteemptybranches.sh openshift-repo
• # delete empty commits
• $ git filter-branch --tag-name-filter cat
--prune-empty -- --all
Wednesday 27 March 13

28. “it’s simple”
Wednesday 27 March 13

29. “it’s simple”
svn sync Svn
Svn
Mirror
Wednesday 27 March 13

30. “it’s simple”
svn sync Svn
Svn
Mirror
git svn
svn-mirror.git
Wednesday 27 March 13

31. “it’s simple”
svn sync Svn
Svn
Mirror
git svn
svn-mirror.git
cleanup
Wednesday 27 March 13

32. “it’s simple”
svn sync Svn
Svn
Mirror
git svn
svn-mirror.git
cleanup
Wednesday 27 March 13

33. “it’s simple”
svn sync Svn
Svn
Mirror
cleanup
git svn
svn-mirror.git
cleanup
Wednesday 27 March 13

34. Check list for migration
1.Create svn-mirror.git from mirrored svn
2.Cleanup Mirror
1. rename master to trunk, Delete dead branches,
Checkout all branches, tag branches to real tags
3.Create individual Repositories
1. Use fast-filter.py to process content, Remove empty
tags/branches, Garbage Collect
4.Clean up Garbage
1. Prune big files, Fix line endings, add .gitignore
Garbage Collect
5. ?
6.Profit!
Wednesday 27 March 13

35. Tips’n’Tricks
Wednesday 27 March 13

36. Train your team
• http://try.github.com
• O'Reilly Videos
• Understand the internals: https://vimeo.com/49444883
• Learn the CLI before using the UI’s
• Ban git push --force
• Green Button Considered Harmful
Wednesday 27 March 13

37. guardians
vs
Kings
Wednesday 27 March 13

38. Git Hub api is your friend
Wednesday 27 March 13

39. Hub
“github cli”
http://defunkt.io/hub/
• $ hub clone maxandersen/autolinkjboss
• $ hub fork
• $ hub pull-request
• “Fixed this bad presentation”
Wednesday 27 March 13

40. Questions ?
https://github.com/maxandersen/jbosstools-gitmigration
Wednesday 27 March 13