The document discusses several Git techniques for reducing network transfer and storage requirements when using Git in large projects or continuous integration systems, including reference repositories, narrow refspecs, shallow clones, large file support, and sparse checkouts. It outlines the benefits and limitations of each technique, such as reducing network transfer and local storage needs but potentially limiting history or breaking workflows. The goal is to help optimize remote repositories, master servers that detect changes, and agent servers that perform builds and jobs.

1. Git in the Large – Tips & Tricks
Mark Waite

2. Git in the Large – Tips & Tricks
Mark Waite – mark.earl.waite@gmail.com

3. Remote
• Central
repo
Master
• Pipeline
cache repo
• Polling repo
Agent
• Pipeline
cache repo
• Local repo
• Workspace
Git in the Large - Concepts
© 2017 CloudBees, Inc. All Rights Reserved

4. • Reduce remote work and network transfer with:
• Reference repository
• Narrow refspec
• Shallow clone
• Large file support
Git in the Large – Help the Remote
© 2017 CloudBees, Inc. All Rights Reserved
• Remote repository
• Includes all history
• Sends requested history
• Includes all large files
• Sends requested large files

5. Reference Repository
• New repositories can “reference” an existing repository
rather than downloading the referenced data
• Positive
– Reduce network data transfer
– Reduce local repository storage
• Negative
– Reference copies not automatically updated
– Destroying reference repository damages all references
© 2017 CloudBees, Inc. All Rights Reserved

6. Narrow Refspec
• Refspec defines history retrieval start points (breadth)
• Positive
– If you only need one branch, narrow refspec retrieves one
– Reduce local repository storage
• Negative
– Copies history only for matching branches
– Refspec patterns severely limited
© 2017 CloudBees, Inc. All Rights Reserved

7. Shallow Clone
• Shallow clone defines history retrieval depth
• Positive
– If you only need most recent commit, use shallow clone
– Reduce local repository storage
• Negative
– Breaks merge workflows
– Change reports can be incomplete
– Command line git only (no JGit)
© 2017 CloudBees, Inc. All Rights Reserved

8. Large File Support
• Git extension that stores large files outside repository
• Positive
– High performance
– Very active development
– Reduces local repository storage
• Negative
– Requires LFS extension
– Requires extra support from hosting provider
– No ssh support (http / https only)
– No submodule support in the git plugin (yet)
– Large file history requires separate request (LFS commands)
© 2017 CloudBees, Inc. All Rights Reserved

9. Git in the Large – Help the Master
© 2017 CloudBees, Inc. All Rights Reserved
• Master
• Scans repositories
• Detects changes
• Computes changes
• Reduce master work and disc use with:
• Reference repository
• Large file support

10. Git in the Large – Help the Agent
© 2017 CloudBees, Inc. All Rights Reserved
• Agent
• Fills workspace
• Builds the job
• Reduce agent work and disc space with:
• Narrow refspec
• Shallow clone
• Reference repository
• Large file support
• Sparse checkout

11. Sparse Checkout
• Sparse checkout defines directories for checkout
(workspace breadth)
• Positive
– If you only need a subtree, sparse checkout creates it
– Reduces local working directory storage
• Negative
– Subtree checkout is not the full tree
© 2017 CloudBees, Inc. All Rights Reserved