Welding Code Together With Git Branches

That's not me in the photo, but ... I've been there.  Ironworker Mike Kulp, 43, climbs a column before connecting a perimeter beam to the 54th floor. Here Kulp is working approximately 825 feet above street level.

That’s not me in the photo … but I’ve been there. Ironworker Mike Kulp, 43, climbs a column before connecting a perimeter beam to the 54th floor. Here Kulp is working approximately 825 feet above street level. (related blog: newyorknatives.com; image credit: Barbara Johnston)

I was an ironworker for a brief period of time at Local 86.  I loved structural ironworking.  And to this day I still have my tools.  If that work paid what my current job does, I’d be doing that for a living. I miss it.

Oddly enough there were some valuable lessons I learned on that job that could serve as an analogy for Git repository setups.

One of those lessons was that the integrity of the foundation of any project can determine whether or not it ends up being a successful one.   In other words, you get in the end what you set yourself up for – good or bad.

That lesson could apply to your mobile projects this way: If you can identify what the “foundational pieces” are for your project and spend some time shoring those up, the effort can help keep petty repository management issues at bay.  A bad repository setup can halt a project for a few days, so it’s important.

Git has a some great features that are great for doing this when your project relies on multiple repositories.  This blog is about one of them:

Subtree Merging: the ability to merge the code from one repository’s into a subdirectory in another repository’s.

Subtree Merging allows you to approach the construction of your mobile applications (or any software) somewhat like putting a skyscraper together … one section at a time.

Foundational pieces are first fixed into place.  For Git that means the basic repository with defined remotes.

Once that work is done, the rest of the building can be built up around that, making modifications and customizations as you go with an engineering and building process.  But the foundations will still remain the same.  For Git that’s the merging of changes in child repositories into your main one (super repository).

This blog assumes you have just a basic familiarity with git.  Perhaps you’ve cloned a repository or two and made some commits.  Even if you have a lot of experience with Git you might find this blog posting a handy reference.

The Setup

Our goal is to create an Android application.  We’ll start off with no repositories on our local computer and progress from there.  Nothing cloned, checked out, etc.

There will be four repositories that we’re going to work with:

  1. MyApp: A repository that will contain the Eclipse project for our Android application.   
  2. SecretSauce: A repository that will contain your employeer’s proprietary reusable library code.
  3. Twanoh, an open source repository that contains some useful functionality and utility code that will help shorten our coding time.  Located at http://code.google.com/p/twanoh/
  4. SuperRepo: a super-repository that allows you to manage the other three repositories without pulling your hair out. More on that later, below.

I also have a starter project generated by Eclipse with a default generated Android application.  I exported it to a tarball called MyApp.tgz. This will be imported into the MyApp repository once it’s created.

And of course to but this guide into a real-world context, there are some basic rules of engagement for the repositories we work with on this project.

  1. MyApp, SecretSauce, and SuperRepo don’t exist yet.  We have to create them.  These would represent proprietary code repositories that your company might control or have internal access too.
  2. Since Twanoh is an open source library, it will be read only.  None of the code in the other repositories can be inserted into Twanoh.
  3. Importing code into SuperRepo from any repository is O.K.
  4. Copying code between any other repository is not allowed.  It will just create un-necessarily duplicated code.

Caveat:

I’m working on a Mac (OSX Mountain Lion), but all the examples here are on the command line so you will have to adjust your commands to whatever system you’re working on.  I assume since Windows is so awesome, that it will be no problem at all for Windows users to mentally follow along and translate what I’m doing there to suit their own needs.

(the great thing about this being my blog as opposed to a published book: I get to be a smartass)

Where was I? Oh, right … the setup.  I’m going to start at the very beginning of a project setup so you can follow along.  Not that you’re dumb.  I just don’t like examples that gloss over too many details. I think you should be able to retrace my steps from the beginning in case you get lost or need to cross-check your own work.

I.

Repository Setup

14_37_WTC-Site---March-2006

Step 1. Lay a carefully constructed foundation for your project so everything else can fit into place. (image: WTC, March 2006)

Environment Preliminaries

It’s helpful to start with a clean slate.  So, the first thing I’m going to do is adjust my Unix prompt so that it’s easier for you to read (you’re welcome):

$ export PS1="\W $ "
~ $

That simplifies the prompt to show the directory I’m in.  Next, I’m going to create a special directory to do all my work in:

~ $ sudo mkdir -p /usr/local/blog-examples/code-branches
~ $

Then – export a shell variable to make it easy to navigate there.  That will make it clear what directory you are supposed to be in as you follow along.

~ $ export PROJ=/usr/local/blog-examples/code-branches
~ $ cd $PROJ
code-branches $ pwd
/usr/local/blog-examples/code-branches
code-branches $

Use git-init to Create Repositories

Now I’m ready to create the repositories I need.  Curious about the switches I passed to ‘git init’ ? Good. Go read about that.  I’ll wait.

Create a directory where I can store all my Git bare repositories (see Git – Setting Up the Server).  One for each repository.

$ cd $PROJ
code-branches $ pwd
/usr/local/blog-examples/code-branches
code-branches $ mkdir -p bare/MyApp; mkdir -p bare/SecretSauce; mkdir -p bare/SuperRepo
code-branches $ ls bare
MyApp     SecretSauce     SuperRepo
code-branches $

… and now to create the repository for MyApp using git-init:

code-branches $ cd $PROJ/bare/MyApp; git init --bare --shared=group
Initialized empty shared Git repository in /usr/local/blog-examples/code-branches/bare/MyApp/
MyApp $

… for SecretSauce

MyApp $ cd $PROJ/bare/SecretSauce; git init --bare --shared=group
Initialized empty shared Git repository in /usr/local/blog-examples/code-branches/bare/SecretSauce/
SecretSauce $

… for SuperRepo

SecretSauce $ cd $PROJ/bare/SuperRepo; git init --bare --shared=group
Initialized empty shared Git repository in /usr/local/blog-examples/code-branches/bare/SuperRepo/
SuperRepo $

Use git-clone to Clone Repositories

Bare repositories are not used for working copies of your repository.  Bare repositories are where you’ll push all your changes.  To do actual work, we need to clone all the repositories we need into working copies.

Create a directory for those and clone the repositories using git-clone:

SuperRepo $ cd $PROJ; mkdir -p working
code-branches $ cd $PROJ/working; git clone ../bare/Myapp; git clone ../bare/SecretSauce; git clone ../bare/SuperRepo
Cloning into 'Myapp'...
warning: You appear to have cloned an empty repository.
done.
Cloning into 'SecretSauce'...
warning: You appear to have cloned an empty repository.
done.
Cloning into 'SuperRepo'...
warning: You appear to have cloned an empty repository.
done.
working $ ls
Myapp SecretSauce SuperRepo
working $

Notice I didn’t clone the Twanoh project yet.  For our purposes, since we’re not modifying Twanoh, we don’t need to clone it.  But we will pull in branches from that project later.

Import the MyApp Eclipse Project into the MyApp repository

I’ve stored the exported Eclipse project in a file called MyApp.tgz and copied it to the $PROJ subdirectory.  I will now import it into the MyApp repository

working $ cd $PROJ; tar -C $PROJ/working -xzvf MyApp.tgz
x ./MyApp/
x ./MyApp/.classpath
x ./MyApp/.project
x ./MyApp/.settings/
x ./MyApp/AndroidManifest.xml
x ./MyApp/assets/
x ./MyApp/bin/
x ./MyApp/gen/
x ./MyApp/libs/
x ./MyApp/proguard-project.txt
x ./MyApp/project.properties
x ./MyApp/res/
x ./MyApp/src/
x ./MyApp/src_secretsauce/
x ./MyApp/src_twanoh/
x ./MyApp/src_twanoh/README
x ./MyApp/src_secretsauce/README
x ./MyApp/src/schilling/
x ./MyApp/src/schilling/richard/
x ./MyApp/src/schilling/richard/myapp/
x ./MyApp/src/schilling/richard/myapp/MyAppActivity.java
x ./MyApp/res/drawable-hdpi/
x ./MyApp/res/drawable-ldpi/
x ./MyApp/res/drawable-mdpi/
x ./MyApp/res/drawable-xhdpi/
x ./MyApp/res/layout/
x ./MyApp/res/menu/
x ./MyApp/res/values/
x ./MyApp/res/values/strings.xml
x ./MyApp/res/values/styles.xml
x ./MyApp/res/menu/main.xml
x ./MyApp/res/layout/main.xml
x ./MyApp/res/drawable-xhdpi/ic_launcher.png
x ./MyApp/res/drawable-mdpi/ic_launcher.png
x ./MyApp/res/drawable-hdpi/ic_launcher.png
x ./MyApp/libs/android-support-v4.jar
x ./MyApp/gen/schilling/
x ./MyApp/gen/schilling/richard/
x ./MyApp/gen/schilling/richard/myapp/
x ./MyApp/gen/schilling/richard/myapp/BuildConfig.java
x ./MyApp/gen/schilling/richard/myapp/R.java
x ./MyApp/bin/classes/
x ./MyApp/.settings/org.eclipse.jdt.core.prefs
code-branches $

I’ll also add  .gitignore and a README file to MyApp before I do my first checkin

code-branches $ cd $PROJ/working/MyApp
MyApp $ echo "Android project for MyApp" > README
MyApp $ echo ".DS_Store" > .gitignore
MyApp $ echo "bin" >> .gitignore
MyApp $ echo "gen" >> .gitignore
MyApp $

First Checkins

Now that MyApp has some code in it it’s possible to check it into the MyApp repository

MyApp $ git add .; git commit -m"Initial checkin - MyApp README"; git push origin master
[master (root-commit) 66812d1] Initial checkin - MyApp README
 17 files changed, 182 insertions(+)
 create mode 100644 .classpath
 create mode 100644 .gitignore
 create mode 100644 .project
 create mode 100644 .settings/org.eclipse.jdt.core.prefs
 create mode 100644 AndroidManifest.xml
 create mode 100644 README
 create mode 100644 libs/android-support-v4.jar
 create mode 100644 proguard-project.txt
 create mode 100644 project.properties
 create mode 100644 res/drawable-hdpi/ic_launcher.png
 create mode 100644 res/drawable-mdpi/ic_launcher.png
 create mode 100644 res/drawable-xhdpi/ic_launcher.png
 create mode 100644 res/layout/main.xml
 create mode 100644 res/menu/main.xml
 create mode 100644 res/values/strings.xml
 create mode 100644 res/values/styles.xml
 create mode 100644 src/schilling/richard/myapp/MyAppActivity.java
Counting objects: 32, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (21/21), done.
Writing objects: 100% (32/32), 358.44 KiB, done.
Total 32 (delta 0), reused 0 (delta 0)
To /usr/local/blog-examples/code-branches/working/../bare/Myapp
 * [new branch] master -> master
MyApp $

And for SuperRepo

MyApp $ cd $PROJ/working/SuperRepo
SuperRepo $ echo "Super repository for MyApp" > README
SuperRepo $ git add .; git commit -m"Initial checkin - SuperRepo README."; git push origin master
[master (root-commit) bef905b] Initial checkin - SuperRepo README.
 1 file changed, 1 insertion(+)
 create mode 100644 README
Counting objects: 3, done.
Writing objects: 100% (3/3), 265 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To /usr/local/blog-examples/code-branches/working/../bare/SuperRepo
 * [new branch] master -> master
SuperRepo $

And finally SecretSauce

SuperRepo $ cd $PROJ/working/SecretSauce
SecretSauce $ echo "Secret Sauce. CONFIDENTIAL. Do not distribute." > README
SecretSauce $ git add .; git commit -m"Initial checkin - SecretSauce README"; git push origin master
[master (root-commit) 09bc950] Initial checkin - SecretSauce README
 1 file changed, 1 insertion(+)
 create mode 100644 README
Counting objects: 3, done.
Writing objects: 100% (3/3), 284 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To /usr/local/blog-examples/code-branches/working/../bare/SecretSauce
 * [new branch] master -> master
SecretSauce $

Take a Break ….

There is a lot of detail in this blog so far.  Take a break.  Come back in a bit … cause this is where things start to get interesting.

II.

Stack Up The Project

14_37_2010_04_27-WTC-Site-Overview---Credit-Joe-Woolhead

Step 2. With a project foundation built, you can now safely add things on top of it. (image: WTC, May 2010)

Review

So, here’s what we’ve got so far:

  • We set up our shell to make things easier by: creating a directory to work in, setting our command prompt to show the current directory, and export an environment variable $PROJ to help us navigate there easily.
  • Our bare repositories are setup using git-init in $PROJ/bare.  They are $PROJ/bare/SuperRepo, $PROJ/bare/MyApp, and $PROJ/bare/SecretSauce.
  • We’ve cloned those three repositories into the $PROJ/working subdirectory and added files to them.

Link SuperRepo To the Other Repositories

If you haven’t done so, you might want to read up on git remotes and the  git-remote, and git-fetch commands.

It may not be immediately clear to you by reading that documentation that one working clone repository can point to multiple remotes at the same time with no ill side effects.  That is, our working copy of SuperRepo can point to many remote repositories.  You can completely flip between working copies/branches from those remote repositories without ever having code from them collide.  It’s as if you can flip between repositories very quickly using git-checkout.

(I don’t cover git-checkout in detail in this blog, but it will be used).

For example, let’s add a remote reference in SuperRepo to MyApp, SecretSauce and Twanoh:

SecretSauce $ cd $PROJ/working/SuperRepo
SuperRepo $ git remote -v add myapp ../../bare/MyApp
SuperRepo $ git remote -v add secretsauce ../../bare/SecretSauce
SuperRepo $ git remote -v add twanoh https://code.google.com/p/twanoh/
SuperRepo $ git remote -v rename origin superrepo
SuperRepo $ git remote -v
myapp ../../bare/MyApp (fetch)
myapp ../../bare/MyApp (push)
secretsauce ../../bare/SecretSauce (fetch)
secretsauce ../../bare/SecretSauce (push)
superrepo /usr/local/blog-examples/code-branches/working/../bare/SuperRepo (fetch)
superrepo /usr/local/blog-examples/code-branches/working/../bare/SuperRepo (push)
twanoh https://code.google.com/p/twanoh/ (fetch)
twanoh https://code.google.com/p/twanoh/ (push)
SuperRepo $

I also renamed the remote origin to superrepo so it’s easy to be reminded what it points to.

Now we can fetch branches for all the child repositories:

SuperRepo $ git fetch --all
Fetching superrepo
Fetching myapp
warning: no common commits
remote: Counting objects: 32, done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 32 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (32/32), done.
From ../../bare/MyApp
 * [new branch] master -> myapp/master
Fetching secretsauce
warning: no common commits
remote: Counting objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From ../../bare/SecretSauce
 * [new branch] master -> secretsauce/master
Fetching twanoh
warning: no common commits
remote: Counting objects: 539, done.
remote: Finding sources: 100% (539/539), done.
remote: Total 539 (delta 322)
Receiving objects: 100% (539/539), 688.41 KiB, done.
Resolving deltas: 100% (322/322), done.
From https://code.google.com/p/twanoh
 * [new branch] javamail_import -> twanoh/javamail_import
 * [new branch] master -> twanoh/master
SuperRepo $

A quick peek at the branches we now have tell us that remote twanoh has a couple branches.  But the other child repositories myapp and secretsauce don’t.  That makes sense because we haven’t added any code to those two yet.

SuperRepo $ git branch -a
* master
 remotes/myapp/master
 remotes/secretsauce/master
 remotes/superrepo/master
 remotes/twanoh/javamail_import
 remotes/twanoh/master
SuperRepo $

Current Repository State

The image below shows the current state of the repository.  The various remotes are shown, and their history graphs are disconnected. This is correct, because we haven’t done any merges yet.

Screen Shot 2013-04-07 at 9.31.44 PM

The SuperRepo working clone, which shows remotes for SecretSauce, MyApp, and Twanoh. Click to enlarge.

Use git-merge & git-read-tree To Import Code

The next step is to import code into SuperRepo from the child repositories MyApp, SecretSauce, and Twanoh. First myapp, since that’s our main application

SuperRepo $ cd $PROJ/working/SuperRepo
SuperRepo $ git merge -s ours --no-commit myapp/master
Automatic merge went well; stopped before committing as requested
SuperRepo $ git read-tree --prefix=myapp -u myapp/master
SuperRepo $ git commit -m"Import MyApp tree."
[master 9cb4e70] Import MyApp tree.
SuperRepo $

read-tree uses the –prefix switch to tell Git that my app code needs to be imported to a subdirectory within SuperRepo.  It’s automatically created for you.

SuperRepo $ ls
README myapp
SuperRepo $

And now we doe the same with SecretSauce, but we tell read-tree to put the code from that repository into the directory myapp/secretsauce.

SuperRepo $ git merge -s ours --no-commit secretsauce/master
Automatic merge went well; stopped before committing as requested
SuperRepo $ git read-tree --prefix=myapp/secretsauce -u secretsauce/master
SuperRepo $ git commit -m"Import Secret Sauce tree."
[master ce552ae] Import Secret Sauce tree.
SuperRepo $

And finally Twanoh.  As I mentioned above, it’s not necessary to have cloned Twanoh anywhere because we’re not making changes to it.

SuperRepo $ git merge -s ours --no-commit twanoh/master
Automatic merge went well; stopped before committing as requested
SuperRepo $ git read-tree --prefix=myapp/twanoh -u twanoh/master
SuperRepo $ git commit -m"Import twanoh tree."
[master d0761cd] Import twanoh tree.
SuperRepo $

The last step is to push all the changes up to the superrepo repository

SuperRepo $ git push superrepo master
Counting objects: 583, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (192/192), done.
Writing objects: 100% (582/582), 1.02 MiB, done.
Total 582 (delta 325), reused 539 (delta 322)
To /usr/local/blog-examples/code-branches/working/../bare/SuperRepo
 bef905b..d0761cd master -> master
SuperRepo $

Now the Git graphical interface shows we’ve merged all the files into SuperRepo

Git's history show how all child repositories are merged into SuperRepo.

Git’s history show how all child repositories are merged into SuperRepo. Click to enlarge.

III.

Merging Updates

14_61_Edited-for-Web-20121016-IMG_7928

Step 3. With your project infrastructure topped out, customize and modify at will. Incorporating changes at any given point in time is essential until the project is complete. (Image: WTC, November 2012)

The setup we have so far could be considered the “Hello World” setup of a git repository. It contains a Hello World app that we can build on top of, and several child repositories that are brought into the mix – our toolkit so to speak.

The important thing about this setup is now we can use a stable process to incorporate other people’s work without having to re-tool our own project.  Whenever a change happens to MyApp, SecretSauce, or Twanoh, we can easily import the changes.

Make Modifications

Modify MyApp.

SuperRepo $ cd $PROJ/working/MyApp
MyApp $ echo "More README content for MyApp." >> README
MyApp $ git commit -a -m"Added to MyApp README."
[master 7c0e3d5] Added to MyApp README.
 1 file changed, 1 insertion(+)
MyApp $ git push origin master
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 325 bytes, done.
Total 3 (delta 1), reused 0 (delta 0)
To /usr/local/blog-examples/code-branches/working/../bare/Myapp
 66812d1..7c0e3d5 master -> master
MyApp $

And SecretSauce.

MyApp $ cd $PROJ/working/SecretSauce
SecretSauce $ echo "More README content for SecretSauce." >> README
SecretSauce $ git commit -a -m"Added to SecretSauce README."
[master a4f9863] Added to SecretSauce README.
 1 file changed, 1 insertion(+)
SecretSauce $ git push origin master
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 335 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To /usr/local/blog-examples/code-branches/working/../bare/SecretSauce
 09bc950..a4f9863 master -> master
SecretSauce $

We’ve simulated some changes, albeit simple, to both the application (MyApp) and a library it depends on (SecretSauce).  Those changes were made and pushed to their respective bare repositories.  SuperRepo doesn’t know about them yet so we have to tell SuperRepo to fetch the changes.

SecretSauce $ cd $PROJ/working/SuperRepo
SuperRepo $ git fetch -v myapp master
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From ../../bare/MyApp
 * branch master -> FETCH_HEAD
SuperRepo $ git fetch -v secretsauce master
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From ../../bare/SecretSauce
 * branch master -> FETCH_HEAD
SuperRepo $

And for the final step. We use git-pull to merge the changes we just fetched into the project.  First MyApp.

SuperRepo $ git pull -s subtree --no-edit myapp master
From ../../bare/MyApp
 * branch master -> FETCH_HEAD
Merge made by the 'subtree' strategy.
 myapp/README | 1 +
 1 file changed, 1 insertion(+)
SuperRepo $

And then SecretSauce.

SuperRepo $ git pull -s subtree --no-edit secretsauce master
From ../../bare/SecretSauce
 * branch master -> FETCH_HEAD
Merge made by the 'subtree' strategy.
 myapp/secretsauce/README | 1 +
 1 file changed, 1 insertion(+)
SuperRepo $

Git has automatically kept track of the directory that each remote branch copied into with read-tree.  So, it’s able to properly update your project.

Here’s the final state of your repository.

The Git Gui now shows our changes merged into our projects repository, SuperRepo.

The Git Gui now shows our changes merged into our projects repository, SuperRepo. Click to enlarge.

Advertisements

And now it's your turn ... comment here or on Twitter (@Androider)

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s