Bluish Coder: git

My mozilla-central git clone is stopping

2012-08-18T20:00:00+12:00

For quite a while I've been maintaining a git mirror of the mercurial mozilla-central repository. My mirror has the history from the mercurial repository but they both lack the original CVS history as this was not converted when the change was made to use mercurial.

Recently a new git mirror containing the project history, including CVS, has become available in the mozilla-central github account. Now that this is available I will no longer be maintaining my own git mirror. The two repositories use different commit id's so are incompatible. I'll be deleting my repository in a few days to prevent confusion - already I've received a few emails asking why I have mine going when an official repository is available and I'd prefer to avoid people accidentally using it and getting confused when it's not compatible with the main one.

It's great to see a more official git mirror finally become available and thanks to those that have supported my efforts in the past.

Performing Compatible Updates of Mozilla Central Git Forks

2011-10-28T16:00:00+13:00

Earlier this year I posted about how I created my git mirror of the mozilla-central mercurial repository. I've been keeping a fork on github updated regularly since then that a number of people have started using.

Users of my mirror have wanted to be able to keep up to date with the mercurial mozilla-central repository themselves, or add updates from other branches of the Mozilla source. It takes a large amount of time to start a conversion from scratch but it's possible to start from the existing git mirror, perform the incremental updates from mercurial yourself, and still stay compatible with the incremental updates I push to my github mirror.

hg-git uses a git-mapfile in the .hg directory of the mercurial clone to keep track of the mapping between mercurial and git commits. By using the same git-mapfile that I use for my mirror you can start your own incremental update from the latest data in the git-mapfile instead of from the beginning. I keep an up-to-date git-mapfile in git-mapfile.bz2. It's compressed with bzip2 as the uncompressed version is quite large.

The steps to start your own update process are:

Clone the mercurial mozilla-central repository
Clone my git mirror as a bare repository in .hg/git
Place the git-mapfile in .hg
Do an hg bookmark -f -r default master to mark the commit to convert to
Perform hg gexport to update .hg/git with recent mercurial commits

The .hg/git directory should now be up-to-date with respect to the mercurial repository. And the additional git commits will have the same SHA id as any commits I push into my mirror when I perform my own update.

The steps to do an incremental update are the normal:

Pull from the mercurial mozilla-central repository
Run hg bookmark -f -r default master
Run hg gexport
Push or pull to/from .hg/git as needed

As a working example, the following shell commands on a Linux system should set up your own repository ready for incremental updating:

$ hg clone https://hg.mozilla.org/mozilla-central
$ cd mozilla-central/.hg
$ git clone --bare git://github.com/doublec/mozilla-central.git git
$ wget http://www.bluishcoder.co.nz/git-mapfile.bz2
$ bunzip2 git-mapfile.bz2
$ cd ..
$ hg bookmark -f -r default master
$ hg gexport
$ cd .hg/git
$ git push --all ~/some/other/repo.git

Incremental updates just repeat the last few commands above:

$ hg pull -u
$ hg bookmark -f -r default master
$ hg gexport
$ cd .hg/git
$ git push --all ~/some/other/repo.git

Note that I use hg gexport and push from the .hg/git repository using the git command instead of doing an hg push and relying on hg-git to do the conversion and push. In my original article I did the latter. The hg-git push method uses slower python based routines to do the push which can take a long time and large amounts of memory on big repositories like mozilla-central. By splitting this up into gexport and using the native git command I save a lot of time and memory.

My Git Workflow for Mozilla Development

2011-04-16T17:00:00+12:00

Since my post on converting mozilla-central to git I've had a few requests about what my git workflow is for Mozilla Development.

When working on bugs I'll create a git branch for development. The workflow in this branch looks like:

git fetch origin
git checkout -b bug/123456 origin/master
...make changes...
git commit
...make more changes...
git commit

Occasionally I'll want to merge with the latest code from mozilla-central. I do this by rebasing my changes on top of the latest trunk code:

git fetch origin
git rebase origin/master

I'll often have multiple temporary branches as I work on different ideas during the fix. I find being able to diff between the branches, cherry pick patches, etc useful.

When I've finished with the fix and want to generate a patch for review I rebase on top of trunk and squash all my commits down into one patch. I do this use git's interactive rebase:

git fetch origin
git rebase -i origin/master
...squash commits into one commit and set commit message to my checkin commit message...
git hgp >~/mypatch.patch

The last command, git hgp, uses an alias I got from Rafael Espindola's Blog. You can install this alias by adding the following to your ~/.gitconfig file:

[alias]
hgp = show --format=\"From: %an <%ae>%n%s%n%b\" -U8

The adds an 'hgp' git command that does the same as git show but includes a header for the committers name. This allows an hg import of the patch to include the correct patch authors details which is useful if you use the checkin-needed keyword for others to commit the patch.

I attach the patch file to the bug for review. If I get review comments I then go back to the git branch and make the necessary changes. Regenerating the patch involves repeating the steps above.

When the patch needs to be committed to mozilla-central it can be imported and pushed directly using mercurial:

hg clone https://hg.mozilla.org/mozilla-central
cd mozilla-central
hg import ~/mypatch.patch
hg push

Git Conversion of Mozilla Central

2011-02-10T14:00:00+13:00

The Mozilla repository for Firefox development, mozilla-central, is a mercurial repository. I prefer using git for version control so I usually do my Firefox development using a git mirror of the mozilla-central repository.

To convert the repository into git I used hg-fast-export. This has worked well over the last couple of years. Unfortunately I've noticed that my git mirror and the official mercurial repository differ slightly. The git mirror contains files that have been deleted or moved in the official repository and lately the mirror has stopped building due to some commits not being correctly converted. As a result I stopped updating my mirror and used mercurial directly.

I still prefer developing using git though so I tried the hg-git mercurial plugin. This took about six days on my laptop to do the conversion and peaked at 14GB of memory. The result was a git repository with a working tree that, according to diff, is an exact copy of the original mercurial repositories working tree. Another nice feature of the hg-git is that commit id's in the resulting git repository are the same across multiple runs of hg-git. This would allow other users to push and pull from a git repository that they converted independently.

To perform the conversion I installed hg-git and ran the following series of commands:

$ mkdir git
$ cd git
$ git init --bare
$ cd ..
$ hg clone https://hg.mozilla.org/mozilla-central hg
$ cd hg
$ hg bookmark -r default master
$ hg push ../git

Six days and 14GB later, the resulting git repository ended up being about 7GB in size. I did a git gc --aggressive on it to get this down to 200MB. Incremental updates of new mercurial commits are done with:

$ cd hg
$ hg pull -u
$ hg bookmark -f -r default master
$ hg push ../git

I've pushed a mirror of this to github at http://github.com/doublec/mozilla-central. I'm using this for development and will see if it is able to continue importing without diverging from the original mozilla-central source.

More information about the use of git for Mozilla development can be found in the followup posts I did on the subject:

Changes to git mirror of Firefox repository

2008-12-04T17:32:00+13:00

I've made changes to the way I manage the git repository holding the import of the Firefox source. Since the bulk of the HTML 5 <video> implementation is now in the main Firefox source I no longer maintain the video branch. I'll be removing that and the Firefox mirror soon.

The main Mozilla repositories have been split into two now. They are mozilla-central, which holds the current trunk code and represents Firefox 3.2 (or whatever the next version will be called), and mozilla-1.9.1, which is what will be Firefox 3.1. The details of that split are described in this post.

I now import these mercurial repositories directly into git using git's fast import tool. This gives me complete history and makes things like bisect and blame a bit easier to work with. The history of mozilla-1.9.1 being based on mozilla-central is also retained, as best as I could manage. These imported repositories can be browsed at http://gitweb.bluishcoder.co.nz. They can be retrieved via git:

git clone git://bluishcoder.co.nz/git/mozilla-central.git
git clone git://bluishcoder.co.nz/git/mozilla-1.9.1.git

The way I work with these mirrors is to create a git repository that has these added as remotes. This way I can switch between the two easily, cherry pick patches between them, and do logs and diffs between them. This is how I set that up:

$ mkdir firefox
$ cd firefox
$ git init
$ git remote add mozilla-central git://bluishcoder.co.nz/git/mozilla-central.git
$ git remote add mozilla-1.9.1 git://bluishcoder.co.nz/git/mozilla-1.9.1.git
$ git fetch mozilla-central
$ git fetch mozilla-1.9.1

When working on a bug I create a branch for that bug, based on mozilla-central/master:

$ git checkout -b bug123456 mozilla-central/master
...make changes...
$ git commit -a -m "my changes"
$ git diff -U8 mozilla-central/master >bug123456.patch
...attach bug123456.patch to bugzilla for review...

Once reviewed the patch is then applied to mercurial and pushed. If I want to apply the same patch to test on the mozilla-1.9.1 branch I just use 'git cherry-pick'. To make things like 'git status' a bit less clutted I have the following .gitignore:

$ cat .gitignore
obj-*
*~
.mozconfig
config.cache
config.log
*.pyc
CVS
.mozconfig*
.svn
cvsco.log
cvsco.log.old
NONE
configure
gitchangelog
gitchangelog.bak
.hg
.nss.checkout
*.swp
.gitignore

Git, Binary Files and Cherry Picking Patches

2007-09-06T17:03:00+12:00

Steve Dekorte has some things he dislikes about git. This post is how I work around these issues in my own git repositories.

Git has a heuristic for detecting binary files. You can force other file types to be binary by adding a .gitattributes file to your repository. This file contains a list of glob patterns, followed by attributes to be applied to files matching those patterns. By adding .gitattributes to the repository all cloned repositories will pick this up as well.

For example, if you want all *.foo files to be treated as binary files you can have this line in .gitattributes:

*.foo -crlf -diff -merge

This will mean all files with a .foo extension will not have carriage return/line feed translations done, won't be diffed and merges will result in conflicts leaving the original file untouched.

Now when you pull from another repository that has changes to a .foo file you'll see something like:

test.foo |  Bin 32 -> 36 bytes

Note that it shows it is a binary file. If you pull from another repository with changes to test.foo you'll get:

Auto-merged test.foo
CONFLICT (content): Merge conflict in test.foo

The file will be untouched and you can change it manually to be the correct version. Either by leaving it untouched, or copying a new file over it. Then you need to commit the merge conflict fix (even if you left the file untouched):

git commit -a -m "Fix merge conflict in test.foo"

The cherry picking of patches works differently to Darcs. There are a couple of ways of handling this, but I use 'git cherry-pick'. If you have a number of contributers with their own repositories that you regularly pull from you can set up remote tracking branches:

git remote add john http://...
git remote add mary http://...

Now when you want John and Mary's most recent patches you can fetch them:

git fetch john
git fetch mary

This does not make any changes to your local branches. It gets and stores their changes in a separate remote tracking branch. If you want to see what John has changed, compared to yours:

git log -p master..john/master

From there you can decide to pull in all John's commits:

git merge john/master

If you want one commit, but not its dependencies then this is where 'cherry-pick' is used.

Given a commit id, 'cherry-pick' will take the patch for that commit and apply it to your current branch. It's used like:

git cherry-pick abcdefgh

This creates a commit with a different commit id than the original, but with the same contents. It needs to be a different id as it doesn't have the same dependencies as the original.

If you decide later you want all John's commits and do a merge which includes the commit that you cherry picked from you might expect conflicts. Git handles this case fine and does an automatic merge, noticing the patches are the same. So it effectively gives you the same functionality as Darcs selective patch pulling, but not as nice a user interface.