Using Quilt with CVS

From SecretLab

When using CVS to work with third party software packages, traditional wisdom says to check the whole source tree into CVS as a vendor branch and make changes against it on the head branch. However, there are problems with this approach. This article describes a different method of tracking a package in CVS by using patch files and the Quilt tool.

Contents 1 Rational 1.1 Problems with CVS 1.2 Localize and compartmentalize changes 1.3 Advantages for developers 1.4 Tradeoffs (disadvantages) 2 Data Format in CVS 3 Quick Start 3.1 Initial CVS Setup 3.2 Unpacking the source 3.3 Basic commands 4 Workflow 4.1 Add a patch 4.2 Modify a patch 4.3 Merge two patches together 4.4 Review process 5 Best Practices 6 Additional Reading

Rational

Most projects are not interested in maintaining and supporting the entirety of an upstream software package, especially large packages like the Linux kernel. Rather, most projects only want to expend effort to support the project's changes to the upstream package. In the majority of cases, the change size is minuscule in comparison with the package itself. Coupled with that are issues with the CVS tool itself which make tracking a projects changes harder than it should be (as listed below).

Tracking patch sets instead of the full package source in CVS is an approach which shifts support effort away from the entire source tree and onto just the needed changes. The rest of this section describes the specific advantages of using the patch series method.

Problems with CVS

1. CVS makes it too easy to cause collateral damage (modified CVS tags, DOS line endings, etc)
2. Collateral damage is difficult to contain and repair
3. Commits to multiple files are not grouped into changesets
4. Branching is hard (most developers never do a branch)
5. Merging is harder (even fewer do merges)
6. Mistakes can be expensive
   • CVS tags for example. If the tree was checked in without inhibiting tag updates, and it isn't noticed right away, trying to get the damage repaired (say when upreving to a newer version of upstream) requires a lot of work and touching files completely unrelated to the project's changes.

Localize and compartmentalize changes

Storing the entire source tree in CVS is like the needle in a haystack problem. Both developers and tools need to do work to filter out the changes from a source tree, and that is assuming that it is even possible to get a clean diff (see problems with CVS above).

If instead patch files are stored, then only the changes that are actually interesting to the project are revision controlled. Revision control, and hence developer attention, is constrained to only the parts of the source tree which need to be changed.

Patch files logically organize changes based on their purpose. Ie. one patch for each new device driver, one patch for each bug fix against the upstream package, etc. Compare this to storing the entire source tree in CVS which is akin to throwing everything into a single bucket.

Advantages for developers

1. Trivial to apply/unapply changes
   • If a patch is suspected to cause problems, a developer can easily test with the patch removed.
   • Can 'bisect/slice/dice/rearrange' the patch series when looking for bugs (without risk).
2. Rebasing to a different version is easy.
   • Simply change the base software version; either to an earlier or later copy.
   • Patches which are no longer relevant can be dropped from the series without deleting them from the repo.
3. Reduce impact of code review
   • Changes can be checked in before review without the temptation to simply say "It's already checked in, I just won't bother with review".
4. Encourages software development best practices
   • Developers know exactly what they have changed.
   • If a mistake is made, other developers will notice
   • Only reviewed & accepted patches get automatically applied to the build
5. Collateral damage is contained
   • Developer mistakes are contained to a single patch file
   • Mistakes are easily repaired.
6. Experiments become cheap.
   • Developers can add patches to the repository without risk of breaking the build.
   • Other developers can try out experimental patches in the repo with little effort or risk
7. Auditing becomes cheap
   • Only the changes are checked in, everything checked in is subject to review (instead of having the entire source tree and needing to first filter out the unchanged code
   • Description of what the purpose of the patch is is contained inside the patch file.
8. Speed!
   • Checking out a large package from CVS (like the Linux kernel) is slow.
   • Unpacking a tarball and applying a patch series is considerably faster.

Tradeoffs (disadvantages)

It is a different workflow which developers need to learn. Developers must be comfortable with using and reading patch files.

Data Format in CVS

Patches are stored in the 'patches' subdirectory. Each patch is a separate file in 'unified diff' format. Quilt creates a 'series' file in the patches subdirectory which is a plain text list of all the patches and the order in which patches will be added.

The series file can be edited 'by hand' with a text editor to add or remove patches from the series. New patches can be added to the series by simply dropping a patch file into the patches directory and adding the patch to the series file. Similarly, patches can be removed by deleting the patch file and updating the series file.

The patch files themselves are plain ASCII text files. They are generated with the diff command using the -u argument. Below is an example of a patch file that corrects a typo in a well known poem (with apologies to Edgar Allen Poe). First, here is the command to generate the patch file:

$ diff -u poem.txt.orig poem.txt

And here is the output:

--- poem.txt.orig	2009-05-01 14:10:15.000000000 -0600
+++ poem.txt	2009-05-01 14:09:06.000000000 -0600
@@ -1,6 +1,6 @@
 Once upon a midnight dreary, while I pondered weak and weary,
 Over many a quaint and curious volume of forgotten lore,
-While I nodded, nearly napping, suddenly there came tapping,
+While I nodded, nearly napping, suddenly there came a tapping,
 As of some one gently rapping, rapping at my chamber door.
 `'Tis some visitor,' I muttered, `tapping at my chamber door -
 Only this, and nothing more.'

As you can see, the patch file shows the changed line plus a few lines of context both before and after the change. The [1] can read a patch file and apply the change to a source tree.

While patch files are plain text, it is not a good idea to modify the patch files themselves 'by hand'. It is very easy to damage patch files if you are not familiar with the format. It is better to use the quilt and diff tools to manipulate patch files.

Quick Start

Initial CVS Setup

Save the following Makefile and check it into a new directory in CVS:

# Configuration
PACKAGE:=linux
VERSION:=2.6.29.2
SRCNAME:=${PACKAGE}-${VERSION}
TARBALL:=${SRCNAME}.tar.bz2

all: unpack apply-all

unpack:
	tar xfj ${TARBALL}
	ln -s ../patches ${SRCNAME}/

apply-all:
	cd ${SRCNAME} && quilt push -a

clean:
	rm -rf ${SRCNAME}

Then, in the same directory, create a directory named 'patches' and check it into CVS also. The patches and series file will get checked into the 'patches' subdirectory.

Unpacking the source

To unpack the package source and apply the series of patches, do the following from the directory with the Makefile:

$ make

This will unpack the package source tree and apply all of the patches listed in the 'series' file.

Basic commands

$ quilt series -v          # Show list of all patches
$ quilt applied            # Show list of applied patches
$ quilt unapplied          # Show list of unapplied patches
$ quilt push               # Apply the next unapplied patch
$ quilt push -n <N>        # Apply the next N unapplied patches
$ quilt push -a            # Apply all of the unapplied patches
$ quilt push <patch-name>  # Apply the patch named <patch-name> (changes patch order)
$ quilt pop                # Unapply the top patch
$ quilt pop -n <N>         # Unapply the top N patches
$ quilt pop -a             # Unapply all patches

Workflow

Add a patch

• Check out source tree

$ quilt new set-csb-specific-extraversion

• Tag files for change and edit the file

$ quilt edit Makefile    # equivalent to 'quilt add <file> && emacs <file>'
<make a bunch of changes>
$ quilt diff
Index: linux-2.6.29.2/Makefile
===================================================================
--- linux-2.6.29.2.orig/Makefile	2009-05-04 11:02:39.000000000 -0600
+++ linux-2.6.29.2/Makefile	2009-05-04 11:03:20.000000000 -0600
@@ -1,8 +1,8 @@
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 29
-EXTRAVERSION = .2
-NAME = Temporary Tasmanian Devil
+EXTRAVERSION = .2-csbrocks
+NAME = CSB Rock Star
 
 # *DOCUMENTATION*
 # To see a list of typical targets execute "make help"
$

• Refresh the patch

$ quilt refresh

• Check it into CVS

$ cvs add patches/set-csb-specific-extraversion
$ cvs commit patches/set-csb-specific-extraversion

Modify a patch

• Goto the relevant patch

$ quilt series -v
+ powerpc-Remove-unused-symbols-from-fsl_devices.h.patch
+ powerpc-install-Bail-with-error-code-on-error-in-in.patch
+ set-csb-specific-extraversion
= flush-cache-in-bootloader
$ quilt goto set-csb-specific-extraversion
$ quilt series -v
+ powerpc-Remove-unused-symbols-from-fsl_devices.h.patch
+ powerpc-install-Bail-with-error-code-on-error-in-in.patch
= set-csb-specific-extraversion
  flush-cache-in-bootloader

• Make your change

$ quilt edit Makefile
<make more changes>
$ quilt diff     # Shows the new state of the patch

• Refresh the patch

$ quilt refresh

• Check into CVS

$ cvs diff -u
<Make sure diff only what you expect has been changed>
$ cvs commit patches/name-of-patch

Merge two patches together

• Apply the first patch

$ quilt goto first-patch-name

• 'fold' in the second patch

$ quilt fold < patches/second-patch-name
$ quilt refresh

• Delete the second patch from the series

$ quilt delete second-patch-name

Review process

• Email patches to the mailing list for review by the team
• Team members review the patch and comment
• Fix bugs, etc, and update patch. Repeat if necessary.

Best Practices

1. Always use 'cvs diff' to confirm you are checking in what you think you are checking in.
2. Check in your patch files early, even if they are not ready. As long as the patch doesn't appear in the 'series' file, the patch will not be automatically applied when unpacking the source tree
3. Only add a patch to the checked in series file after it is peer reviewed and declared good.

It's a good idea to have two categories of patches in CVS:

• Completed patches
  • Appears in the checked-in series file
  • Peer reviewed and accepted
  • Must not be modified directly (see below)
• Incomplete patches
  • Not yet complete
  • Not yet reviewed
  • Does not appear in the checked-in series file
  • Can be changed at any time.
  • (optional) patch name is prefixed with primary developer's name; to be renamed when complete

Incomplete patches are checked in so that other developers can access them if they need to, but they don't automatically get applied so there is no risk of them breaking the build. It is also a good idea for a developer other than patch author to handle moving the patch from the incomplete to the complete state.

Completed patches are considered to be stable. If additional changes need to be made, then a new patch should be written, tested and reviewed which applies on-top of the completed patch. When the new patch is complete, then it can be merged or 'folded' into the original patch.

Additional Reading

• quilt manual page
• Linux Kernel in a Nutshell Appendix A has a section about Quilt
• quilt home page
• How to Survive With Many Patches paper by Andreas Gr¨unbacher from Suse