BuildContrib Cookbook

This is a cookbook of recipes for using BuildContrib.

BuildContrib is a powerful, multi-faceted build system, used to build TWiki extensions and the TWiki core itself. It started out as a simple perl clone of ANT, but has gained a diverse range of other functionality since then.

Extensions should be checked into subversion, so that's assumed here. Anyone who develops outside subversion is asking for trouble, and is pretty much on their own. See SubversionReadme for help getting started with it.

Insert the name and type of your module below, and hit "rewrite instructions" to rewrite the instructions for your module. See http://twiki.org/cgi-bin/view/Plugins/WebHome for a description of the different types of extension module.

Recipe 1: Setting up your dev environment

The best way to dev is in a subversion checkout. See SubversionReadme for help in checking one out. Once you have a checkout area, follow the installation instructions to configure it as a running TWiki so you can test your code in the browser. Then set these environment variables.

• export TWIKI_HOME=/home/twiki or the equivalent for your shell.
• export TWIKI_LIBS=$TWIKI_HOME/core/lib:$TWIKI_HOME/core/lib/CPAN/lib or the equivalent for your shell. Note that it's a path with two directories on it.

The beauty of using a checkout area is that if you make a boo-boo, it's easy to revert your changes, and you get the entire published Extension Library to crib, extend or work from. And you are always testing against the latest code.

Once you have a working TWiki, you need to checkout BuildContrib, if you haven't already done that:

cd /home/twiki svn co http://svn.twiki.org/svn/twiki/trunk/BuildContrib BuildContrib 

Now you have checked it out, use pseudo-install.pl to link it into the core, thus.

cd /home/twiki/core perl pseudo-install.pl !BuildContrib 

If you want to be able to check your work into Subversion, you need to be authorised. See SoYouWantToBeATWikiDeveloper

Recipe 2: Creating a new extension

This recipe describes the process of building a new extension called ElectricPlugin, using the create_new_extension.pl script installed with BuildContrib.

1. cd /home/twiki
2. perl BuildContrib/create_new_extension.pl ElectricPlugin

This will create the ElectricPlugin file hierarchy, including build.pl, DEPENDENCIES and MANIFEST. At this point, the Extension Topic (and all other files) are copied from Empty(Plugin/Contrib), and you will need to change those to reflect what you are working on.

Recipe 3: Making MANIFEST and DEPENDENCIES files

The MANIFEST file lists files that are released with the extension. Not all files in the source tree will be released; for example, it is unusual to release developer tests and support scripts such as those used for benchmarking.

The DEPENDENCIES file lists external dependencies, such as those on CPAN modules, or other TWiki modules.

After you run create_new_extension.pl the initial MANIFEST and DEPENDENCIES files are empty. You can generate draft contents thus:

1. cd /home/twiki/ElectricPlugin/lib/TWiki/Plugins/ElectricPlugin
2. perl build.pl manifest
3. perl build.pl dependencies (needs the CPAN module B::PerlReq)

These targets "guess" the content of the files; you need to apply some intelligence to derive the actual content.

You can run these targets whenever you want, even after you have generated what you believe are correct MANIFEST and DEPENDENCIES files.

Recipe 4: Making unit tests

We strongly recommend using test-first development, and ensuring that you have unit tests in place for all your extensions. Unfortunately many people are too lazy, or too stupid (like the author of BuildContrib), to write tests for their extensions. If you are one of the wise:

1. mkdir /home/twiki/ElectricPlugin/test/unit/ElectricPlugin
2. Create a testcase. The easiest way to do this is to base it on an existing testcase. For an example, see /home/twiki/CommentPlugin/test/unit/CommentPlugin.
3. To run the tests, cd /home/twiki/ElectricPlugin/lib/TWiki/Plugins/ElectricPlugin and perl build.pl test
   • build.pl will print the command line used to run the tests, so you can easily copy-paste-modify the command line to home in on a failing test
   • do not ignore test failures - maintaining a quality test suite is a major help to yourself and others in maintaining extensions

Recipe 5: Debugging unit tests

Sometimes a unit test will fail (that's why you have them, after all) and you will need to debug. This is not so easily achieved with build.pl since it fires off separate processes to do the hard work. Fortunately there are some simple tactics you can use to help debug a test. Say that you want to know why, in your ElectricPlugin, test_Something reports a failure though it is supposed to work.

Recipe 5a: Isolate the failing test

1. grep test_Something /home/twiki/ElectricPlugin/test/unit/ElectricPlugin/*.pm to find the .pm that contains the failing test
2. Run perl build.pl. Note the command line it prints as it starts the test run. It'll be something like this:
   perl -w -I/home/twiki/ElectricPlugin/lib -I/home/twiki/core/lib -I/home/twiki/lib -I/home/twiki/lib/CPAN/lib -I/etc/perl -I/usr/local/lib/perl/5.8.8 -I/usr/local/share/perl/5.8.8 -I/usr/lib/perl5 -I/usr/share/perl5 -I/usr/lib/perl/5.8 -I/usr/share/perl/5.8 -I/usr/local/lib/site_perl -I. /home/twiki/TWIKI4/test/bin/TestRunner.pl /home/twiki/ElectricPlugin/test/unit/ElectricPlugin/ElectricPluginSuite.pm
3. Copy the command and paste it into the shell. Change the name of the .pm that is run to the one you found in step 1 that contains the failing test
4. Edit the source of the failing test, and change the names of all tests that are not failing to something that doesn't start with test. I usually s/sub test/sub detest/g
5. Run the command. Now the single failing test is the only one run, allowing you to focus in on the problem with print STDERR statements, or.....

Recipe 5b: Use a debugger

The following recipe will guide you to a Perl debugger session which stops at the first line of sub test_Something. It is less scary than it looks at first glance, you only need to type in what's in boldface.

1. $ cd /home/twiki/ElectricPlugin/lib/TWiki/Plugins/ElectricPlugin
2. $ export PERL5OPT=-d
   • ...or the equivalent for your shell. This environment variable contains options which well be passed to all invocations of perl from now on.
3. $ perl build.pl test
   • Fire off the tests.
4. Loading DB routines from perl5db.pl version 1.28
(...)
main::(build.pl:48): $build = new BuildBuild();
DB<1> c
   • Just continue, we are not going to find bugs in build.pl (we hope!)
5. Running tests in /home/twiki/ElectricPlugin/test/unit/ElectricPlugin/TemplateToolkitPluginSuite.pm
(...)
main::(/home/twiki/test/bin/TestRunner.pl:3):
3: require 5.006;
DB<1> b postpone ElectricPluginTests::test_Something
   • Daughter DB session, maybe in an new xterm. At the current line, your test suite has not yet been compiled, that/home/twiki/TWIKI4/test/bin/TestRunner.pl /home/twiki/ElectricPlugin/test/unit/ElectricPlugin/ElectricPluginSuite.pm's what postpone is for.
6. DB<2> c
   • Start TestRunner.pl. Some test cases may run before test_Something (the ordering is hardly predictable), but eventually...
7. ElectricPluginTests::test_Something(ElectricPluginTests.pm:57):
57: my $this = shift;
DB<3>
   • Eventually, you're where you can start tracing. Phew! After you're done, you have to quit ( q) twice, for both parent and daughter session.

Final hint: If all your test cases fail, but perl never stops at the break points you set with this recipe, set a breakpoint at the init routine: b postpone ElectricPluginTests::set_up

Recipe 6: Testing your release package

You build and test your release package as follows. For this you need two shells, we'll call them "shell 1" and "shell 2", and a browser.

1. If you followed recipe 5, remember to perl pseudo-install.pl -uninstall ElectricPlugin first, or bad things will happen
2. In shell 1, cd /home/twiki/ElectricPlugin/lib/TWiki/Plugins/ElectricPlugin and then perl build.pl release.
3. In shell 2, cd /home/twiki and perl ElectricPlugin/ElectricPlugin_installer. This will run your freshly built instaler script. It should find your just-built ElectricPlugin/ElectricPlugin.tgz and offer to install it. Let it.
4. Test in the browser, check the documentation etc etc

The reason for using 2 shells is so if you make a mistake, you can edit and rebuild in shell 1 without wearing your fingers out cd'ing back and forth.

You can use a similar approach to installing in another TWiki install on the same machine - for example, if you want to test against an earlier release.

1. In shell 2, cd to the root of the other TWiki installation (say /home/old)
2. export TWIKI_PACKAGES=/home/twiki/ElectricPlugin to point to the package you just built
3. perl /home/twiki/ElectricPlugin/ElectricPlugin_installer, and it will install in /home/old

Recipe 7: Uploading your work

The best way to upload your release package is to use the upload target.

cd /home/twiki/ElectricPlugin/lib/TWiki/Plugins/ElectricPlugin

and then

perl build.pl upload

It will offer to upload to TWiki.org (you can change this to upload to another repository, for example if you are using BuildContrib to build company-specific plugins). Use your TWiki username and password. It will cache them to a local, unencrypted file, $HOME/.buildcontriblogin! If you can't use the upload target, then manually upload/attach the files that were built when you did perl build.pl release.

Note that the perl build.pl upload command will attempt to upload any attachments in the metadata of the Plugin topic. If your plugin has attached any files that you don't intend to upload, delete the metadata or the upload will fail with missing file errors.

-- Contributors: CrawfordCurrie, SvenDowideit, HaraldJoerg, GeorgeClark

Discussion

I've added "Debugging unit tests" after one long evening's cursing why all my "usual" debugging methods would fail. The recipe will shorten considerably if we could add either an additional option like perl build.pl -d test, or an additional target like perl build.pl test_debug.

-- HaraldJoerg - 27 Sep 2006

In order to get build.pl to work for a plugin, I had to add a symlink, linking twiki/core/lib -> twiki/lib.

-- GeorgeClark - 29 Aug 2008

That shouldn't be necessary, George. The latest BuildContrib knows about /core/. Can you please raise a bug report for this and tell me what happens when you don't have the symlink. Thanks.

-- CrawfordCurrie - 29 Aug 2008

Opened TWikibug:Item5970 for two errors in build.pl that I encounter on my system.

-- GeorgeClark - 30 Aug 2008

"Step 6: Testing your release package" I had to copy the ToolTipPlugin.tgz file from the twiki/ToolTipPlugin directory into twiki/core. If I followed the steps without copying the file, the installer downloaded a copy of the prior version and installed the wrong version.

Suggest Step 3 should say. cd to /home/twiki/core and copy the .tgz release file into the current directory. Step 4 should then be to run perl ../ElectricPlugin/ElectricPlugin_installer and reply "y" when prompted to use the existing file.

-- GeorgeClark - 18 Sep 2008

Based upon all the other instructions on this page, when you set TWIKI_HOME to, for example, /home/twiki, should not TWIKI_LIBS be then set to /home/twiki/core/lib (rather than /home/twiki/lib)? I would change it myself, but I'm having tons of problems with this, so I'd rather not drag anyone down with me.

-- AaronLWalker - 2010-12-01

Yes, you are right! Please feel free to fix above generator.

-- PeterThoeny - 2010-12-02

Made the following change:

 - * =export TWIKI_LIBS=$TWIKI_HOME/core/lib:$TWIKI_HOME/lib/CPAN/lib= or the equivalent for your shell. Note that it's a path with two directories on it. + * =export TWIKI_LIBS=$TWIKI_HOME/core/lib:$TWIKI_HOME/core/lib/CPAN/lib= or the equivalent for your shell. Note that it's a path with two directories on it. 

-- Terje Ness Andersen - 2013-08-28