Main

Using conditional build steps to speed up your Jenkins PHP builds

Michiel Rook

At my client Spotney, we have a pretty solid (and common) build infrastructure for our PHP projects; SVN commits are checked out by Jenkins, tests are run by PHPUnit, Sonar runs static code analysis, and artifacts are built and deployed to a staging environment by Phing. However, some of the code relies pretty heavily on (complex) db queries, adding the need for DbUnit style tests. The nature and quantity of the tests, combined with a slow VM (possibly related to this Xdebug issue) meant that our buildtimes were becoming prohibitively long.

An interesting post by @pascaldevink triggered a conversation and sparked an idea. I started working on our build setup, eventually resulting in a 60-70% decrease of our build times!

Here’s how I did it.

Starting point

Let’s assume we have a fairly standard Jenkins job. The job checks out an SVN repository, and periodically scans that repository for changes, triggering a new build if any are detected.

Each build of the job performs three steps:

  • Run phpunit
  • Run phing (target “build”)
  • Invoke Sonar (using the Jenkins Sonar plugin – this plugin also allows invoking Sonar as a post-build action, but that option requires Maven)

After the build steps, the job publishes the test and code coverage reports, and archives the generated artifacts.

Disabling code coverage and Sonar for regular builds

Two of the most obvious optimizations (also suggested by Pascal) are disabling code coverage on all tests and disabling Sonar runs during regular Jenkins builds. We define regular as either manually started by a user, or by an SCM trigger.

Disabling code coverage generation in PHPUnit is easy, simply remove the “coverage-xxx” elements from the logging section of your phpunit.xml configuration file (see this section of the PHPUnit manual). Disabling Sonar is trivial too, just remove the last build step from the job.

However, this is not an ideal solution: we do want to generate code coverage information and run Sonar at some point, such as during nightly builds, preferably without cloning our existing job. This means that we’ll need to skip code coverage and Sonar invocations on all but the scheduled (nightly) builds.

The Sonar plugin supports excluding SCM triggered builds (“Skip if triggered by SCM Changes”), but that only works if you use the post-build action. Additionally, we need to be able to change the PHPUnit configuration – one file to enable code coverage generation, one file to disable it.

Conditional build steps

The Conditional BuildStep plugin wraps one or more build steps in a conditional execution block. One of the possible conditions is the build cause, i.e. was the build triggered by a user, an upstream project, a timer, an SCM change, etc. etc.

First we define the steps that should be taken for each nightly build of our job. These steps should only be executed when the build is trigger by a timer.

We add a “Conditional steps (multiple)” build step, setting the condition to “Build Cause” and the Build Cause to “TimerTrigger”.

Conditional Sonar Build Config [Jenkins]2

Then we add our three (original) build steps:

Conditional Sonar Build Config [Jenkins]3

As stated before, regular builds are those that are triggered by a user or an SCM change.

We again add a “Conditional steps (multiple)” build step. The condition for this step is a little more interesting, as seen below. We combine two Build Cause conditions (one set to “UserCause”, the other to “SCMTrigger”) using the Or condition.

Conditional Sonar Build Config [Jenkins]4

We then add two build steps: the first will run PHPUnit without code coverage (note that we are specifying a different configuration file here), the second one will run Phing.

Conditional Sonar Build Config [Jenkins]5

Note that in the above build steps we’re invoking Phing from the shell instead of using the Phing plugin. Unfortunately this plugin is currently not supported as a conditional build step (probably because of this JIRA issue).

Build schedule

As a final step we need to update our build schedule.

Conditional Sonar Build Config [Jenkins]1

This will ensure our job runs somewhere after midnight (between 12:00 AM and 2:59 AM to be precise).

The end result:

  • A nightly scheduled build, with all the bells and whistles enabled
  • User and SCM triggered builds run (a lot) faster

Please let me know if you think this post is useful, or if you have any other Jenkins/PHP optimization tips!

Phing development update

Michiel Rook

The last Phing development update was almost a year ago, so I guess it’s about time for a discussion of what’s new in Phing world.

2.5.0 and Semantic Versioning

The most important news in this update is the brand new 2.5.0 release (pushed out yesterday). We bumped the minor version number because we now (try to) follow the Semantic Versioning spec, and this release introduces new functionality. Our changed roadmap reflects this as well.

This version closes the following issues:

#979 svncommit: invalid switch ignoreexternals
#977 phpunit Task doesn’t support @codeCoverageIgnore[…] comments
#972 SvnCopyTask: remove “force” from documentation
#971 TokenSource does not work
#969 PHPUnit task does not report diffs for failed assertions
#968 Proper handling of STDOUT and STDERR
#963 XSLT task fails with fatal error on PHP 5.4
#962 DbDeploy: infinite loop in case if directory not found
#961 DbDeploy: checkall output isn’t informative
#960 Documentation of Dbdeploy task
#959 Bug in SvnListTask Version 2.4.14
#958 Property wrapped in if/then structure is not substituted by it’s value
#954 Paths becoming part of S3 file names on Windows
#953 Add PHP extension check to Available Task
#952 Properly document how to load environment variables as properties
#951 S3Put throws “Source is not set” exception
#949 SymfonyConsoleTask improvements: checkreturn and output of command

 

Other goodies

The past few months Phing has seen a few interesting additions:

  • Composer packager support (we’re listed on Packagist as well)
  • phploc task
  • apply task
  • improved phpdoc(2) support
  • lots of documentation, bug and other fixes

GitHub

The move to GitHub in early 2012 has been very successful in speeding up Phing’s development. We can definitely still use your help though, so fork our project and submit a pull request. Thanks!

Phing development update 01/04/2012

Michiel Rook

Roughly a month ago I posted the first in, what should become, a series of development updates concerning Phing. This second blog post details the most important updates during the past four weeks as Phing is moving steadily towards another release (2.4.10 is due out next week).

User Guide starts move to DocBook format

The biggest commit last month was without a doubt the merge of Johan Persson’s docbook5 branch, which adds a DocBook 5 version of the entire Phing User Guide,  accompanied by its own rendering infrastructure. This feature has been on my personal wishlist for a long time, and it’s finally around the corner! The new docs will probably not make it into 2.4.10, but, at some point in the coming weeks, the ‘live’ user guide (on the Phing website) will be switched to the new format. The 2.4.11 packages will also contain the new docs.

Liquibase docs

After succesfully merging the Liquibase tasks last month, Stephan Hochdörfer completed the addition by submitting a pull request containing the necessary documentation!

PhpDocumentor

You’ve probably heard about this, the popular documentation tools phpDocumentor and DocBlox have merged! Phing supports the new phpDocumentor releases by adding a phpdoc2 task.

PHP 5.4 compatibility

Phing itself seems to work fine with the recently released PHP 5.4, however, the unit tests experienced some problems – these have been corrected to make sure the (unstable) snapshot builds are pushed out regularly.

PHAR package

Starting with the upcoming 2.4.10 release, each new version of Phing will also be available as a PHAR package. Initially, this will just contain Phing itself (similar to the PEAR package). After that, the package will be expanded with a few popular dependencies (depending on feedback).

Various (bug)fixes, the 2.4.10 release

Last but not least, here’s a list of tickets that have been closed since the last update.

#870 Can’t find ParallelTask.php 6 days
#828 SelectorUtils::matchPath matches **/._* matches dir/file._name 12 days
#844 symlink task – overwrite not working 12 days
#840 Prevent weird bugs: raise warning when a target tag contains no ending tag 12 days
#820 Type selector should treat symlinks to directories as such 2 weeks
#868 Git Clone clones into wrong directory 2 weeks

My Ideas For March

Michiel Rook

Last year, Chris Shiflett started the Ideas for March movement, an effort to revitalize the interesting conversations that happen(ed) on blogs. This year, several people are refreshing that effort.

This is my pledge to write (blog) more about what I see and do, the projects that I work on, the tools that I use daily, the people I meet, the events I visit – the things that inspire me. I will be blogging about all things Phing, hopefully post a blog or two about Gearman, SOAP, ZF, security and numerous other subjects that have been stuck in my grey matter, but never managed to make it into a blog post.

Plus, and I’m very excited about this, my good friend Joeri is busy working on a redesign of this site, which will hopefully make all that new content more appealing and easy to read!

Phing development update

Michiel Rook

Starting today, I’ll post regular updates on the development of Phing (which is, as you might know, a PHP build tool based on / inspired by Apache Ant).

For the past three years I’ve been the lead developer on this open source project, and thus responsible for deciding the direction of the project, integrating the various contributions, publishing releases, etc. Should you be interested, Lorna Jane Mitchell published an excellent post detailing some of the challenges faced by open source project leads.

The last few months have been especially interesting, as I’ve moved the Phing source code to GitHub. This has greatly improved the number of contributions and in general speeded up the development of Phing.

Let’s discuss some of the interesting commits that were made during the past two months!

Try/Catch task

A feature that has been requested before (and has been on my personal wishlist) is the ability to run task(s) whenever a task fails (very similar to PHP’s try/catch statement). During my talk at the PHPUK2012 conference, one of the attendees asked when this feature could be expected. So, while waiting for my flight home I decided to get busy and add the trycatch task.

<trycatch property="prop.testTryCatchFinally">
	<try>
		<fail>Tada!</fail>
	</try>

	<catch>
		<echo>In &lt;catch&gt;.</echo>
	</catch>

	<finally>
		<echo>In &lt;finally&gt;.</echo>
	</finally>
</trycatch>

Parallel task

After a very interesting talk by Jeroen Keppens (@jkeppens) during this year’s PHP Benelux conference, Mike van Riel (@mvriel), the author of DocBlox, and me set out to enable some form of parallel processing in both Phing and Docblox. Mike authored a generic library, which I used in the new parallel task. This task is still very experimental (and it only works on Linux due to some OS limitations), but it’s definitely a nice example of how the community works and inspires!

<parallel threadCount="100">
	<echo>Job 1</echo>
	<echo>Job 2</echo>
	<echo>Job 3</echo>
	<echo>Job 4</echo>
	<echo>Job 5</echo>
	<echo>Job 6</echo>
	<echo>Job 7</echo>
	<echo>Job 8</echo>
	<echo>Job 9</echo>
	<echo>Job 10</echo>
	<echo>Job 11</echo>
	<echo>Job 12</echo>
	<echo>Job 13</echo>
	<echo>Job 14</echo>
	<echo>Job 15</echo>
	<echo>Job 16</echo>
</parallel>

Liquibase

Together with Stephan Hochdörfer (@shochdoerfer) I’ve been working on integrating his Liquibase (a Java database refactoring / change management application) tasks into the Phing core. The liquibase tasks will complement the dbdeploy task and assist the user with database changes / rollbacks.

ApiGen

ApiGen is a Nette framework based documentation generation tool for PHP 5.3. Phing now supports this tool through the new apigen task, contributed by Jaroslav Hanslík.

RNG schema updates

For a while now, Phing ships with a RNG schema to validate your build files. However, this schema was incomplete and did not contain some of the more recent tasks and changes. Johan Persson has done some invaluable work to get the RNG schema up to date.

Various (bug)fixes, the 2.4.10 release

Steady progress has been made towards the 2.4.10 version (which is scheduled to be released early April). Below is an overview of the tickets that have been closed so far:

#519 Extend mail task to include attachments 5 days
#334 Run a task on BuildException 6 days
#849 Symfony 2 Console Task 2 weeks
#835 JSL-Check faulty 2 weeks
#850 Typo in documentation – required attributes for project 3 weeks
#853 PHP Error with HttpGetTask 3 weeks
#671 fix CvsTask documentation 4 weeks
#852 Several minor errors in documentation of core tasks 4 weeks
#851 RNG grammar hasn’t been updated to current version 4 weeks
#790 Make it easy to add new inherited types to phing: Use addFileset instead of createFileset 5 weeks
#847 Add support for RNG grammar in task XmlLint 5 weeks
#846 RNG grammar is wrong for task ‘foreach’ 5 weeks
#833 Exec task args with special characters cannot be escaped 5 weeks
#587 More detailed backtrace in debug mode (patch) 5 weeks
#834 ExecTask documentation has incorrect escape attribute default value 5 weeks

Besides those commits, a good amount of other (small) fixes and contributions will make it into 2.4.10. A complete list of commits can be found here. Keep those pull requests coming! 🙂