Looking around to improve some of the performances of project-builder.org, I found an interesting perl module: Parallel-ForkManager.
Its man page shows how you can quickly parallelize a loop in order to benefit from the numerous cores that each system today has. It took me just a couple of minutes to transform project-builder.org, so that the generation of all build files becomes a parallel operation, thus reducing dramatically the time it was taking – at least on my Xeon X5680 @ 3.33GHz !!
Very easy as you can see in the changeset, and very efficient. And if you don’t have the module, you can still work in serial mode, but once you tested it, you don’t want to go back 😉
Next step, is to also apply that feature to the build in itself on multiple VMs or VEs, but that will require a bit more work, as the loop is not so easily usable as of now. More on that later, once the code is re-architectured.
Tags: packaging, perl, project-builder.org, vcs-pkg
Bruno Cornec's Blog 
2011/01/08 at 02:29 |
[…] on top of that quite a new set of features have been added since last version, in particular the parallelism of tasks. After the work on sbx|cms2build, I’ve done now build2pkg and also build2ve. It […]
2011/01/20 at 01:52 |
[…] phases which can drastically reduce build time for packages. For this to work, as explained earlier, you need the perl module Parallel::ForkManager installed on your system. The VMs|VEs may benefit […]