project-builder.org: more parallelism

Looking around to improve some of the performances of project-builder.org, I found an interesting perl module: Parallel-ForkManager.

Its man page shows how you can quickly parallelize a loop in order to benefit from the numerous cores that each system today has. It took me just a couple of minutes to transform project-builder.org, so that the generation of all build files becomes a parallel operation, thus reducing dramatically the time it was taking – at least on my Xeon X5680 @ 3.33GHz !!

Very easy as you can see in the changeset, and very efficient. And if you don’t have the module, you can still work in serial mode, but once you tested it, you don’t want to go back ;-)

Next step, is to also apply that feature to the build in itself on multiple VMs or VEs, but that will require a bit more work, as the loop is not so easily usable as of now. More on that later, once the code is re-architectured.

About these ads

Tags: , , ,

2 Responses to “project-builder.org: more parallelism”

  1. project-builder.org: nearly ready for 0.10.1 « Bruno Cornec’s Blog Says:

    [...] on top of that quite a new set of features have been added since last version, in particular the parallelism of tasks. After the work on sbx|cms2build, I’ve done now build2pkg and also build2ve. It [...]

  2. Project-Builder.org: 0.10.1 published and next steps « Bruno Cornec’s Blog Says:

    [...] phases which can drastically reduce build time for packages. For this to work, as explained earlier, you need the perl module Parallel::ForkManager installed on your system. The VMs|VEs may benefit [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 105 other followers

%d bloggers like this: