more parallelism

Looking around to improve some of the performances of, I found an interesting perl module: Parallel-ForkManager.

Its man page shows how you can quickly parallelize a loop in order to benefit from the numerous cores that each system today has. It took me just a couple of minutes to transform, so that the generation of all build files becomes a parallel operation, thus reducing dramatically the time it was taking – at least on my Xeon X5680 @ 3.33GHz !!

Very easy as you can see in the changeset, and very efficient. And if you don’t have the module, you can still work in serial mode, but once you tested it, you don’t want to go back 😉

Next step, is to also apply that feature to the build in itself on multiple VMs or VEs, but that will require a bit more work, as the loop is not so easily usable as of now. More on that later, once the code is re-architectured.

Tags: , , ,

2 Responses to “ more parallelism”

  1. nearly ready for 0.10.1 « Bruno Cornec’s Blog Says:

    […] on top of that quite a new set of features have been added since last version, in particular the parallelism of tasks. After the work on sbx|cms2build, I’ve done now build2pkg and also build2ve. It […]

  2. 0.10.1 published and next steps « Bruno Cornec’s Blog Says:

    […] phases which can drastically reduce build time for packages. For this to work, as explained earlier, you need the perl module Parallel::ForkManager installed on your system. The VMs|VEs may benefit […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: