Posts Tagged ‘project-builder.org’

project-builder 0.16.1 is now available

2020/08/19

When you start updating something, you often finish working on a unrelated but necessary topic before being able to finaly solve your first problem 🙂

So I had received remarks on issues to install MondoRescue on latest Ubuntu vesions (18.04 and 20.04) because of the lack of afio and also star in the distribution. Indeed both have been droped in Debian since quite some times in fact 😦

So in that case you have various choices :

  1. give up (not my state of mind anyway !).
  2. point them to the sources of the tools so they help themselves.
  3. move to more supported tools (that’s something that is now planned).
  4. provide users with the appropriate packages.

The two last possibilities are the nicest to me and the users. So I first build a round of packages for afio for all latest Debian and Ubuntu distributions, at least to unblock users. I already had the build infra in place, and a spec file for rpm, so wasn’t too hard to do it. Next one on the list is s-tar. And that’s where the perturbation happened 😉

I realized I had a hole in project-builder as it was always supposing that upstream tar files were in tar.gz format. That’s not a problem when you develop the upstream project yourself, because you generate indeed these files, but when upstream is s-tar and provides tar.bz2 files, suddenly you have a bug to solve 😉 Interestingly enough I didn’t had any issue since 13 years I’m working with the tool.

It wasn’t that hard to fix as the places referencing a source tar.gz file were easy to find, but that had implication also in the 150+ VMs and CTNs I’m managing to build packages. All of them also needed to have a project-builder.org update in order to support that correctly. Now project-builder doesn’t make any assumption on the nature of the tar file (but still requires a tar file BTW). A lazy solution would have been to just unpack/repack s-tar and voilà ! But I want to be able to use the upstream delivery untouched (checksum included).

So due to that project-builder.org 0.16.1 (to show incompatibility with 0.15) is on its way to the ftp server, and built at the moment for the various distributions. Then i’ll have to update all my VMs and CTNs with that new version, and finally I’ll be able to generate s-tar packages.

Then I’ll be back to what I’m really working on, which is the 3.3.0 of MondoRescue !!

Code is not confined !

2020/04/05

Well, people are, at least in my country (France). But code, isn’t and can still be delivered on server once new versions are available !

And I have been able these last days to work more intensively of my code, which was well needed. As I still want to publish MondoRescue 3.3.0 I have to put my build farm in order to do that correctly. MondoRescue’s code is frozen for now, but project-builder.org‘s wasn’t and was in need of fixes. So I solved all the one I wanted, and I’m now publishing version 0.15.3, and have updated all my VMs (170) and containers (40) with it, so ready to start next week on building MondoRescue, once project-builder.org‘s own build is done.

Here is a list of what’s new with this version:

Fix #177– newproj command works again
Fix #184– with lots of performance improvements for sbx2build phase, back to early days
Feature– Adds internal YAML availability as last resort
Fix #180– by adding the pb_check_v function
Fix #181– with working *2docker commands to build the full stack for a project
Fix #183– by pushing the repomd.xml.asc file for SUSE and fix also repo names
Fix #140– where one extra line was generated in the pbcl
Feature– Adds install2pkg command
Fix #171– by initializing correctly pb_env_init in pbgetparam
Feature– Adds support for fedora 31, centos 8

As I write, code is appearing on ftp://ftp.project-builder.org/ and you’ll be able to use it starting tomorrow use the instructions to install from packages for your distribution (urpmi, dnf, yum, zypper, apt are your freinds !)

An end and a new start

2019/03/22

After nearly a year, I’ve now been declared in remission with my lymphoma. So that’s the end of this period. But I’m not completely done, as I’ll need to regularly check it’s not coming back. Being cured, is only possible after 5 years of remission. And I now need to return all the care my wife gave to me to her, as she was diagnosed with a breast cancer back in october 2018, operated immediately and she’s now in the mid of her own chemiotherapy, before radiotheray. So the real end should rather be around june this year, if everything continues to be for the best for her as it also seems to be (she had a much less streamlined relationship with the hospital, but that’s another story).

So now, it’s time to start again working full time to HPE (next week officially, even if with the recent events, I’ve been more than half time) and to come back to my FLOSS projects that have been suffering themselves meanwhile.

And the first one, as it’s the basis of the rest forme, is project-builder.org. I already had made modifications to use YAML type of configuration files, instead of my own format up to now (well, when I started back in 2007, YAML wasn’t as popular as it is today) but didn’t had time to test on other projects and publish and of course I also fixed some bugs such as:

So as of now, version 0.15.1 of project-builder.org is available and ready for you to test and thus improve your Continuous Packaging approach as part of your automation chain. Enjoy !

And now that my production toolset is ready again, I can come back at last to MondoRescue to publish that version that as been waiting for way too long now. Expect news in the coming weeks.

Meet during the Open Source Summit in Prague

2017/10/16

Getting approval for my participation at that conference was the toughest I ever had in the past, but I’ll be there again after having spoken in all the previous editions.

This time, this presentation will be made by a duo, as I’ve worked since the last LinuxCon EMEA with the FOSSology project to help them adopt a Continuous Packaging approach and Michael Jaeger from Siemens who leads the project will be with me to cover the topic. On my side, I’ll explain how project-builder.org was used, the setup done with Vagrant and containers to generate packages for 7 different Linux distributions (ongoing effort as we speak ;-), automating fully the production of packages as part of the LinuxFoundation build infrastructure for FOSSology. And of course the changes I had to make to the code to better support the project needs !

So I’ll be back in Prague (no problem as I love that city !) next week, to celebrate also the 10 years of project-builder.org, just after my week at DockerCon which is really starting today !

Meet at DockerCon Europe

2017/10/12

This is a late announce, as I wasn’t sure to be able to attend the event, but now everything is on track and I’ll be at DockerCon Europe next week ! One of my proposals has been selected as a 20′ session in the Community Theater and I’m glad as this is the first time I can participate to DockerCon. Project-Builder.org is using Docker since September 2014, so I’ll be able to celebrate the 3 years of Docker usage in pb after the 10 years of pb itself 😉 More seriously my goal is to promote again Continuous Packaging and demonstrate how beneficial it has been to introduce Docker in the chain, and how my upstream projects, as well as FOSSology are benefiting from it. Come to see that !

Of course, HPE is a platinum sponsor of the event so if you’re one of our customers, I’d also be happy to talk with you around your Docker projects as HPE is sponsoring my travel there ! (Thanks to the management to support again this despite hard times for travel requests and the HPE Next plan). Maybe I’ll still have a job when I’m back … or not 😉 Anyway, will see later as I’m now busy preparing my slides and demo for DockerCon among other stuff. See you there !

10 years of Project-Builder !

2017/09/24

Well, I think it’s worth to celebrate that milestone ! I’m now publishing officially the version 0.14.6 of project-builder.org for its 10th anniversary.

Everything started to solve an issue I had when I took over the maintenance of the MondoRescue project. It was very difficult in 2005 to consume the project. People had to download a tar file, follow a very succint recipe to build and install it and then being finally able to use it. The tool was supporting a lot of Linux distributions, but was difficult to use by these same distributions. So I started to build some shell scripts to help me deliver Linux distributions packages to solve that. I realized that I was duplicating stuff to address Fedora and RHEL, SuSE and SLES, Mandrake and Mandriva, … Even when building for other types such as Debian and Ubuntu or Gentoo. Typically all these distributions want a summary, a description, dependencies, changelog, … And I hate duplicating stuff. Also I realized I wasn’t the only one in need of this type of support. LinuxCOE, a peer project at HP was also in need of this, as well as many others.

So in 2007, I decided to rewrite my scripts in a more powerful language, perl (thanks Larry Wall ! and BTW I continue to find it way superior to a lot of younger ones, calling themselves more modern :-)). And version 0.5 was soon published as I used my vacation that year to code a lot for it.

It was way before I started to blog, and blog about it was among the first articles I decided to write. Of course, along the years, features have been developed, capabilities extended. I’m now able to generate project-builder.org it self and MondoRescue for 130 tuples of distributions. THat’s a proof this is possible, and that a tool like that can help achieving it.

10 years ago, I had to invent a configuration file format to support my needs. I wanted to have the least possible number of hard coded elements in project-builder.org. I created my own format, pretty simple mimicing the perl hash being: keyword param = value[,value]. At that time, YAML and JSON weren’t that popular, and I wasn’t aware of perl modules to support these, despite the fact they were existing, and my knowledge of CPAN wasn’t that accurate.
But at least I had inheritance mechanisms, and fucntions to deal with it, whch proved to be extremely useful to migrate.

So It’s now time, for the next 10 years to adopt a format, nearly as simple as mine, as easy to map to perl hashes as mine, and I chose YAML, benig lazy and finding it easier to manage manually (like mine was). Of course, this will have some consequences, as some old build environment don’t provide it so I may drop support for some very old distributions. If needed BTW something can be done as we have the source !! Version 0.15.1 is already in progress, and project-builder.org is now able to build itself, including in VMs or Docker containers for other distributions using that new format. And due to the mechanisms in place, I’ve kept easily all the features, just moving to a new format. I’ll start to work soon on migrating the conf files of MondoRescue to have another project working with that version. I’m also working in HPE with another team that is starting to use it as well, and I chose to adopt for them the new version to benefit from the way forward.

Finally, as I mentioned previously, the tool is now in use by the FOSSology project as well in the Linux Foundation build infrastructure. With the latest version 0.14.6, now FOSSology builds at least native packages for CentOS 7 on a CentOS 7 platform. Now we’ll work on the generation of packages for the other distributions the project want to support (which will be pretty easy now the infra is in place), in order to cover the topic during the Open Source Summit in Prague in October, providing I get approval to travel there (as I’m traveling quite a lot these days).

So happy birthday to project-builder.org and let’s start working on the next YAML based version, and help more projects adopt it !

It’s been a while

2017/09/12

When I look at the date of the last post, indeed it’s been a while since I took time to write here.

Many reasons to that, I took time off this summer to lead the infrastrcuture track at the RMLL, then to make music, for the 28th year in a row with 3 concerts and a diaporama, to visit Normandy with part of my family (Caen, Rouen, the Bayeux Tapestry, Honfleur, and many other places), back to work for some days and then doing another concert with the Variations ensemble I lead around François Ier in Monêtier les bains.

Then back to work. On top of my normal activities for HPE, I tried to incorporate various fixes into MondoRescue to finally publish 3.3.0, but that has again been delayed 😦 It seems I have a hard time publishing it, despite getting good patches or making them myself. I destroyed a system while doing tests, which forced me to update to Mageia 6 published meanwhile to put it back on track ! That didn’t help, and also I now need to check again but in VMs to avoid that issue and fix it for good (was too confident my modified code was working, but I was obviously wrong !). This is linked to fixes in mindi in order to fix size of initrd and its automatic computation. THat should be part of 3.3.0 as a consequence.

In order to solve support for multiple GPG keys signing my packages, I’ve had to go back to project-builder.org, and enhance it. Which lead to the publication of 0.14.6 (still not fully finished, despite code being frozen). And also I had work during my vacation time on chaging the format of project-builder.org conf files, from our own, to a YAML based one. For now, it’s just a change in format, not a semantic one, but that ill allow to improve some aspects, and also I get for free a syntax analysis with the perl YAML.pm module, which is nice (add -v to get the conf file concerned). So version 0.15.1 should come soon, once my 2 main users outside of myself won’t complain about bugs anymore.

One of these projects is FOSSology, and I should present the joint result of our common work with Michael Jaeger on the build infrastructure we put in place for the project to do Continuous Packaging at the upcoming Open Source Sumit in Prague in October, but it seems HPE doesn’t want to support my trip (I still didn’t get travel approval yet) and I may have to pay for it myself for the first time since I started talking at this conference back in …Prague in 2011. If you know anybody who would sponsor me, that would be great !!

Speaking of conferences, I’m this week at the Open Source Summit in Los Angeles, as part of my HPE job, where I lead a round table this morning on containers (are they the future of IaaS ?).

So despite having been quiet for long, Ive been busy as usual, and trying to improve the software I’m contributing to, andour global ecosystem, as well as the musical one. Hopefully you’ll get more news soon, as new releases and travel arrive !

Meet at the RMLL 2017

2017/06/07

This year the RMLL are coming back, from the 1st to the 7th of July and in my region, in St Etienne ! And this year, I am not only a speaker, but also a co-track leader for the Server track.

With my talks, You’ll be able to learn about the latest news for MondoRescue, project-builder.org, Docker, Redfish and python-redfish, Mageia and much more from my co-speakers (Ethereum, CMS, Cloud Native app, Unikernels, …) and even more from Keynote speakers as well.

Lodging is now available, so book quickly and let meet us there !

This week, gift continues with project-builder

2017/03/08

Normally, today I receive gifts, priviledge (really ?) of being older 🙂 But I can also make gifts myself ! So after python-redfish 0.4 yesterday, I’m announcing the release of project-builder 0.14.3 today (and maybe more to come later this week !)

This project seems to have a quicker pace than MondoRescue, maybe due to the fact I’m using it daily, and that it’s now used by python-redfish, MondoRescue, and also FOSSology (WIP more on that later again). I realize I didn’t announced previous versions 0.14.1 last July, and 0.14.2 last January, but lots have been added since.

I’ve worked on this version to improve deb support (no arch pkg) and especially deb repositories (with GPG keys) as well as OpenSUSE GPG signed repositories as well. Previous versions brought support for latest distros (Fedora 25, Ubuntu 16.10 e.g.), robust build using Docker in addition to VMs or chroot VEs, fixes some issues around conf files analysis, improving rpm package name for smooth upgrade between test and stable versions, renaming temporary dirs for consistency, as well as moving the main conf file outside of /etc reserved for local modifications, … Changelogs have all the nasty details.

One of the task I’d like to make now is to move the conf file format to JSON, to benefit from existing libs to manage them, and make them even easier to deal with. That would allow some other improvements afterwards, but too early to talk about.

Oh, and BTW, we decided to also publish python-redfish 0.4.1 (with its packages made with project-builder.org to keep this article’s focus 😉 ) to fix a nasty error we made when pushing to pypi. Enjoy as well !

Meet during LinuxCon EMEA

2016/09/23

After having talked on project-builder.org during the latest LinuxCon NA in Toronto, it’s now time to prepare for LinuxCon EMEA in Berlin.

I’l again talk about packaging, this time “Packaging for Linux Distributions with Docker“. And I’ll also replay the now popular (!) “101 Docker training” I’ve also delivered in Toronto, this time trying to deliver it before the one Jérome Petazzoni from Docker is doing on Docker orchestration, in order to allow for a complete overview of the Docker ecosystem.

Of course I’ll enjoy attending many sessions on various topics, and will see if in EMEA as well Docker Orchestration is the new Eldorado this year (In Toronto many talks and booth were dedicated to that theme). At least my colleague Mike Bright will cover it well !

See you around !