Posts Tagged ‘perl’

Random thoughts after 3 days down under and 1 around Linux.conf.au

2014/01/07

Australia !
Back to Australia !
Shark Bay
Just after my arrival in Perth last Friday, I had a bit of time to walk in the city in order to have a view on it and feel the atmosphere.
View from the hotel
It’s an interesting mix of modern buildings and a bit less modern buildings !
Mix of buildings
Of course, Australia is a young country, with (from my point of view) mostly young people around (that’s where I see I’m getting old !). People are very nice, friendly and always ready to help with a smile. It’s my third time here, and it’s always been like that, that’s probably why I like coming back here.
Australian Steak
I had a dinner in what I found a strange place, the shed, where I had to show my ID before entering, which was pretty empty despite the size. Maybe because it was a bit early. The dinner was good, but I really felt alone !
That's where Perth performs savant music

It was then time to go back to the hotel for some mail, and the preparation of my Monday presentation.

Being in Australia during europe winter time is just great. As great that you can receive too much sun on your head ;-) Especially when you go to such a nice place as Fremantle !
Main Fremantle Street
I did that on Saturday (for once I had time around an event) in that city which is similar to a french mediteranean city like Cagnes sur mer e.g. (more from a climate perspective than an architectural one)
Hall Town
and so I passed the full day around, enjoying the view on the Indian Ocean (my first time),
Site seeing from Round House
and beach (nearly empty as very windy)
Fremantle beach
and the nice food you can eat on the harbour
View on the harbour
(remember that as french we put a lot of emphasis on this, and even more because it’s me ;-))
Lunch

One of the main building of Fremantle is their prison, which has been added to the World Heritage List of Unesco as a testimony of the role of convicts/prisoners sent to Australia to work there. One of the oldest building of the region, even if it’s just less than 200 years (old european reaction ;-))
Fremantle prison

It’ was also interesting to see that Fremantle uses Wikipedia to document their sites. They are adding QRCodes pointing to Wikipedia pages in order to make your visit more rich. That’s the first time I see that usage BTW.
Wikipedia QR code

Sunday was passed doing more preparation around project-builder.org release 0.12.3 and the presentation. I just took some time in the afternoon to go back in the city
Skyline view some strange trees in parks
Tree
and bells tower
Bells tower
before recognizing the itinerary to go to Linux.conf.au
LCA2014 Banner
hosted at the University of Western Australia
UWA
where we could register.
UWA main building
Despite not being at LCA for the first time, I attended Rusty Nelsson session on first timer LCA attendees, which was a nice way to start the event.First time session
After some more discussion at a nearby bar, it was time to finalize my presentation, and work on the demo. Took me up to 3AM, but was still not finished as I wanted !

On monday, LCA was starting.
Start of LCA
The first keynote was about Whistle-blowing (les lanceurs d’alerte for my french audience who like me would not happen to know what it is) and how their perception is changing in our post-Snowden era.
Dr Suelette Dreyfus
That was an interesting presentation of her finding through a survey they organized to gather data. Especially after having seen the keynote at LinuxCon EMEA around Living in a Surveillance State by Mikko Hypponen, which I highly recommend to watch,as it’s a very enlightning one.
Dr Suelette Dreyfus
However, during the Q&A session, I found a bit strange that the audience didn’t want to listen to the question from a doctor around how to determine what is good and what is bad, which was much more interesting that what they felt it could be. And I think he was right, as this is a real concern that should rather be dealt with, instead of just asking the person to shut up. The limit of what can be disclosed or not whether you are in a government or a private company wrt ethic aspects is really a difficult topic to solve and more debate should be left so more interesting input could be heard and answers brought to improve the quality of the debate.

And also it’s interesting that the spot is now only on the NSA, whereas we tend to forget that some months ago, we had the same questions around Facebook. Of course, people give willingly their info to Facebook while theydon’t to the NSA, and that’s a bad thing. However, it doesn’t rmove the other threat from companies like Facebook which should continue to be explained to all IT users and consumers.

There was a remark on the fact that US own 5 millions records of autralian citizens who have travelled to the US. Again, it’s easy to avoid that: do not travel to the US. If you want to go there, it seems normal to me they can ask for info beofre to protect themselves. Of course, the level of what is asked should be reasonable wrt the effect researched, and people have modification right son it and insurance it won’t be provided to other entities. But here I think they are in their own right to gather them. Of course, again nothing to do with what we’re learning regularly thanks to E. Snowden revelations with what the NSA is doing.

I passed the rest of the day mostly in the Continuous Integration miniconf which was also hosting my talk with the exception of a presentation around ansible and vagrant I wanted to follow in the sysadmin mini-conf. It turned out that this presentation wasn’t so interesting due to the lack of demo (no network). Too bad as I’m also testing ansible as time permits on my side and would have like to get insights.

So after that I took some time to finish my slidset for the pres of the afternoon and the demo I wanted to make. I think I’ll never been able to finish a presentation days in advance completely. Don’t get me wrong, I have plenty of material availblle and I can make any 3 days workshop on the fly just using what I already have. However, I like to tailor the presentation to the audience, the context, polish one or two stuff, remove slides and some new ones… And same with the demo. So even if I started early before coming, I just finished to publish project-builder.org 0.12.3 the morning (time for all packages to build), realized that some feature that I hadn’t tested enough were not working, so making just a last patch (c) to have my demo shiny !

So I missed some sessions in between and went back to the CI miniconf and really heard Anita Kuno’s talk on 3rd party testing with OpenStack (a good one on latest aspects of the project I wasn’t aware of – this one is moving soooo fast !).
Geology Building at UWA

And finally all went well (at least from my point of view of course, ask one of the around 40 people of the the audience for objective feedback !). Demo worked as expected (thanks for preparation ;-)) and as usual I din’t had enough time to cover all what I wanted to address. Will see now if my Continuous Packaging approach is supported more in Australia, and if project-builder.org gets adopted there to create more upstream packages for some projects. Those interested can find my presentation on SlideShare. The video is also available.
Perth Lighted building
End of day was dedicated to the CI BoF and a dinner with a lot of HP peers working on OpenStack on various aspects (always amazed to meet so many people involved in it as time passes that I wasn’t knowing before).

But that’s not the end ! More “awesome” presentations to see the days coming. And more pictures are available at https://plus.google.com/photos/112434061686721373729/albums/5966194867275136465

It starts as a happy new year indeed

2014/01/04

Well, would you really be surprised if i say that the begining of 2014 looks as promising (and as busy of course) as the end of 2013 was ?

While travelling to Linux.conf.au in Perth (28 hours door to door but could it be a better start for 2014 !) I had some time to look at Project-Builder.org, subject of my conference on Monday and also to MondoRescue.

For Project-Builder.org, I’ll publish a new version for LCA (0.12.3) which will fix rpmbootstrap for lastest Fedora (18-20), Mageia 3 and also other recent distros. Also as SVK is now broken on Mageia 3, I’m using git-svn to work on my laptop when remote, and thus pb supports it now. Some various fixes since last year release will also be done (better printing with VM/VE name for logs, improved ntp support, pbdistrogetparam command to easy shell interface…), but globally the software is pretty stable for my usage, so now it’s your turn to tell me what would be useful for you in it. I also need to make a newer presentation for Monday with a short demo.

For MondoRescue I’m not ready yet to publish a final version, so I have made for now a first beta for the 3.2.0 version. First this will be a new 3;x version as there are lots of changes in the code that may make it less reliable due to the adoption of dynamic memory allocation instead of static in a lot of functions. If I compare 3.0 and 3.2:

  • 3.0
    • total mr_asprintf usage : 646
    • total sprintf usage : 674
    • total strcpy usage : 560
    • total fgets usage : 79
    • total malloc_string usage : 367
    • total MAX_STR_LEN usage : 370
  • 3.2
    • total mr_asprintf usage : 1223
    • total sprintf usage : 166
    • total strcpy usage : 388
    • total fgets usage : 28
    • total malloc_string usage : 129
    • total MAX_STR_LEN usage : 221

Which means that a lot has been done in code changes between 3.0 and 3.2. But also a lot remains to do if we want to have no malloc_string nor MAX_STR_LEN at all. And also if I made mistakes in the mr_malloc/mr_asprintf/mr_free usage, the commands will crash. THat’s why I need your tests here ! I’ve made a full bakup restore cycle (which has revealed some issues that I fixed), but I’m far from being confident in the result.

Another consequence is that this version will probably have less fixes than previous ones as most of the time has been passed on code modernization.

And it will require new mindi and mindi-busybox versions. And these versions should finally work for systemd crazyness of removing /bin and /sbin (which broke MondoRescue for a too long time now). And you’ll need new perl modules in order to have it work correctly as I finally rewrote some bash mindi functions into perl to have what I consider better code.
Which means Fedora 18-20, Mageia 3 should work out of the box (at least my Mageia 3 now works correctly).

And expect again more pacthes to MondoRescue durnig the 28+ hours back home at the end of next week ;-)

So I wish to all readers a happy new year 2014 and hope it will start for you as it has for me !

Meeting at Linux.conf.au in Perth

2013/12/21

I’m again lucky to be able to fly in a plane for 20+ hours the first week of January to attend Linux.conf.au ! (And of course the 20+ hours of fly back ;-))

I’ll talk on Monday 06 January 2014 during the miniconf dedicated to Developer, Testing, Release and Continuous integration Automation of project-builder.org and the idea of Continuous packaging.

This has been possible only thanks to HP‘s support, its Open Source investment council, and the kindness of our VP & Deputy General Counsel, Cloud Computing and Open Source Eileen Evans who continues to support my travels at many Open Source conferences, and should be greatly thanked for that.

I’ll make a new version of the tool available to celebrate this, not that a lot has changed, but some fixes should be made available as packages, and it’s a long time since I published the last one. In fact, for me the tool is pretty stable and usable. I recently added git-svn support, as svk is broken with latest perl versions such as on my Mageia 3 :-( So learning slowly git, and still not liking it that much, but I have not a lot of other choices.

I still need to improve rpmbootstrap for the latest Fedora which changed their mirroring structure. Hopefully this will be done on time.

And as I’m on it, Merry Christmas and Happy New Year to you all readers !

Finally time to publish pb 0.12.2

2013/03/01

As you can see with the few messages I had time to post on this blog, the end of 2012 start of 2013 has been pretty busy, and I’m late in delivering the 2 projects I’m leading. So this week, I decided it was time to make a 0.12.2 version of pb, and make it available. Was asked by my colleagues of FOSSology, specially to add Fedora 18 support, so I also build my VMs to make packages on this distro.

BTW, a bit of ranting for Fedora once more: no perl by default :-(, and no ifconfig nor route command either which is breaking MondoRescue of course, and I guess tons of other software around. These guys don’t care about past, but don’t provide compatibility tools either !! So I’ll have to make new patches, just to do the same as what was done, but with another command again. Not to speak of systemd which I still have issues to deal with :-( That doesn’t make stuff go faster !

Anyway, the version is now out, no official bugs fixed, but a lot of small stuff here and there which were desrving a release. No time to test Fedora 17/18 VE yet, so you’ll have to do that yourself if you want. I also had a look at virsh usage in combination with pb, and it’s again not as easy as it could seem to be. Especially port redirection I’m easily using by launching qemu-kvm manually with the -redir or hostfwd option doesn’t seem to be possible with the user mode network through virsh (neither manager nor CLI). Will have to post on their ML to see how they do that, if they can !

And MondoRescue has been very late. I really need to publish a verion, but I still have some blocking bugs I really would like to get rid of: CTRL+ALT+DEL not working anymore during restore, some LVM issues on RHEL, some grub issues on SLES… Hopefully at the end of next week I’ll have made progresses.

Of course patches are much easy to integrate, but I receive more bug reports than them ;-)

And also back to preparing the HP internal TES event, Solutions Linux confs, submitting to LinuxCon, working on an FLOSS ITIL stack, learning more OpenStack, looking at Intel’s TXT…. so many things I’d like to do or learn and will never have time to ! Maybe at least I’ll talk about that another time.

About to publish project-builder.org 0.12.1

2012/05/22

When I looked at the date of the current project-builder.org current stable version, I had a shock ! It has been nearly a year since I published that version. I should not stay that long without providing updates :-( But you know what it is, you always think that you should still add that latest cool feature, and then that other, + fix this problem, … and you end up one year later with a great development version, but still no public stable one so others could benefit from your work.

So it’s time ! Especially as I have committed a lot of patches from an HP colleague, Eric Anderson, who is using the tool (and in its devel version !) to support packaging projects he is working on for HP. So I worked hard since the 8th of May to integrate what he has made available on github, since he got HP’s approval through the OSRB. And I have integrated I think more than 90% of his enhancements, which now will make 0.12.1 even more appealing than before.

But I’m not as a good guy as some think ;-) Some of the patches seemed strange to me, or could have an impact which I wasn’t completely sure of, so there is still some work Eric did, that has been left appart for the moment. Sorry for that, but I’m a slow guy, using this project on a daily basis, so I really need to be convinced before accepting a patch. But I was convinced by a lot ;-) I’ll now contact him back through the project mailing list, and see how we can work on these remaining points for 0.12.2 so he has a tool working much better for him. Among the nice stuff Eric has brought is clearly a much improved GIT support (read working !), and Debian/Ubuntu support as well (as he is using them natively). He also added support for stopping with the first error, or not as this is an option. Plus lots of improvements all around.

And he will also beneft from my own modifications, which hopefully improved the tool as well ! The last one, being with the configuration file management, now fully done in memory, instead of re-reading all conf file each time (which could also have some advantages, but speed improvement is what I was looking for here). In particular, in this version, the documentation has been improved a lot and should help more beginners. I have added a new getconf option to help following the value of configuration items (useful as pb uses lots of config files !), checkssh to verify what has been delivered on the repository, wrt what should have been, cleanssh to clean up the remote repository for test versions. A new -t option has been added to create local packages more quickly (when only the local distro will be used). Patches support has also been extended to .deb distributions. I have also introduced a new script, pbmkbm, which is aimed at creating customized boot media (project-builder.org make boot media), that will be the basis of an evolution of mindi in the MondoRescue project (doesn’t work as of now, and more on that later on). Some enhancement requests and bug fixes have also been done (#99, #585, #93, #522, #103, #105, new distro support for Fedora 15 and 16, Ubuntu 11.10, 12.04, OpenSuSE 11.4, 12.1 at least). #101 is the last on the list for 0.12.1 and as soon as it’s mostly done, I’ll publish it. Promised !

Stay tuned as usual ;-)

md2mb.pl improved to better support migration from Kmail to Thunderbird

2012/04/11

It’s just incredible how the Open Source world works. You could never believe it before being yourself part of it.

It started with a simple poor small script I wrote for myself to support the migration of my kids to Thunderbird from KMail. And I blogged about it, as I thought it could be interesting as I didn’t found myself a working tool to do it.

It turned out it was the most read article of this blog ! Grumph ! I thought people could be interested by MondoRecue or Continuous Packaging, my bad ;-)

Of course, when I have something to do, I prefer to write a script. Because old guy like me know perfectly you’ll have to do the stuff twice, and thus a script is useful !! And it was prooved again when I migrated my last kid with it again. And the script was improved at the same time. But I knew there were still some flaws, as one colleague found it as well, and had an issue with it (some mails not seen correctly even if imported in subfolders) !

So last month I received feedback from another user, who took that script and improved it greatly, fixing btw the above mentioned problem. And so that small script has now as many contributors as Mondorescue and twice as many as project-builder.org.
Edward Baudrez did a great job to improve the script, adding lots of CLI options capabilities to make it more versatile, fixing mail import in subfolders, error message handling, adding POD doc, …

I still have one or two improvements I’ll discuss with him to try making it even better, but that’s already quite an interesting piece of code !

So as it’s now even more useful than before, those interested by that migration should download and test it in their environment, and continue to send patches ;-) and make me more mazaed by the power of our community, for large as well as small stuff !

The project-builder.org Fosdem Vidéo is now available

2012/03/24

The presentation I made at Fosdem this year has been video taped by the great video team during that event. It’s now available online for you to replay (as well as on the project’s ftp server). If you want the associated slideset, just download it !

The latest development have been around fixing the bug on file names, improving test versions build. And we now have new requests on the devel mailing list and, even more interesting, patches proposals to improve pb work with Git and other bug fixes, that should make their way before I issue 0.12.1.

Let’s meet at Fosdem 2012 in Brussels

2012/01/30

I’ll attend Fosdem again this year next week-end in Brussels. I’ll deliver a talk on Project-Builder.org as a support for a Continuous Packaging cross Operating Systems development.

There are some news with the tool, and hopefully a new version, and some future evolution that I’d like to communicate. I also plan to present less slides, and have a more concrete demo to help people see the value of the approach.

While not presentting, I’ll probably be around near the Mageia booth or around my HP colleagues attending the event as well (Bdale Garbee, Martin Michlmayr, Hugo Roy). Don’t hesitate to come and chat !

Migrating from KMail to Thunderbird: The revenge

2012/01/08

After migrating 2 of my kids and my wife from Kmail to Thunderbird last year, I finally decided this week-end to finish the last migration for my first daughter on her Mageia distribution.

I previously made unsuccessful tries, as her environement was different, with many more subdirectories, and special chars, so it didn’t work with the previous version of the script.

Now with the revisions 1389 and 1390 of the md2mb.pl script, I have successfully migrated her environment, without any manual intervention.

Hopefully, seeing the number of times the previous post was looked at, it will be again useful (even more now that it works better:-)) for others. I even clarified the license in revision 1391 for you to use more easily.

Happy migration !

Third Day at OWF 2011

2011/10/28

Last but not least day at OWF this year. Yep, this is a saturday, but this was the code time for me, so probably the most interesting no ;-) ?

I started the day my making first 2 interviews for the WebTV of the event, one in french and another one in english explaining Open Source at HP. That’s what you get by sponsoring events ;-) So now, my kids can compare me with every other product seen on the TV !! By the way, it allowed me to understand that the professional video world is still unable to use FLOSS only technologies to perform such activities, and if you have spare time, and no defined project to work on, that would be a great addition to our community.

Then it was time to attend the sessions of the day and take some notes to share with you.

Lua and its ecosystem by François Perrad.

Lua is developed in Brazil using a closed development approach (just tar balls), even if it is Open Source since 1993, under the MIT license (which François underlined as important for its adoption by the embedded applications).
It’s a scripting language, fast, powerful and lightweight.

v5.1.4 is the stable version (from 2008)
v5.2 is now ongoing
(Some people still use v4 for its C API and the apps developped with it)

The language is concise but powerful: EBNF Grammar of 1 page and a Reference manual of 100 Pages (with C API, and libs included). It is 12.5 kLOCs and the binary size is 150 kB.
There are only 8 types and 21 keywords.
Lua has a table type like a hash, which allows to make everything.
It also has a userdata type to manipulate C structs/objects e.g.

Lua is portable (written only in C89 – but C++ compatible)
It exists for Linux, OSX, Windows, Android, iOS and Lego (eLUa). Other implementations exist: LuaJIT, LLVM-Lua, MochaLua, Jill (Mobile)

Lua is small but powerful (such as perl, python, ruby, subject of this conf) – It supports regex
François then showed examples of factorial and fibonacci programs written with it.
It is very easy to embed with C programs.

Lua modules are heterogeneous (no CPAN), so build, doc, test and packages are multiple.
Lua has Modules such as sockets, POSIX, XML, DB, wxLua, GTK, JSON, Corba, …

François gave then some application examples: textadept (text editor, 2kLOC C + 6 kLOC Lua), Wireshark (added scripting support with Lua), awesome (Window Manager), redis (key/value tool, server side scripting with Lua)
Lua has too few sponsors: Adobe (Lightroom), SocialMediaPress, CSTUG, OCé.

François complained that there is not a lot of return to the community in term of code from these sponsors. But after all, my take on this is that this is certainly due to the choice of license (permissive) and of development model which doesn’t invite or force contributions to be brought back. One of the reason I chose the GPLv2 for my projects. Clearly it seems interesting if you need to add scripting support to a software. now, IMHO not done to develop applications directly.

Colin Charles (Monty Program AB)
Chief Evangelist on MariaDB (he left when Sun bought MySQL)

Charles gave 3 conferences during the day to paint n state of the Art of MySQL and its ecosysem.
He started by presenting the MySQL Diaspora

Charles started by some MySQL history.
The MysQL release policy is: alpha, beta and GA. 5.0 was released in 2005.
InnoDB was bought by Oracle in 2005, which created problems. The Falcon engine, which was targetted for 6.0, was the solution but it never saw the light.
5.1 was released in 2008. Lots of bugs were present due to an early release and lack of tests. 5.2 was canceled (It should have been open to community contribs)
5.4 was the next release in the roadmap in 2008.
In 2009, Sun was bought by Oracle. As a consequence 5.4 & 6.0 were canceled.
5.5 was realeased in 2010.
5.6 is available at labs.mysql.com (2011).

InnoDB + memcached in front of it, is the NoSQL approach taken (even if Oracle doesn’t believe in NoSQL, surprise, surprise !!)
Half of the downloads at mysql.com were for Windows. But most people on Linux, OSX, BSD use the packaged MySQL from their OS which thus reduces the number of downloads for these platforms and gives a wrong impression of the usage. Secondlife e.g. used MySQL out of Debian.
However, distributions tend to include only a small subset of MySQL features. There are only 5 engines compiled in Fedora/RHEL !

Charles underlined that the real uniqueness of MySQL is its ability to use multiple engines very easily (more on that in a dedicated conference below).
He detailed the commercial ecosystem around MySQL for add-ons (InfoBright, TokuDB, ScaleDB, Nitro, SolidDB, RitmarkFS) as well as HardWare vendors too (Kickfire, Virident, Schooner – going away with Oracle) as well as 3rd party engines (that Oracle doesn’t like either)

Charles then gave definition of terms: Fork == do not merge back (Drizzle e.g.) – Branch: rebase all the time (MariaDB, Percona e.g.). He then expained the different MySQL children.

  • Percona started from 5.0 (it took Google/Facebook code on top of MySQL). So Percona is MySQL with InnoDB enhancements (coming from FaceBook/Google) and minimal server changes. They have 1000+ customers in the USA. They provide more fine grained diagnostics. (Google uses MySQL/Percona for Adwords). It provides FlashCache which makes it a great offering for SSD support. They also provide the Percona Toolikit (lots of perl scripts, it is the former maatkit UDFs) to make admin life easier. Their business model is mainly around consulting, and some support.
  • Drizzle is based on MySQL 6.0 whiwh was never released. It took them 3 years to release the GA, just recently, aimed to be the RDBMs for OpenStack. However, due to the time it took, OpenSTack now uses SQL Alchemy which supports multiple RDBMs. Drizzle is Unix only – No Windows. Drizzle has a Micro-kernel architecture so everything is a plugin (including optimizer, query caches, …). Fully written in C++ now (no C anymore), it targets Cloud. It uses a lot of other FLOSS tools (including Google Protocol Buffer). Patches from MariaDB are difficult to integrate as it’s a fork.
  • MariaDB started from 5.1, and is now with 5.5/5.6 also including Percona patches and abandonned patches. MariaDB is Percona and more storage engines and lots of improvements (including µs precision, extended stats for slow query log, threads pool support (eBay), pluggable authentication, virtual columns, NoSQL, subqueries (finally in 5.3 !!), GUI for Windows).
  • Other branches are OurDelta (pet project – died), MepSQL (just Facebook patches, died), XAMPP (Windows)

For NOSQL in MySQL, use MariaDB ! Its HandlerSocket provides 700000 queries/s vs 400000 queries/s for standard MySQL. It also provides Dynamic Columns, memcached interface to InnoDB and NDB (only in the labs of MySQL, MariaDB is waiting for publication to include it – droped in 5.1, maybe back in 5.6/5.7)

MySQL cluster is an external product, which will probably be integrated in main tree with version 5.6/5.7.

From a support perspective, many companies can deliver it: Oracle (MySqL ent), Percona (MySQL, Percona, MariaDB, Drizzle), SkySQL (MySQL, MariaDB, Drizzle, distros), and many others.

Charles announced that pap.fr just issued a press release about their migration to MariaDB.

He then underlined the difficulty to work with Oracle: There is no public roadmap for MySQL. MySQL is now OpenCore: threads (connection) pool is closed source (original patches from eBay !) with only API for community dev. PAM (AD plugin) code came from MariaDB and is now also closed source. InnoDB Hot Backup however has always been closed source. But xtrabackup is FLOSS. InnoDB has never been Open Source, and they are rebasing from what is published monthly.
These problems comes from the contribution agreement that Oracle forces MySQL contributors to adopt to integrate patches upstream, which gives them the possibility to close the code instead of integrating it inside the public version. Oracle is once again making foes here.

From a performance perspective, XtraDB in 5.5 > InnoDB in 5.5 > XtraDB 5.1.
Connectors remains 100% the same whatever the backend.

Scaling is the issue. Hopefully, working with HP on this we can help Monty programs make progresses on this side.
To a question I asked around Monty Programs, Charles explained that Monty Program is only developing around MySQL. Their business model is to sell 3rd level support and dedicated consulting around development. They have no interface with ordinary customers.

The ecosystem is now in fact split between multiple actors, compared to when MySQL AB was doing everything: Monty Program as the “Lab”, SkySQL for the support and Percona for the consulting, even if the last two overlap a bit.

I was lucky to lunch with Charles, so we could discuss into more details around the community, and some of the points upper were not presented during the formal presentation but are the result of my understanding during our talk. All mistakes are of course mine ;-)

During the next session, I had a discussion with Charles Schulz, administrator of the LibreOffice.org foundation to get some news of this community, and they have lots of plans, some announces that will happen in October, but that I can’t reveal for now. Stay tuned however as it’ll be great !!

Then I attended the second session from Charles Colin around Storage Engines of MySQL/MariaDB

The main engines are: MyISAM, InnoDB, NDB (aka Cluster), Falcon (dead), Archive, Federated(X), Merge, Memory, and the one from Partners, Community, Customers…
The Engines list also includes AWS, OpenLDAP, CSV, Blackhole, Q4M (queuing mecanism), …

Charles then explained the default choices:
MySQL 5.5 => InnoDB by default, MyISAM for temp files
MariaDB 5.5 => XtraDB by default, Aria for temp files (more secure, as crash safe)
Percona 5.5 => XtraDB by default, MyISAM for temp files
Drizzle => HailDB (the only engine), MyISAM for temp files

The value proposition of MySQL/MariaDB around the various engines is unique. It brings flexibilty very easily, from ACID compliant InnoDB, to memory engine for performance, to archive which compresses data up to 80%
There are of course differences between engines with regards to storage (side note advise; use XFS), indexes, memory usage, transactions support (ACID), …
To see what engines are supported in your version, use show engines (12 present in MySQL, 10 in MariaDB)
Engines are plugins handled as loadable modules after 5.1.
There are some commercial engines: InfoBright (Data warehouse), TokuDB (cloud)
To use a specific engine with a table use: create/alter table … engine=XtraDB
Distributions tend to remove engines in their delivery (half only in Fedora/RHEL).

Indexes use all type of algorithms available: B+trees (InnoDB), Red-black trees (memory), R-trees (MyISAM) + hash indexes (memory, NDB, InnoDB).
For MySQL benchmarks, Charles recommended to use mysqlslap, to which I’d also tend to add the excellent HammerOra, which will allow you to make comparison with other RDBMs very easily !

Charles then explained the specifics of some engines:

  • MyISAM: excellent insert perf, small footprint, full text support. No transactions, no foreign key. Usage: log, audit, Datawarehouse. Was preferred earlier. Not usable with DRBD. Tuning hint: Use key_cache_segments = 1 in MariaDB.
  • InnoDB: for OLTP, raw disk support (useful for Windows). Tuning is also required. Use innodb_file_per_table good when locking, innodb_buffer_pool_size to memory*0.8 if dedicated. show engine innodb status gives more stats on MariaDB. InnoDB supports row & statement replication. Hint: Use a primary key (if not, it will be done for you)
  • Archive: good for auditing (can’t delete in it, only SELECT and INSERT), uses libz
  • Federated was disabled in MySQL but FederatedX has replaced it in MariaDB.
  • FederatedX: Used by Cisco to store centrally from routers. Good for synchronous replication.
  • Memory: loosing popularity in favour of memcache (BTW there is also a memcache engine !) Used for lookup, session, temp or calculation tables. Of course, when server goes down, rows are lost.
  • Aria: MariaDB only. Planned to be competing with InnoDB and was concurrent to Falcon (now dead). Crash safe MyISAM, may become ACID.
  • PBXT: ACID. Transactional. Good for SSD storage. Popular in Germany. Supports efficient BLOB. Should be more known.

To install a new engine just issue: INSTALL PLUGIN myengine SONAME ‘engine.so’

Charles reminded that a mix and match of engines is frequent.
He underlined that backup is engine dependant. LVM snapshots are a way to solve this.
Monitoring is also engine dependant.
Wordpress uses MyISAm, Wikipedia/mediawiki prefers InnoDB except search, SugarCRM uses MySQL.

Was time for me to take the mic ! and talk about
Continuous packaging with Project-Builder.org by Bruno Cornec (HP)

Brilliant presentation as usual ;-) Find it online and hopefully at OSDC.fr where the Vidéo should appear soon.

Open Build Service by Vincent Untz (SuSE)

I stayed in the room to listen to Vincent, who insisted on the new name of the former OpenSuSE Build system which is now the Open Build Service (Vincent, I’ve now patched my presentation to be aligned ;-))

Even if goals are similar (building more easily packages and supporting a continuous packaging approach, there are still some differences I think between the tools. First, of course, language used to develop them (perl for Project-builder.org and Ruby on Rails for obs), but also on pb side the willingness to share metadata across packages, in a VCS and providing a macro system to support it, the support of VMs *and* Virtual Environment* and now *Remote Machines*, with tools such as rpmbootstrap to help users managing them, the standalone aspect of the tool and it’s ease of deployment (just 3 perl packages + 2/3 deps max – fewer LOC). Also pb supports gentoo, and Solaris, and is very easy to extend as everything is now managed through configuration files.

Advantages on obs side are its great Web based interface, its reporting mecanisms, and its capability to rebuild projects based on modifications of other projects. obs announces a very large user base, and packages built, even if when you divide by the number of distro tuples it’s less of course. pb just supports 5 different projects so has a much less reduced installed base.

Vincent did a very good presentation, as usual, showing also how it had a positive impact on the OpenSuSE project by having all the build infrastructure rebuilt to be Open Source (GPL) in order to support the project. Another big plus of their infrastructure is their BaaS approach (Build system as a Service), whereas with pb you need to create it, even if it’s easy. However, Vincent underlined that OBS as BaaS is only accessible for Open SOurce projects. If you’re building other software types, then we are at parity as you need to deploy the tool in-house.

As I underlined at the end of my talk before welcoming Vincent, whatever the tool you use, what is the most important, is to promote continuous packaging, as a good practice of today for projects (Open Source or not). Everybody’s new mantra should become “package early, package always” !

Charles then made his last presentation of the day (busy guy !) on MariaDB.

He reminded that it is a branch of MySQL, community developped, feature enhanced and backward compatible.
Monty Program is the main sponsor of MariaDB (58% of patches come from Monty Program, 42% from commmunity members having commit access)

MySQL is now an Open Source/Core product not a project.
MariaDB is an Open Source project, with the goal to be 100% compatible and a drop-in replacement of MySQL.
It is stable (production ready) and GPLv2. XtraDB (vs InnoDB) is enabled by default.
Charles then covered the various versions published:

  • 5.1: Spend lot of time on build system (Buildbot + VMs ;-) for 5.1. No compiler warnings (contributed back). Table Elimination (aligned with big RDBMs) improves perf. Pool of threads (or connection pool, good for short running queries).
  • 5.2: Pluggable authentication. Easy to write in 1/2 days AD auth/PAM auth as Oracle published recently. Virtual columns. Includes ShpinxSE (Full Text Search Engine)
  • 5.3: NoSQL (HandlerSocket – direct access to InnoDB in perl DeNA, Dynamic columns, Group Commit, Replication, mytop, optimizer, optimized subqueries). That code may not be given back to Oracle due to Oracle recent announcement.

Everything done in the open, ML, bugs, IRC, code, worklog, KB (1400 articles in 1 year, cc-by-sa) …

Future: GIS. Merging takes more and more time. Future may be to fork to solve the Open COre MySQL approach, which would be IMHO good news for the FLOSS world, as it was sith LibreOffice e.g.

Blocks in perl6 by Christian Aperghis-Tramoni

A real course on how perl6 manages some blocks (functions where particularly detailed). Not a lot of notes, here as everything is online, and this was mostly code explanations, so I prefered to follow, rather than taking useless notes that nobody would read anyway ! What I mostly retained is that perl 6 brings some huge differences with regards to perl 5, certainly in the good direction, but will need an adaptation from old perlers such as myself, not following closely perl 6 development as Christian, and also that it’s not yet ready for usage, as lots of features are not there yet, and no optimisation is available, so from a production perspective, it’s unusable now. Perl 5 has for sure a very long life in front of him :-)

All in all, OWF 2011 was again a great event. However, I would prefer for next year that they concentrate mostly on the Think part, which is quite unique to this conference and makes all its value, whereas code and experiment are already well represented at RMLL or Fosdem. See you there next year hopefully !

Note: Some of my photos related to this event are now available at https://picasaweb.google.com/112434061686721373729/OWF2011


Follow

Get every new post delivered to your Inbox.

Join 102 other followers