Posts Tagged ‘perl’

Migrating from KMail to Thunderbird: The revenge

2012/01/08

After migrating 2 of my kids and my wife from Kmail to Thunderbird last year, I finally decided this week-end to finish the last migration for my first daughter on her Mageia distribution.

I previously made unsuccessful tries, as her environement was different, with many more subdirectories, and special chars, so it didn’t work with the previous version of the script.

Now with the revisions 1389 and 1390 of the md2mb.pl script, I have successfully migrated her environment, without any manual intervention.

Hopefully, seeing the number of times the previous post was looked at, it will be again useful (even more now that it works better:-)) for others. I even clarified the license in revision 1391 for you to use more easily.

Happy migration !

Third Day at OWF 2011

2011/10/28

Last but not least day at OWF this year. Yep, this is a saturday, but this was the code time for me, so probably the most interesting no 😉 ?

I started the day my making first 2 interviews for the WebTV of the event, one in french and another one in english explaining Open Source at HP. That’s what you get by sponsoring events 😉 So now, my kids can compare me with every other product seen on the TV !! By the way, it allowed me to understand that the professional video world is still unable to use FLOSS only technologies to perform such activities, and if you have spare time, and no defined project to work on, that would be a great addition to our community.

Then it was time to attend the sessions of the day and take some notes to share with you.

Lua and its ecosystem by François Perrad.

Lua is developed in Brazil using a closed development approach (just tar balls), even if it is Open Source since 1993, under the MIT license (which François underlined as important for its adoption by the embedded applications).
It’s a scripting language, fast, powerful and lightweight.

v5.1.4 is the stable version (from 2008)
v5.2 is now ongoing
(Some people still use v4 for its C API and the apps developped with it)

The language is concise but powerful: EBNF Grammar of 1 page and a Reference manual of 100 Pages (with C API, and libs included). It is 12.5 kLOCs and the binary size is 150 kB.
There are only 8 types and 21 keywords.
Lua has a table type like a hash, which allows to make everything.
It also has a userdata type to manipulate C structs/objects e.g.

Lua is portable (written only in C89 – but C++ compatible)
It exists for Linux, OSX, Windows, Android, iOS and Lego (eLUa). Other implementations exist: LuaJIT, LLVM-Lua, MochaLua, Jill (Mobile)

Lua is small but powerful (such as perl, python, ruby, subject of this conf) – It supports regex
François then showed examples of factorial and fibonacci programs written with it.
It is very easy to embed with C programs.

Lua modules are heterogeneous (no CPAN), so build, doc, test and packages are multiple.
Lua has Modules such as sockets, POSIX, XML, DB, wxLua, GTK, JSON, Corba, …

François gave then some application examples: textadept (text editor, 2kLOC C + 6 kLOC Lua), Wireshark (added scripting support with Lua), awesome (Window Manager), redis (key/value tool, server side scripting with Lua)
Lua has too few sponsors: Adobe (Lightroom), SocialMediaPress, CSTUG, OCé.

François complained that there is not a lot of return to the community in term of code from these sponsors. But after all, my take on this is that this is certainly due to the choice of license (permissive) and of development model which doesn’t invite or force contributions to be brought back. One of the reason I chose the GPLv2 for my projects. Clearly it seems interesting if you need to add scripting support to a software. now, IMHO not done to develop applications directly.

Colin Charles (Monty Program AB)
Chief Evangelist on MariaDB (he left when Sun bought MySQL)

Charles gave 3 conferences during the day to paint n state of the Art of MySQL and its ecosysem.
He started by presenting the MySQL Diaspora

Charles started by some MySQL history.
The MysQL release policy is: alpha, beta and GA. 5.0 was released in 2005.
InnoDB was bought by Oracle in 2005, which created problems. The Falcon engine, which was targetted for 6.0, was the solution but it never saw the light.
5.1 was released in 2008. Lots of bugs were present due to an early release and lack of tests. 5.2 was canceled (It should have been open to community contribs)
5.4 was the next release in the roadmap in 2008.
In 2009, Sun was bought by Oracle. As a consequence 5.4 & 6.0 were canceled.
5.5 was realeased in 2010.
5.6 is available at labs.mysql.com (2011).

InnoDB + memcached in front of it, is the NoSQL approach taken (even if Oracle doesn’t believe in NoSQL, surprise, surprise !!)
Half of the downloads at mysql.com were for Windows. But most people on Linux, OSX, BSD use the packaged MySQL from their OS which thus reduces the number of downloads for these platforms and gives a wrong impression of the usage. Secondlife e.g. used MySQL out of Debian.
However, distributions tend to include only a small subset of MySQL features. There are only 5 engines compiled in Fedora/RHEL !

Charles underlined that the real uniqueness of MySQL is its ability to use multiple engines very easily (more on that in a dedicated conference below).
He detailed the commercial ecosystem around MySQL for add-ons (InfoBright, TokuDB, ScaleDB, Nitro, SolidDB, RitmarkFS) as well as HardWare vendors too (Kickfire, Virident, Schooner – going away with Oracle) as well as 3rd party engines (that Oracle doesn’t like either)

Charles then gave definition of terms: Fork == do not merge back (Drizzle e.g.) – Branch: rebase all the time (MariaDB, Percona e.g.). He then expained the different MySQL children.

  • Percona started from 5.0 (it took Google/Facebook code on top of MySQL). So Percona is MySQL with InnoDB enhancements (coming from FaceBook/Google) and minimal server changes. They have 1000+ customers in the USA. They provide more fine grained diagnostics. (Google uses MySQL/Percona for Adwords). It provides FlashCache which makes it a great offering for SSD support. They also provide the Percona Toolikit (lots of perl scripts, it is the former maatkit UDFs) to make admin life easier. Their business model is mainly around consulting, and some support.
  • Drizzle is based on MySQL 6.0 whiwh was never released. It took them 3 years to release the GA, just recently, aimed to be the RDBMs for OpenStack. However, due to the time it took, OpenSTack now uses SQL Alchemy which supports multiple RDBMs. Drizzle is Unix only – No Windows. Drizzle has a Micro-kernel architecture so everything is a plugin (including optimizer, query caches, …). Fully written in C++ now (no C anymore), it targets Cloud. It uses a lot of other FLOSS tools (including Google Protocol Buffer). Patches from MariaDB are difficult to integrate as it’s a fork.
  • MariaDB started from 5.1, and is now with 5.5/5.6 also including Percona patches and abandonned patches. MariaDB is Percona and more storage engines and lots of improvements (including µs precision, extended stats for slow query log, threads pool support (eBay), pluggable authentication, virtual columns, NoSQL, subqueries (finally in 5.3 !!), GUI for Windows).
  • Other branches are OurDelta (pet project – died), MepSQL (just Facebook patches, died), XAMPP (Windows)

For NOSQL in MySQL, use MariaDB ! Its HandlerSocket provides 700000 queries/s vs 400000 queries/s for standard MySQL. It also provides Dynamic Columns, memcached interface to InnoDB and NDB (only in the labs of MySQL, MariaDB is waiting for publication to include it – droped in 5.1, maybe back in 5.6/5.7)

MySQL cluster is an external product, which will probably be integrated in main tree with version 5.6/5.7.

From a support perspective, many companies can deliver it: Oracle (MySqL ent), Percona (MySQL, Percona, MariaDB, Drizzle), SkySQL (MySQL, MariaDB, Drizzle, distros), and many others.

Charles announced that pap.fr just issued a press release about their migration to MariaDB.

He then underlined the difficulty to work with Oracle: There is no public roadmap for MySQL. MySQL is now OpenCore: threads (connection) pool is closed source (original patches from eBay !) with only API for community dev. PAM (AD plugin) code came from MariaDB and is now also closed source. InnoDB Hot Backup however has always been closed source. But xtrabackup is FLOSS. InnoDB has never been Open Source, and they are rebasing from what is published monthly.
These problems comes from the contribution agreement that Oracle forces MySQL contributors to adopt to integrate patches upstream, which gives them the possibility to close the code instead of integrating it inside the public version. Oracle is once again making foes here.

From a performance perspective, XtraDB in 5.5 > InnoDB in 5.5 > XtraDB 5.1.
Connectors remains 100% the same whatever the backend.

Scaling is the issue. Hopefully, working with HP on this we can help Monty programs make progresses on this side.
To a question I asked around Monty Programs, Charles explained that Monty Program is only developing around MySQL. Their business model is to sell 3rd level support and dedicated consulting around development. They have no interface with ordinary customers.

The ecosystem is now in fact split between multiple actors, compared to when MySQL AB was doing everything: Monty Program as the “Lab”, SkySQL for the support and Percona for the consulting, even if the last two overlap a bit.

I was lucky to lunch with Charles, so we could discuss into more details around the community, and some of the points upper were not presented during the formal presentation but are the result of my understanding during our talk. All mistakes are of course mine 😉

During the next session, I had a discussion with Charles Schulz, administrator of the LibreOffice.org foundation to get some news of this community, and they have lots of plans, some announces that will happen in October, but that I can’t reveal for now. Stay tuned however as it’ll be great !!

Then I attended the second session from Charles Colin around Storage Engines of MySQL/MariaDB

The main engines are: MyISAM, InnoDB, NDB (aka Cluster), Falcon (dead), Archive, Federated(X), Merge, Memory, and the one from Partners, Community, Customers…
The Engines list also includes AWS, OpenLDAP, CSV, Blackhole, Q4M (queuing mecanism), …

Charles then explained the default choices:
MySQL 5.5 => InnoDB by default, MyISAM for temp files
MariaDB 5.5 => XtraDB by default, Aria for temp files (more secure, as crash safe)
Percona 5.5 => XtraDB by default, MyISAM for temp files
Drizzle => HailDB (the only engine), MyISAM for temp files

The value proposition of MySQL/MariaDB around the various engines is unique. It brings flexibilty very easily, from ACID compliant InnoDB, to memory engine for performance, to archive which compresses data up to 80%
There are of course differences between engines with regards to storage (side note advise; use XFS), indexes, memory usage, transactions support (ACID), …
To see what engines are supported in your version, use show engines (12 present in MySQL, 10 in MariaDB)
Engines are plugins handled as loadable modules after 5.1.
There are some commercial engines: InfoBright (Data warehouse), TokuDB (cloud)
To use a specific engine with a table use: create/alter table … engine=XtraDB
Distributions tend to remove engines in their delivery (half only in Fedora/RHEL).

Indexes use all type of algorithms available: B+trees (InnoDB), Red-black trees (memory), R-trees (MyISAM) + hash indexes (memory, NDB, InnoDB).
For MySQL benchmarks, Charles recommended to use mysqlslap, to which I’d also tend to add the excellent HammerOra, which will allow you to make comparison with other RDBMs very easily !

Charles then explained the specifics of some engines:

  • MyISAM: excellent insert perf, small footprint, full text support. No transactions, no foreign key. Usage: log, audit, Datawarehouse. Was preferred earlier. Not usable with DRBD. Tuning hint: Use key_cache_segments = 1 in MariaDB.
  • InnoDB: for OLTP, raw disk support (useful for Windows). Tuning is also required. Use innodb_file_per_table good when locking, innodb_buffer_pool_size to memory*0.8 if dedicated. show engine innodb status gives more stats on MariaDB. InnoDB supports row & statement replication. Hint: Use a primary key (if not, it will be done for you)
  • Archive: good for auditing (can’t delete in it, only SELECT and INSERT), uses libz
  • Federated was disabled in MySQL but FederatedX has replaced it in MariaDB.
  • FederatedX: Used by Cisco to store centrally from routers. Good for synchronous replication.
  • Memory: loosing popularity in favour of memcache (BTW there is also a memcache engine !) Used for lookup, session, temp or calculation tables. Of course, when server goes down, rows are lost.
  • Aria: MariaDB only. Planned to be competing with InnoDB and was concurrent to Falcon (now dead). Crash safe MyISAM, may become ACID.
  • PBXT: ACID. Transactional. Good for SSD storage. Popular in Germany. Supports efficient BLOB. Should be more known.

To install a new engine just issue: INSTALL PLUGIN myengine SONAME ‘engine.so’

Charles reminded that a mix and match of engines is frequent.
He underlined that backup is engine dependant. LVM snapshots are a way to solve this.
Monitoring is also engine dependant.
Wordpress uses MyISAm, Wikipedia/mediawiki prefers InnoDB except search, SugarCRM uses MySQL.

Was time for me to take the mic ! and talk about
Continuous packaging with Project-Builder.org by Bruno Cornec (HP)

Brilliant presentation as usual 😉 Find it online and hopefully at OSDC.fr where the Vidéo should appear soon.

Open Build Service by Vincent Untz (SuSE)

I stayed in the room to listen to Vincent, who insisted on the new name of the former OpenSuSE Build system which is now the Open Build Service (Vincent, I’ve now patched my presentation to be aligned ;-))

Even if goals are similar (building more easily packages and supporting a continuous packaging approach, there are still some differences I think between the tools. First, of course, language used to develop them (perl for Project-builder.org and Ruby on Rails for obs), but also on pb side the willingness to share metadata across packages, in a VCS and providing a macro system to support it, the support of VMs *and* Virtual Environment* and now *Remote Machines*, with tools such as rpmbootstrap to help users managing them, the standalone aspect of the tool and it’s ease of deployment (just 3 perl packages + 2/3 deps max – fewer LOC). Also pb supports gentoo, and Solaris, and is very easy to extend as everything is now managed through configuration files.

Advantages on obs side are its great Web based interface, its reporting mecanisms, and its capability to rebuild projects based on modifications of other projects. obs announces a very large user base, and packages built, even if when you divide by the number of distro tuples it’s less of course. pb just supports 5 different projects so has a much less reduced installed base.

Vincent did a very good presentation, as usual, showing also how it had a positive impact on the OpenSuSE project by having all the build infrastructure rebuilt to be Open Source (GPL) in order to support the project. Another big plus of their infrastructure is their BaaS approach (Build system as a Service), whereas with pb you need to create it, even if it’s easy. However, Vincent underlined that OBS as BaaS is only accessible for Open SOurce projects. If you’re building other software types, then we are at parity as you need to deploy the tool in-house.

As I underlined at the end of my talk before welcoming Vincent, whatever the tool you use, what is the most important, is to promote continuous packaging, as a good practice of today for projects (Open Source or not). Everybody’s new mantra should become “package early, package always” !

Charles then made his last presentation of the day (busy guy !) on MariaDB.

He reminded that it is a branch of MySQL, community developped, feature enhanced and backward compatible.
Monty Program is the main sponsor of MariaDB (58% of patches come from Monty Program, 42% from commmunity members having commit access)

MySQL is now an Open Source/Core product not a project.
MariaDB is an Open Source project, with the goal to be 100% compatible and a drop-in replacement of MySQL.
It is stable (production ready) and GPLv2. XtraDB (vs InnoDB) is enabled by default.
Charles then covered the various versions published:

  • 5.1: Spend lot of time on build system (Buildbot + VMs 😉 for 5.1. No compiler warnings (contributed back). Table Elimination (aligned with big RDBMs) improves perf. Pool of threads (or connection pool, good for short running queries).
  • 5.2: Pluggable authentication. Easy to write in 1/2 days AD auth/PAM auth as Oracle published recently. Virtual columns. Includes ShpinxSE (Full Text Search Engine)
  • 5.3: NoSQL (HandlerSocket – direct access to InnoDB in perl DeNA, Dynamic columns, Group Commit, Replication, mytop, optimizer, optimized subqueries). That code may not be given back to Oracle due to Oracle recent announcement.

Everything done in the open, ML, bugs, IRC, code, worklog, KB (1400 articles in 1 year, cc-by-sa) …

Future: GIS. Merging takes more and more time. Future may be to fork to solve the Open COre MySQL approach, which would be IMHO good news for the FLOSS world, as it was sith LibreOffice e.g.

Blocks in perl6 by Christian Aperghis-Tramoni

A real course on how perl6 manages some blocks (functions where particularly detailed). Not a lot of notes, here as everything is online, and this was mostly code explanations, so I prefered to follow, rather than taking useless notes that nobody would read anyway ! What I mostly retained is that perl 6 brings some huge differences with regards to perl 5, certainly in the good direction, but will need an adaptation from old perlers such as myself, not following closely perl 6 development as Christian, and also that it’s not yet ready for usage, as lots of features are not there yet, and no optimisation is available, so from a production perspective, it’s unusable now. Perl 5 has for sure a very long life in front of him 🙂

All in all, OWF 2011 was again a great event. However, I would prefer for next year that they concentrate mostly on the Think part, which is quite unique to this conference and makes all its value, whereas code and experiment are already well represented at RMLL or Fosdem. See you there next year hopefully !

Note: Some of my photos related to this event are now available at https://picasaweb.google.com/112434061686721373729/OWF2011

Meeting during LinuxCon in Prague or OWF in Paris

2011/09/05

I’m happy to have been informed that my proposal of presentation around FOSSology for LinuxCon 2011 in Prague has been accepted (too bad the others on MondoRescue or Project-Builder.org weren’t. Hopefully a next time).

However, for Project-Builder.org you can attend the presentation during the upcoming Open World Forum in Paris and discuss with me about everything Open Source and Linux and HP !

So some way to meet across Europe soon with you 🙂

Fosdem 2011 Report – Day 2

2011/02/15

Second day at Fosdem. This year, I decided to go and visit the perl community, as I’m more and more coding in perl, and would like to learn more about news, additional modules, and meet more perl hackers !

Gabor SzaboUsing Perl6 today
No real 6.0 announce. Already published and improving. Gabir gave lots of info on perl5/perl6 differences:

  • hash element prepended with % (including pointers)
  • array element prepended with @ (including pointers)
  • Expressions in {} are interpreted and executed
  • Variable may be typed
  • Chaining conditions 23 <= $age <= 42 (avoids and)
  • Junction if $age == 2|3|4 (avoids or)
  • Easier access to array elements (including in pairs, …) with missing parts managed – matrix type of computation
  • New ‘Z’ operator which allows array combination
  • lazyness allow infinity to exist: doesn’t generate the full list anymore from scratch. my @x = 1..Inf exists !
  • functions parameters number are checked, type as well. So we can now pass multiple arrays in functions without attribution issue. It also manages optional params.
  • Types can be defined by the user. Constraints can be put on params.
  • Operators can also be combined (Z and ~ – concat) on arrays e.g. (Notion of meta operator, hyper operator, user created operator)
  • Perl 6 manages classes methods (public, private)
  • Regex also have evolved a lot – Grammars can also be defined (based on regex) and inherited as well.

One quote I really liked: “Perl6 is fully buzzword compatible”.
A very intresting presentation on perl’s future, well present in fact, even if I’ve not tested it up to now. Looks promising but a huge change.

Damien Krotkine (Curses::Toolkit)

Damien advertized the French perl event (French Perl Worskhop 2011) in Paris, Perl Dancer community, his book, Perl Moderne. As it was also warmly recommended by Dominique Dumont (author of Config::Model), I bought it, and even had the author’s signature on it !! I started it in the train back from Brussels, and indeed it’s a good one, focussing on specific topics, (so not a bible), but very nice to read, and informative. Of course, as you guessed with the name, it’s in french 😉

Damien then talked briefly about Curses::Toolkit, as all these advertizing took a bit too much time IMO. He covered:

  • Curses::Toolkit curses binding for perl inspired by GTK
  • Why Curses::Toolki ? Existing Curses is too low level, Curses:UI buggy and inflexible; Curses::Widget
  • Curses::Toolkit real toolkit with widgets and events, OO, kbd, mouse, timer events driven, using POE.
  • Curses::Toolkit uses themes and is very easy to customize

Damien then concluded by making an impressive demo of modern caracter based interface

Mark Overmeer (Perl Data Structures)

Excellent talk by Marc who covered in 1 hour as many topics as he had on his list, without exhausting it !

He underlined:

  • the importance of scalars in perl. Used for everything. Consumes 28 bytes each (because they can store multiple values (dual var/$!)
  • False values in perl: undef, 0, 0.0, “”, “0” (the most dangerous). die "no fn" unless $fn is a mistake as filename “0” is false even if it exists. Recommends using length instead. $x = $temperature || 20 is also a mistake if temperature is 0. Recent perl provides operator // for that for undefined
  • An array is a copy of a list of params (he insisted on difference between list and array)
  • Context is the specificity of perl. An expression can not be understood without its context in perl. There is a void context. An array in scalar context provides its length. There is list context (@a = 3; works). Good way to loop on arrays foreach my $x (@a) { print $x; }. $x[3] is a promise of a scalar.
  • Array affectatation: @x[1, 3, -1] = 6..8. Also valid is (getpwnal $x)[3,-1] (gives uid/shell). getpwnam in list context gives the 10 elements. In scalar gives the uid (you only know by reading the man page).
  • In hashes, you can delete elements with delete $a{b} . $a{b} = undef is different. if exists $a{b} gives true with undef. if defined $a{b} gives false.
  • Array knows the order, hash doesn’t and is 20 times less performant. $h{time} is a hash but $h{(time)} calls the time function and get result as a key. @h{'x','y','z'} = 1..3 works (again promise of an array) and creates a hash initialized.
  • Tip my %str2errno = reverse %errno2str (list context). Another tip: @h{keys %y} = values %y;

Dense and useful session.

I took a pause and came back half an hour later, time to say helllo to Bdale, and some Mageia friends.

Stefan Hornburg (Template::Zoom)

I had more problems finding that talk interesting. I may not be the right audience, but also the monocord sound of the voice wasn’t helping. Stefan explained that:

  • Base is separation of Web design and programming
  • Some templates do not respect this (Template::Toolkit, HTML::Zoom)
  • T::Z provides static HTML file and spec file
  • Use Interchange (FLOSS e-commerce server) and ITL language
  • Config can be done with XML or Config::Scoped

Examples were given.

SawyerX (Moose: Postmodern metaclass-based object system for perl5)

After the previous talk, it was refreshing to see the enthusiasm that SawyerX deployed to convince us how Moose was wonderful. And I must confess that even if I’m not a big fan of Object Oriented approach (showing my age here !), I was ready to try after that talk. Exceelent one IMO.

First he advocated metacpan.org. Then he went on explaining that objects in perl 5 is a blessed hashref (bless {}, __PACKAGE__; (name space of the package), that the new method is manual, self is the invocant and he underlined problems with std Object in perl. And hows that half code is really needed.

He went on with Moose.
define an object in Moose is as simple as:

package ...
use Moose
1;

has name => (
is => 'rw', # or ro
isa => 'Str', # Attributes have type constraints Str, Int, ArrayRef, Regexp, HashRef[Str] + inheritance + own types
);
# there are setter/getter methods

inheritance is as simple as:
extends 'ParentClass'; (may have multiples)

Roles are behaviours (not a class of its own) is as simple as:
with 'a_role'; (may have multiples)

Hooks are ways to change the behaviour from inside Moose:

before leaving => sub {
my $self = shift;
$self->action;
};
after leaving => sub {
my $self = shift;
$self->action;
};
around login => sub {
my $orig = shift;
my $self = shift;
$self->ecurity_check and $self->$orig(@_); # runs the login method only when security check is ok

Some attributes:

default => 3,
default => sub { {} }, # but rather use builder
required => 1, # it's required
lazy => 1, # Only do the action as late as possible, especially for infinite loops
builder => 'build_it' # Moose doesn't know it so you have to code
sub build_it {
my $self = shift;
...
};
clearer => 'clear_it', # Moose knows it so you have to code, it clears the value as it never existed,
# but do not go back to default
predicate => 'has_it' # it checks that an attribute value exists (including undef) - doesn't create anything
lazy_build => 1 # same as lazy =>1 + lots

The final quote said: “Moose produces beautiful, clean and stable code and is here to stay“.

Additional modules:
MooseX:: SimpleConfig automatic creation of structure from config file
MosseX::Getopt
Catalyst is now based on Moose. Perf penalty is minimal especially for long running apps.

Incredible talk. Worth having no lunch for hearing it.

Then I attended an unplanned session, not on the paper program:

Alex BalhatchetWriting readable and maintainable perl

That talk covered some good tips and tricks on how to write perl code that is just here to stay as well !
Generic advises:

  • use strict is mandatory (avoids typos)
  • use warnings
  • use autodie # make open() and others die on error
  • use feature ‘say’;
  • Advertise CPAN (90000 – mostly well documented) makes code more readable/maintainable. Problem of choice (use testers report, rating date). Task::Kensho gives good recommendations.

Best practices:

  • code in paragraphs
  • throw exceptions (rather than error codes) with die (catch them with eval, or now try/catch
  • use builtins (use readline($fh), warn instead of print STDERR, glob instead of
  • use Scalar::Util List::Util LIst::MoreUtils (comes with most perl) brings min, max, last, first, …
  • Be consistent with existing code base (inconsistent is worse than unreadable)
  • Make sure there are tests, and good tests.
  • Perl::Critic (and perlcritic) does static analysis (referes the whole Perl Best Practices book !!)
  • Perl::Tidy (and perltidy) makes code more readable.

The presentation is available at http://www.sledeshare.net/kaokun

Gabor SzaboPadre
Gabor came back on scene to present Padre, IDE in perl5 for perl. Focus on beginers or occasional users.
I was interested for my son to whom I teach some perl now, and who would be a perfect candidate to use that tool (/me being a vi/vim/gvim user since 24 years, and not ready to change ;-))
He underlined that use diagnostics improves error msgs. He mde a demo of multiple Padre features. ALt-/ shows the contextual menu (for keyboard people !). Variable replacement based on content (differentiate $x from $x[0] and $x{“foo’} and replaces the right one.

Paulo CastroPackaging Perl and it’s deps

Paulo explained his problem: delivering a version of perl with all its modules (400+) for its applications.
He showed multiple possibilities and the one finaly retained, which consists of having a single package including perl and all the CPAN modules built for that version, using a local mirror to build it.
He also mentioned pmtool for inventory, CPAN::Site module and the CPAN Distroprefs features orientation.

Even if interesting, as a packager myself, I still find odd to use that approach, instead of using the packaging format of the underlying Linux distribution to perform this. Even for 400+ packages. In fact for that type of work, I’d myself pick and chose the Linux distribution having the most perl packages already done, and provide the rest myself. Debian or Mandriva/Mageia could be a good start in that perspective.

It was then time to pack everything and go to the train station and back to Grenoble. The 4 and half hours in the train were used to clean up my perl code in project-builder.org, as that series of presentations gave me lots of ideas and energy to do it !!

i look forward participating next year, and hopefully, doing submissions earlier as a speaker again.

Continuous Packaging Build Cloud with project-builder.org

2011/02/10

So now that I have your attention with this interesting Cloud buzzword, I can develop the idea i’m adding to project-builder.org at the moment.

Currently project-builder.org supports building in Virtual Machines (VM) or Virtual Environments (VE) aka chroot, which are all managed by the machine running the pb command. But with the expansion of the project I’m working on in our joint HP/Intel collaboration, we want to be able to support Continuous Packaging also on machine for which we can’t have a VM or a VE hosted, such as an Itanium HP-UX one or a Sparc Solaris one.

Of course, it’s always possble to log on one of these systems, install pb, give access to the VCS/CMS repository and voila, you’ve shiny new packages. But you (as I) want more no ?

So I’m realizing a new series of patches so that pb can launch build operations on a Cloud of machines, as long as you can connext to them through SSH. In fact the process is very similar to what pb currently does to build in a VM, except that this time, it can be a remote machine. I found it much easier to support than I thought in the code, proof that the design is not too bad and allows for easy improvements.

Also I made recently some good code cleanup after my stay at the FOSDEM where I attended a lot of perl sessions, that gave me energy to do that … during the 4 and half train trip I had to go back home.

And today, it took me less than 20 minutes and no cloud in the horizon to prepare 2 new VMs in order to support Debian 6.0 for future project releases. Tooling stuff is really helpful.

So expect a new version of pb RSN, in order to provide that additional support.

CDDBeditor

2011/01/30

I’ve been upset these last monthes by the fact that my usual applications which were allowing me to publish on FreeDB CDs content were not working anymore. I was using kscd from the KDE project, but the latest version with KDE 4.x is just not interesting anymore, with that feature having disappeared.

Same for audex which doesn’t provide it. Grip is Gnome based and doesn’t allow for category edition.

Too bad as it’s an area where a graphical tool is very interesting in general.

So, I decided to write a small tool to help me doing that. So here is CDDBeditor. Not a nice and fancy tool. But it provides to me what I need: CDDB data edition and re-send by mail.

It’s perl based (always a good way to lear), based on the CDDB_get CPAN module for the CDDB features, and also uses Newt, as a way to learn how to use this environment. It could be very seful for future versions of project-builder.org or even MondoRescue.

I use it since August 2009, and to my great surprise, I remarked that I was among the best contributors to Freedb.org with it !! More over, as I’m alone to use it (well I think so at least ;-)) I now know I created 274 entries in FreeDB last year with it. Still far from all the CD for which I got an entry already entered previously by others in FreeDB,but my small stone to the whole wall.

Project-Builder.org: 0.10.1 published and next steps

2011/01/20

So, finally Project-Builder.org 0.10.1 has been published.

A lot has been done in that version, so look at the ChangeLogs files for details. In particular, that version of pb is not backward compatible with the previous one, thus the change of version (from 0.9 to 0.10). Which also means that it is highly recommended to update your VMs/VEs with that new version of pb with the setupvm|ve command.

Another incompatibility is with the cms2* commands which now really take the content of the CMS/VCS not the sandbox status. In order to get the previous behaviour, please use the sbx2* commands instead. man pb is probably helpful 😉 In particular, if you build from files (via http/ftp) and not a VCS, you’ll have to use the new sbx commands to have your package build correctly, as there is no CMS in that case.

This version now supports parallelism during the sbx|cms2build, sbx|cms|build2vm|ve phases which can drastically reduce build time for packages. For this to work, as explained earlier, you need the perl module Parallel::ForkManager installed on your system. The VMs|VEs may benefit from it but it is not required, and only useful if you have lots of cores on your host, to enable multi-cpus VM|VEs.

A small bug has been found post-release for those of you using the announce command. What ? nobody ranted ? Well, as I’m probably the only one using that feature, sounds obvious 😉

I’ll now also work more with my FOSSology colleagues to help them produce packages for their just announced 1.3.0 version with pb.

More interesting is what will come next: As part of our joint work with Intel in our Solution Center, Project-Builder.org is part of a new solution stack we’re developing with their collaboration around RISC/Unix to IA/Linux migration. Continuous packaging was identified as a key aspect, and helping our customers support multiple platforms in parallel is indeed very useful. So it was agreed, also following an internal presentation and some surveys made during it, that our next supported platform had to be HP-UX.

So I’m working with the help of Josh Zhao on adding this, and this is expected for 0.10.2. And another colleague Nicolas Doualot is helping setting up a demo environment to showcase our Continuous Packaging approach.

Signing packages is also high on my list, but is still a problem on RPM based distro. Hopefully I’ll be able to come up with a solution after some FOSDEM discussions !

Migrating from KMail to Thunderbird

2011/01/10

This week-end, after passing too much time to try to understand contact management in KMail (which was working quite well in KDE 3.x but is completely flawed IMHO in KDE 4.x), I decided with my wife it was time after 8+ years to try another mail reader.

Thunderbird was of course the first candidate, due to the familiarity she had with Firefox, Outlook (at school) and the fact it’s not that different from KMail anyway in its behavior and interface. I must say I was impressed to see my wife just say that she didn’t mind changing Mail reader, in order to get lacking features. Probably her familiarity with Open Source has also created an open mind for computing programs, and the fact she already uses multiple tools without major issue, makes her willing to try new ones.

So I started to work migrating her data from KMail to TB. I found a small program in order to help md2mb.pl. However, it was far from being complete, and I decided to rewrite a new one, based on the ideas from the other one, in order to have a better migration support. So here is a new md2mb.pl which hopefully will be useful for others as well.


#!/usr/bin/perl -w
#
# Program to import a maildir kmail environement into a thunderbird one.
# (c) Bruno Cornec under the GPLv2.

use strict;
use File::Find;
use File::Copy;
use File::Basename;
use File::Path;

my $cmd="formail";
# CHANGE AS YOU WISH
my $oldroot = "/beatrice/.Mail.sav";
my $newroot = "/beatrice/.thunderbird/30cq2rn3.default/Mail/Local Folders/";
# Is the newroot a file (1) or a dir (0)
my $nrisfile = 0;
my $debug = 0;
# END CHANGE

$debug++ if ((defined $ARGV[0]) && ($ARGV[0] eq "-v"));
print "DEBUG MODE, not doing anything, just printing\n" if ($debug);
if ($debug) {
print "CMD1: mkdir -p $newroot\n" if ((not -d "$newroot") && (not $nrisfile));
} else {
mkpath("$newroot",0, 0755) if ((not -d "$newroot") && (not $nrisfile));
}
system("$cmd /dev/null 2>/dev/null") == 0 or die "cannot find formail on your \$PATH!\nAborting";

find(\&md2mb,($oldroot));

sub md2mb {

if (-f $File::Find::name) {
return if (
($File::Find::name =~ /\.ids$/) ||
($File::Find::name =~ /\.sorted$/) ||
($File::Find::name =~ /\.index$/) ||
($File::Find::name =~ /\/cur\//) ||
($File::Find::name =~ /\/new\//) ||
($File::Find::name =~ /\/tmp\//));
}
if (-d $File::Find::name) {
return if (
($File::Find::name =~ /\/cur$/) ||
($File::Find::name =~ /\/new$/) ||
($File::Find::name =~ /\/tmp$/));
}
if ($debug) {
print "CURR: $File::Find::name\n";
}
my $destname = $File::Find::name;
$destname =~ s|^$oldroot||;
$destname =~ s|\.([[:alnum:]éèçàù\s]*)\.directory|$1.sbd|g;
if ($debug) {
print "DEST: $destname\n";
}
my $cdir = dirname("$newroot/$destname");
my $outputfile="$newroot/$destname";
$outputfile="$newroot" if ($destname =~ /^\s*$/);
if (-d $File::Find::name) {

my @files = (,);

if (@files) {
if ($debug) {
print "CMD2: mkdir -p $cdir\n" if (not -d "$cdir");
} else {
mkpath("$cdir",0, 0755) if (not -d "$cdir");
}
}
foreach my $file (@files) {
next unless -f $file; # skip non-regular files
next unless -s $file; # skip empty files
next unless -r $file; # skip unreadable files
$file =~ s/'/'"'"'/; # escape ' (single quote)
# NOTE! The output file must not contain single quotes (')!
my $run = "cat '$file' | $cmd >> '$outputfile'";
if ($debug) {
print "CMD3: $run\n";
} else {
print "Copying maildir content from $File::Find::name to $outputfile\n";
system($run) == 0 or warn "cannot run \"$run\".";
}
}
}
if (-f $File::Find::name) {
if ($debug) {
print "CMD2: mkdir -p $cdir\n" if (not -d "$cdir");
print "CMD3: cp $File::Find::name $cdir\n";
} else {
mkpath("$cdir",0, 0755) if (not -d "$cdir");
copy($File::Find::name,$cdir);
print "Copying mailbox content from $File::Find::name to $outputfile\n";
}
}
}

Warning: this code doesn’t work from the blog as wordpress makes some modifications, so download it from here

Run it first with a -v option in order to see what will be done. Be careful as I’ve just fixed the problems I had on my side, but probably not all of them. Especially, take care of file names with “‘” in them. they won’t work well.

So migration has been done, and of course more re-organization of the mailboxes. But now everything works fine, including contact management (that were imported using the LDIF exchange format).

And my wife has already sent a couple of mails, so I guess it will not take her long to be very familiar with TB.

Once again this shows all the power of FLOSS: freedom of choice, no info hidden, open format, and voilà.

project-builder.org: nearly ready for 0.10.1

2011/01/08

Next version of project-builder.org won’t be 0.9.11 as expected ! This is due first to the incompatibilities introduced with 0.9.10. a pb 0.9.10 cannot work in a VE|VM with the current devel version of pb, and the VE|VM will have to be updated with setupve|vm.

Also on top of that quite a new set of features have been added since last version, in particular the parallelism of tasks. After the work on sbx|cms2build, I’ve done now build2pkg and also build2ve. It implied more modifications in order to have each script file generated during the preparation phase unique so that each VE could get it correctly. I’ve tested this night with 2 VEs to build pb packages, and it is now working quite well, and that’s where having a multi-core, multi-thread machine such as min base on a Core i7 Xeon helps a lot aand makes a difference.

The last remaining part is to also add that feature to VMs. I’m very near from it, I just need to handle the fact that communication based on SSH with each VM has to use a different port, whereas now it’s the same for a given project. Hopefully this will be done over the week-end, which will allow publication of the new stable version of pb in sequence.

This feature is also useful for my colleagues working on the FOSSology project to generate packages for their 1.3.0 version on their side.

project-builder.org: more parallelism

2010/12/14

Looking around to improve some of the performances of project-builder.org, I found an interesting perl module: Parallel-ForkManager.

Its man page shows how you can quickly parallelize a loop in order to benefit from the numerous cores that each system today has. It took me just a couple of minutes to transform project-builder.org, so that the generation of all build files becomes a parallel operation, thus reducing dramatically the time it was taking – at least on my Xeon X5680 @ 3.33GHz !!

Very easy as you can see in the changeset, and very efficient. And if you don’t have the module, you can still work in serial mode, but once you tested it, you don’t want to go back 😉

Next step, is to also apply that feature to the build in itself on multiple VMs or VEs, but that will require a bit more work, as the loop is not so easily usable as of now. More on that later, once the code is re-architectured.