Meeting at the first Distro Recipes

I’ve been kindly invited for the first Distro Recipes event in Paris the 4th and 5th of April.

As I have an internal HP meeting on the 4th, I’ll be only available at the end of that day, but will present on the 5th how Hardware manufacturers work with Linux distributions, giving the example of HP. I’ll also monitor a round table aound “Linux distributions: differences and commonalities” where we will try to have polite discussions about what makes a distribution unique, and what is instead worth sharing by collaborating. Finally I’ll also present during the lightning talks “Project-Builder.org: packaging for multi-OS Open Source Projects“

So won’t have that too much time outside of presentations, as you can see, but would be happy anyway to meet with MondoRescue or Project-Builder.org or HP/Linux users and talk with them.

Anyway a great event to be in, as the list of speakers is really interesting, all majors distros being represented, and for sure very interesting new contacts to make, and hopefully the curiosity to discover these other distros that you don’t use Come for the same reasons, and see you there !

Time to drop flash

I’ve never been anti-non-FLOSS: I’ve used StarOffice back in 1995, when it would allow me to not use a Windows PC, but to do everything I had to do with a LInux system. I’ve used and still use AcrobatReader (and Okular). And on LInux I’m using flash, especially to look at Video published, such as on http://youtube.com

But today, trying to get an update for flash, I read on Adobe’s Web site that Flash Player 11.2 would be the last version for Linux. Only security fixes will continue to be provided. Well so instead of being an incitation to move back to Windows (you dreamed guys ) or adopting Mac, It’s an incitation to drop flash usage as much as possible, and use more open video format.

Don’t get me wrong, I’m always favouring open format, and free, libre, open source software. But I’m also practical, and if I need to use a software to have my work done which is not FLOSS, I dislike that but can use it as long as it’s not core to my activity. And honestly, flash is probably the last one with regards using non-FLOSS on my systems. Flash is not core anyway. It’s for entertainment mostly so I’m ok with a proprietary plugin, especially when Mageia does a great packaging work making it very easy to use.

But now, if Adobe doesn’t care about Linux users, then all Vidéo providers shouldn’t care either about the flash format and start moving off it ASAP. (including french TV for their news).

When I think about the “awesome” presentation I had today at LCA, about native (without plugin) Video conferencing between Firefox and Chrome, using native HTML5 WebRTC format, I think all these funcky formats are just doomed to disappear anyway. The richness of Open Source, and it’s rapid evolution pace doesn’t allow anymore to companies, even the size of Adobe, to resist. And for sure their decision will accelerate the move. Especially as mobile users, who are mostly Linux users nowadays, ar using more video content.

So many thanks to those who are working on such standards and techno; it will make our lives much more easier, and still fun in a near future.

Meeting at Linux.conf.au in Canberra

I’ll soon be lucky to be able to be in a plane for some 20 hours in order to reach down under and be in Linux.conf.au in Canberra ! It will be my second time in Australia after my previous presentation on MondoRescue in Sydney in 2007. This time I’ll organize the cross-distributions MiniConf on Tuesday the 29th of January 2013.

And I’m so happy to have fantastic speakers such as Bdale Garbee or Monty Taylor among others ! I anticipate it will be a great Miniconf. So fell free to come and participate, you’ll have the best people to give you answers

And as usual, if you want to talk about packaging, disaster recovery, open source or early music, feel free to come by and talk with me. I look forward discovering another part of thies great country in two weeks.

Meet at HP Discover next week

Hello,

I’ll be at the major HP event (HP Discover) next week in Frankfurt, Germany from the 4th to the 6th of December, delivering 2 sessions, and attenting some others which look very promising.

You may find me on the Red Hat booth or the Intel booth, if you want to talk about code and projects (MondoRescue, Project-Builder.org, UUWL), Architecture, FLOSS Governance, FLOSS @ HP or in general. Will be happy to exchange with you around these topics.

You may find more details (in french) on the sessions I’ll be delivering on the event blog site.

See you there !

Presenting FOSSology at LinuxCon, San Diego next week

I always find strange to be accepted as a speaker to LinuxCon on a subject for which I’m much less an expert than the other ones I proposed for which I’m leading the projects ! It happened last year for the EMEA event, and same stuff again this year for the US one.

But I won’t be criticizing here, as it’s my first possibility to visit the US west coast, and also my first time as a speaker to LinuxCon US so Champagne !! So I’ll be talking about FOSSology, the HP sponsored GPL Licenses analyzer tool.

So if you happen to be around, and want to discuss abour FLOSS, MondoRescue, Project-Builder.org, HP and Open Source, or something else such as early music, then feel free to come and talk. Well I’m sure you won’t come to see me, won’t you, but once you’re there to see the stars, just come and say hello

Meeting at Solutions Linux 2012

Well, I arrived this evening in Paris, and I’m ready to participate to a new Solutions Linux this year. Still remember the first time I came to Linux Expo (as it was named at that period) back in 1999, as an exhibitor for Medasys, manging the HP booth.

My main duties are on Thursday where I’ll lead 2 round tables and as time permits the relative sessions: COMMUNAUTES DU LIBRE : ORGANISATION ET DEFIS and ADMINISTRATION SYSTEMES / CLUSTERS – GESTION D’ECOSYSTEME MATERIEL ET LOGICIELS LIBRES.

The rest of the time, I’ll attend sessions to discover new tools, visit the various booth to meet with new communitites, and of course shake hands of the many people I know there and with who I’ll discussed a lot

Hope to see you there even if I do not deliver any session myself this year.

Let’s meet at Fosdem 2012 in Brussels

I’ll attend Fosdem again this year next week-end in Brussels. I’ll deliver a talk on Project-Builder.org as a support for a Continuous Packaging cross Operating Systems development.

There are some news with the tool, and hopefully a new version, and some future evolution that I’d like to communicate. I also plan to present less slides, and have a more concrete demo to help people see the value of the approach.

While not presentting, I’ll probably be around near the Mageia booth or around my HP colleagues attending the event as well (Bdale Garbee, Martin Michlmayr, Hugo Roy). Don’t hesitate to come and chat !

Second Day at LinuxCon EMEA 2011

After a busy first day, here is the report for my second day at LinuxCon EMEA 2011, which started directly with some sessions (I skept the plenary for once):

Distributed redundancy by Roopesh Keeppattu – Huawei

Redundancy is about availability, by duplicating components to avoid unavailability of the service.
Availability measured with ’9′.
4 nines is 1 hour per year, 5 nines means 5 minutes, 6 nines 32 seconds.
Major types of redundancy: standby (cold – the other server remains unpowered, warm – all servers powered, hot – all servers provide identical services).
ALso notion of N modular redundancy (N servers in parallel).
1:N redundancy = 1 standby for N active units.

Traditional redundancy: mainly based on backup HW systems, with similar capabilities so large CAPEX and OPEX.
So moving to distributes redundancy to reduce costs.
1:1 scenario taken in account
Instead of duplicating HW, there is a duplication of process instances on a set of servers
3 models available: live migration of OS/VM or of processes or of pre-distributed processes

• live migration OS/VM: preserve states, less complex from application point of view but higher migration time due to size of what to transfer.
• processes migration: you encounter more complex migration design, which has to be part of the application, and gain on the data to transfer. This method can also provide dynamic load distribution. But you need pre-failure detection for failover.
• pre-distributed process: you increase again the ressource optimization, availability and switching time but also the complexity, the need of additional SW to deal with states and the linkage to the application.

The future is in redundancy in the cloud, and ressource abstraction.

I was hoping for a more in depth presentation, and was not satisfied by this one, as it didn’t go into details, just remaining at the surface. However, this is a critical topic for most customers today in their Linux adoption.

Experiences booting 100s of thousands to millions of Linux VMs by Andrew Sweeney – Sandia National Lab

Managing a large number of VMs presents some challenges and horror stories (such as filling fill the switch CAM table, creating VM feed back loops, finding some unique bugs or odd behaviour). Even 0.01 % of error is 100 VMs in their case.

They tried multiple technologies such as lguest, QEMU, KVM, NOVA. They are using a mixed of technologies due also to hardware limitations.

Guests configurations are computed at runtime. Everything is stored in RAM. They treat VMs as an application process. They use standard tools, the same TCP stacks, kernel…

They are using VMatic to generate the images and boot 1000 VMs in < 3 minutes.
Another tool used is Gproc (Cluster Management tool written in Go) allowing O(ln(n)) execution time.. It scales beyond 200K+ instances. Web based interface.

The first cluster type was:

• Using PXE boot for booting physical host Hypervisors and then start the guests.
• In July 2009, they reached 1 Million VM with lguest and 4600 Dell Super Computer (256 lguest per node) bootleneck being RAM.

They then created KANE (sort of their own cloud approach) made of 520 nodes with 12 GB RAM with Video cards (because that was more expensive to remove them !!)) 13 racks, 40 nodes/rack, 1 PDU/rack.

Then they developed a strongbox ARM cluster made of 490 nodes 512 MB RAM little power needed with lguest.

Then they started Megatux 2.0 to reach a higher number of VMs. Everything is virtual. They even created a network creation language. Use virtual quagga & linux virtual routers (+ physical) and virtual VDE switches. It supports multiple OS normally, but for Windows they got many blue screen (ipconfig before IP is sup, ping before IP is up, …). They’re using KSM a lot (and made patches) and various approaches to reduce VM footprint. gproc is used after the initial boot to push the VM images and start the VM + aggressive KSM.
Cold boot to experiment is performed in 7 minutes. 1 daemon per host to regulate KSM, VM state.

Interesting problem to collect info from 1 Million of nodes overloaded and where to store it ? Using network sniff, VM inspection. For that they used a MongoDB backend populated at runtime. Data collected in in best effort mode.
They’re looking at using KVM tool instead of QEMU/KVM to reduce memory footprint and AXFS (Advanced XIP FS) combined with cramfs.
Next steps with Android, more realistic network usage, improved monitoring, data visualisation and error handling.

A good talk on very unusual context with some interesting issues to consider, even if far from being current problems as of now.

I then met with my colleague Sue Paylor, who is one of our excellent FLOSS expert in EMEA, and that was again a good talk exchanging about our respective customer experiences, how to improve HA with Linux, and lots of various topics.

Providing High Perfomrance Round table (instead of SuSE Keynote)

Ludek Safar, Ministry of Interior, Czech Republic approched the linux topic from the desktop side, and they’re now moving to the Data Center (Oracle instances on physical hardware and the rest in Xen VMs including java based custom devs.). They help by giving publicity for some FLOSS projects. The choice of an enterprise distribution is specifically to be the linkage with the communitites. He likes the embedded approach with regards to the fully integrated hypervisor which provides the perfect cloud solution for them.

Dr. Udo Seidel, Amadeus explained that they started 9 years ago with Linux. They have done lots of internal developments including lots of mission critical workloads. Participating to events is key to keep good technical exchanges, influence the developments, give feedback. He really likes the flexibility and the open mindset. However he is still missing a central approach around role based manageemnt (a la AD).

Andreas Pöschl, BMW explained that they started back in 2003 for servers, and in 2006 decided that Windows and Linux were the strategic OS on x86. They run SAP on Linux e.g. and desktops on Windows. They do virtualization (1000 VMs) with Xen, including 16 cores 64 GB VMs for SAP. They don’t do direct contributions, but rather provide use/test cases for large configurations, and rely on their distribution providers to do the return. Sharing what they do with Linux is also important to improve the ecosystem. He insisted on the freedom of choice which avoids vendor lock-in and also marked his appreciation for the large set of possibilities offered by FLOSS. He is still concerned by boot time. BMW has requirements around storage and scale out, so they appreciate the work done on Btrfs. He mentioned usage of Linux in GENIVI that will bring infotainment to the end users.

Nils Brauckman underlined that the SuSE company, is organized to take this feedback and make it available upstream, doing that since 20 years, as well as providing mission critical solutions to customers, and detailed the new features brought into Linux 3.0 (btrfs rollback, snapshots, trace capabilities, …) bridging the gap between Unix and Linux. He underlined also for SuSE the new agility brought by being back as a separate Business Unit, operating like a single company.

I like more and more this type of round table, as it gives concrete production example of FLOSS usage, and show how serious customers are today, and also how far they want to push their usage, which creates interesting challenges for us !

It takes a community/village to raise a Distribution by Tim Burke, Red Hat

"Unix was a job, Linux is a crusade" Tim said it’s awesome to be a part of RHEL as well as OLPC.
He started by showing a large set of stars in the sky (glibc, LVM, X.org, Linux), independant stars that only come together when gathered in a distribution, which give them visibility. Then he showed the various actors, hardware vendors, translators, designers, lawyers, testers, and distribution vendors as well. The real competitors of Red Hat are VMWare, Microsoft, not the other collaborative groups such as other distribution makers. He explained the relationship around the kernel between upstream, Fedora and RHEL. He also underlined the benefit of working upstream such as they did around the Real Time extensions, instead of coming with a large patch developped separated.

The role of distribution makers is also to coordinate with hardware vendors (I’m well placed to know that !). Distribution can help create communities such as for AMQP, which was a real common need among FSI companies, as they know how to do it.

Mantra is "get it upstream first". Being divergent is being ignored, costs more, represents more work.

Time then gave some numbers:

• 80% of Fortune 500 run Linux.
• 92% of supercomputers for healthcare or analytics run Linux.

He mentioned the OVA to bring up in the stack integrated solution based on KVM.
No keynote without cloud, so Tim had to mention it and noticed Linux usage in it, and the integration characteristics it requires, very near from the one you have to make a distro.

A good talk, but not as pushy as the one made by Jim Whitehurst

How Linux runs the World of Finance by Christoph Lameter, Graphe Inc.

Christoph started by explaining the various players (Stickes, traders, banks, …) and explained their needs of speed. This creates the need for certain technologies (Real Time, kernel, binaries and network optimisation, RDMA APIs, fast C++ code, processor caches). One problem is the limitation of speed of light (even if that may change !). That sounded like a joke first, but is very serious !! 200 µs to go round the earth. It creates limitations to signaling of events.

We’re moving from manual to automated trading. Hours vs ms, human vs compute/algo, 30-60 trades/min vs 1000/s. Manual is used as a backu p mechanism only today.

The case for Linux is because you can modify what you want, and such win against competitors by speed improvements. The first there wins ! Windows couldn’t make it in term of latency in its network stack. Linux was already used for Internet, large companies such as Amazon, Facebook, … All major stock exchanges are on Linux today. Commercial solutions vendors focus on Linux. Solaris is diminishing after Oracle bought Sun.

Distributions used are mainly RHEL, some SLES (Germany mainly), a bit of Gentoo and Ubuntu/Debian.

There are still some challenges for Linux in Finance: involvement upstream is rare, as they want to protect their advantages. Regression in kernel components is creating higher latencies (so some still run RHEL 3 !). Christoph Gave an example of a customer having a 200% regression moving from RHEL4 to RHEL5.

The Forward path is with direct access to hardware (OS bypass) to gain on latency. RT linux does not scale and increases average latency. RT linux is used by exchanges not traders.
Linux dominates finance for the forseeable future. Common hardware looks like supercomputers today (Numa). HPC goes mainstream. Offload technology is seen with suspicion by the community. So again no willingness to contribute these improvements upstream.

One of the best presentation of the day, with lots of anecdotes, and a visible knowledge of the topic end to end.

Where is the Money in Open Source? Business Models and the Marketing of Open Source Technologies by Nithya Ruff, Wind River Systems

Nithya created a story to illustrate this talk. 3 communities: producers, distributors, consumers.

• Producers are interested by solving problems. License used is key. It’s all about meritocracy. How do developers make money ?: being hired by a company, consulting contracts, venture funded, sponsorship/grants/donations.
• Why consumers use linux: no vendor lock-in, comparable perf and high quality, time to market and savings, choice and flexibility, empowerment,, innovation and transparency
• Distributors make it available for consumers with support, favour FLOSS adoption making it safe to use, employ developers, solve some issues and contribute back, market FLOSS, and serve as a liaison between consumer and developer. Successful business models are subscription, services fee, training, books but also proprietary extensions

Marketing FLOSS is different. You need to clearly articulate your added value in the ecosystem. So you have to add value. (TTM, ROI, Integration, risk mitigation)
Prediction, by 2021, 100000 infrastructure core endpoints and 1B mobile endpoints and 20B MtoM endpoints.
Even more need to collaboration between the various communities.

I was expecting a bit more from such a presentation. Good for beginers, but lacks new thoughts on our ecosystem.

ReaR by Dag Wieers

I was particularly interested by this presentation as ReaR is a MondoRescue competitor, and Dag is mister rpmforge, mrepo, … so was really curious to attend it.

Rear provides a Disaster Recovery Workflow in bash. Its framework is easy to use and extend. It supports HP SmartArray, SW Raid, DRBD (not MondoRescue !), LVM, multipath, ext2,3,4, xfs, jfs, vfat. It supports tape, ISO, USB, eSATA, NFS, CIFS, rsync, HTTP, FTP, SFTP. It also provides back-ends with TSM, HP DP, Bacula, …

ReaR works on RHEL4,5,6. It’s shipped with SLES (the one distribution on which it’s tested).

It saves storage info and network info. It has local GRUB integration, serial console support, network and SSH key integration, syslinux management.

Dag then explained the use case of the Belgian Federal Police (HP-UX to Linux migration using Ignite before):
Developers prefered USB usage for flexibility instead of OBDR (also lack of OBDR support by latest HP HW). It manages labels on tape and USB devices. For this project, they support a central DR server with PXE boot and control the HTTP PUT upload with ACLs.
They provide a tool to detect when changes are needed to relaunch ReaR by cron.

In the future they plan to work on: better rsync support (like rsnapshot or rbme), more backup backends, PXE integration, code base reorganization, release process, website+doc, dev tools.

Dag made backup and restore demos.

I really liked the presentation. Dag is an excellent presentor, and has accomplished a huge work to improve the tool.If only I could also have some brilliant contributors like hom for my project !!

So after the presentation, I introduced myself to Dag, and we ended up talking together most of the evening during the dinner organized in a central place of Prague. We talked not only about DR, on which we share a lot of common ideas, but also about a large set of other topics, some of them HP related such as webOS future, … I like making new relationships during evens like LinuxCon as you end up talking with luminaries and that helps a lot enrich your own vision.

Some pictures of this event are available on Picasa.

First Day at LinuxCon EMEA 2011

First LinuxCon ever in EMEA this year !! I’m more than happy to see it at least on our continent, and was glad to be selected to give there a talk (after the one I had made in Brasil last year).

As every conference, this one is starting with keynotes.

Keynote on a world without Linux – Jim Zemlin (Linux Foundation)

Jim was celebrating here the 20 years of Linux. He looked at how would be the world without Linux and the answer as you can guess it is nowhere ! He underlined the high number of Android devices, bind on Linux naming the Internet, and he also looked at so;e quotes and predictions from Bill Gates/Steve Ballmer to show the evolution – from the caner up to Miscrosoft contributing to the Linux kernel this year ! He also used lots of video presented on the Linux Foundation Web site. Jim is absolutely partial, and that’s good to hear

Kernel Hackers Round table (Linus Torvalds, Paul McKenney, Alan Cox, Thomas Gleixner moderated by Lennart Poettering)

Some notes of that open discussion. Linus put emphasis on not breaking user space. He gave the example of the introduction of a 2.6.40 version instead of 3.0 to help some programs to remain compatible ! He underlined that breaking things on purpose should be avoided, and counter-examples were given of security issues that forced the kernel community to break the kernel ABI. Linus used to run an old a.out binary from years ago (COFF format) to ensure the compatibility level, even if it has not done so for some time now. Linus said that the Open Source approach makes modifications much more easy and allow to deal with kernel complexity better (contrary to common belief that would imply that managing such a large community without stritc rules and methods would be impossible.

The average age of the kernel summit participants is increasing of one year every year said Linus Which is linked to the maturity of the community, and the fact that it takes time to take over subsystems. There are lots of new contributors, including young ones, making very few changes. There is not really an age problem in their opinion. Thomas also added that you need a balanced aproach that only older people can bring in a project of that size and complexity.

Linus said that ARM made some stupid decisions and had a lack of standards until very recently, especially with regards to x86 where Intel is playing the game fairly. Kernel support for ARM is 10 times the size of Intel’s because of the need to support multiple variant. Which Linus is sad about as he thinks this is the most important platform outside x86. But they are getting better, Linus is much happier today than 6 months ago even if there is still work to do.

Linus mentioned that he runs 3 (three) FireWall to protect his environment ! And I thought I was paranoid zith my 2 . About SMP he first said who cares ? Now zith the high number of cores everywhere, even in phones, it’s seen just as normal. So who cares about cgroups, VMM, … Well, some need it and are ready to pay for the penalty. And who knows how it will evolve.

At the end Linus said that he is trusting people sending patches not companies.

It’s always interesting to hear what these guys have to say, and anyway Linus is my hero

Tizen – Dawn Foster (Intel)

I was interested to hear what was behind Tizen just announced recently. I was a bit disappointed as no architecture has been validated yet, so nothing concrete to announce here

Tizen is HTML5 based for application development and offers WAC API (favour code reuse across platforms/devices) and it provides a FLOSS ecosystem.
The first release is expected to be in Q1CY12.
The transition from Meego is possible, but Tizen is not a derivative from Meego, it’s a new project and some Meego maintenance activity for 1.2 is still planned. Compliance will also be reviewed compared to Meego and they want to have it less rigid.

Dawn said that they would rather publish what exists and is in place, rather than what was done with Meego (announces made too early).
She insisted on the various communities, and means of communication (IRC, ML, Wiki, …)

She gave the mic to a representative of the Mer project which goal is to take Meego code into a new direction (Core optimized for HTML5/QML/JS) (Cf: http://merproject.org).

The question around Qt availability is not clear now. Anyway once open sourced, the community could make it happen

As said earlier, the architecture is still not out, and should be really soon now. The devices targeted are Handset, TV, smart phones, tablets…
They want to align more the Governance model and the reality of the governance with regards to Meego.

So promising, but not yet concrete. Also remains to see the position of this new OS compared to Android and …WebOS

File and Storage Systems – Ric Wheeler (Red Hat)

Ric started by mentioning that Linux has a world class storage, supporting a wide variety of device types, and scales well (GB/s of IO, IOP increase for PCI-e, 100′s of TB).
So what’s wrong ? Well, e.g. keeping up with competition’s management platforms (VMWare in particular) especially around storage management. He underlined that standards around array offload functions are not driven by Linux companies. And that ease of use on Linux is still hard. Linux has several level of layers (MD, DM, LVM, FS, mount options)

Linux has powerful and sophisticated CLI tools, but no good library today to manage storage (no abstraction layer, typically around snapshoting e.g.).
Making things easier implies identifying common operations per use case, a common API, reducing the options of mount and mkfs, and avoid jargon (LUN, ALUA, barrier, …).

He then mentioned some ongoing projects:

• Btrfs: single interface to LVM, RAID, ease of use.
• Fsadm: keep the stack but provide a simple interface. (controls FS and LVM)
• Standardized options between FS and kill dead options. Default options are critical
• Oracle storage connect (Joel Becker) in python recently open sourced. GPL/Proprietary license for plugins from EMC/HP/…
• Libstoragemanagement (Tony Asleson – Red Hat) under the LGPL and look for interesting contributors. similar to the Oracle project: a vendor neutral API to allow for storage array management (cloning, mirroring, snapshots, …).

There are vendor APIs: VAAI (vSphere API for Array Integration) and also work on automatic offload operations.
Ric took the snapshot example: btrfs do it at FS level, LVM at block dev, storage arrays at HW). Users should be able to choose.He also cited the copy example: for SCSI (SCSI token based copy offload) and NFS (in NFS4.2 as server side copy)

Ric has the art of making these complex topics very easy to understand by his abilty to syntheticly present them, and give a good overview of where we are and where we go.

I skipped the Mission impossible session, which I found not that interesting, after attending a couple of minutes, in contrast to a very promising title.

Freedom out of the Box! – Bdale Garbee (HP)

Impossible however to miss that one ! Bdale is another one of my FLOSS heros

Bdale started by explaining what the FreedomBox was: A personal server running FLOSS designed to create and preserve personal privacy, running on cheap power-efficient plug computer server that individuals can install in their own homes.

Political aspects as well as privacy aspects (who shares what with whom) were clearly explained and this was obvious that this new device is thus contributing to building a privacy-respecting federated alternatives to contemporary social networks.

As its cousin the OLPC, It favours mesh networking.
The software is based on the Debian project (focussing on freedom as well, being international, multi-architecture, and benefiting from a strong infrastructure). Bdale indicated that the future Debian stable should have everything to create a FreedomBox out of the box.

Bdale then described the FreedomBox Foundation (FBF) relying on 4 pilars (technology, user experience, publicity and fund raising with industry relations). Ease of use is central, as some pieces of software are complex to configure.
The FBF has now various Working groups, so contributors have plenty of areas to contribute to !

DreamPlug was first selected for the implementation platform (made by GlobalScale Technologies) using a Marvell Kirkwood (ARM on chip) processor with 512 MB of RAM + 2 GB of Flash + a 2 GB microSD card for the kernel and root FS + 2 x Gb Ethernet ports + Wifi + USB + e-SATA + SD socket + audio. Quite amazing in such a form factor !
The Marvell uAP chosen has some technical challenges (FW and driver outside of kernel tree – which probably won’t change in the future – user space tools were binary only, now GPL). They gave their modifications back for GPL u-boot (better late than never

How to trust first a Freedom Box ? A study is ongoing with Smartphones to facilitate initial key exchange (Stefano Maffuli). Debcamp before Debconf 2011 was useful to create a great community to work on various topics.
First application to appear could be a secure XMPP chat one

This topic, is a very sensible one currently, after the population move in arabic countries. Privacy should remain a concern of every day, as our freedom, not only in software, is precious, and technology should be here to help us reinforce it rather than alienate us. Bdale is supporting a great initiative, first of this type, and that should allow us in the future to have a real P2P Social Network, not control by a central entity.

The it was time for me to jump on stage:
FOSSology a GPL compliance tool – Bruno Cornec (HP)

FOSSology is still a unique tool, developped by a great team lead by Bob Gobeille (HP), and deserve that we pass time to advertise it. I made a status of the current versions and their features, calling for more contributions to enhance the platform. I was happy to meet with the dutch translator of the tool, and to have some interesting questions about SPDX support, leading to some animated talks !

The lack of web/ftp availability for the project, due to the Linux Foundation infrastructure is still hurting the project, as well as SPDX. Hopefully this shold be solved soon now.

12 years of FLOSS license Compliance: A historical perspective – Bradley M. Kuhn (SFC)

Bradley started by explaining the GPL quickly. He compared it to the US constitution.
He also explained how it works in theory and in reality, especially when people don’t respect it.
If social presure doesn’t work, you need to go to court for copyright enforcement (same as the MPAA !) but for good reasons. (at least we hope !)

GNU Emacs was the first GPL’d program and its copyright was never infringed.
GCC was the second. More interesting for proprietary SW companies. Next (the company) was the first GNU GPL violator (so Steve Jobs !!) with the Objective C front-end. Violation was resolved quickly with code publication.

GNU tar was used by lots of backup companies, which were also violating the GNU GPL. Sysadmin found them, and all but one violator came into compliance. Last GNU tar enforcement was mid-2002. The company decided to remove tar and rewrite it.

Nothing concrete for SFC to get from a court (money or injunction – already done – but no code, which is the ultimate goal)

Bradley then reminded the Linksys (Cisco) history with busybox (Erik Andersen) and Linux (Harald Welte). Compliance takes soooo long. In that case, Broadcom was the upstream. Source was finally released, but the driver remained proprietary (due to FCC policy prohibiting it). OpenWRT FLOSS project spawned from that release. Harald was frustrated by the time it took in the FSF to launch that action and he created gpl-violations.org in order to go to court earlier than what FSF was doing. He organised 8 lawsuits in Germany (2005-2008) getting mostly injunctions.

How to fund enforcements ? The violators should be paying. (The SFC had a compliance program costing 10kUSD per Software which is too expensive so doesn’t work).

He acknowledged that dual licensing (a la MySQL) is a corruption of the GPL.
SF Conservancy is helping Erik Andersen since mid-2006 with copyright enforcement (request queue is > 300 right now). Lawsuits become necessary. Goal is to settle with full compliance (get the source code). Money and injunction is a consolation price only.

He then explained how bad some OEMs are by not providing code to their customers and letting them be accused of violations.

He then talked about the build environment underlining that normaly the GPLv2 forces people to also release script to compile and also to install. The GPLv3 phrases it even better.

He advertized a lot FOSSology vs BlackDuck, mentioning anyway that it doesn’t solve the redistribution issue (which is a human task to do, where tool are just helping). He mentioned that there is a free software to scan binaries (didn’t give the name however).

He also mentioned that HP was a fair participant to the ecosystem, Scott Peterson (now at Google) being very responsive to his queries around compliance questions.

Another view, more centered around trials and legal actions, that have become a necessity to have our licences respected. I just hope I’ll never have to be involved in this myself, as it sounds like a lot of headaches in perspective !!

Some pictures of this event are available on Picasa, and I was so happy with my new Nikon D7000 which makes so great pictures in such difficult conditions. I’ll have a problem going back to the D70 now

Third Day at OWF 2011

Last but not least day at OWF this year. Yep, this is a saturday, but this was the code time for me, so probably the most interesting no ?

I started the day my making first 2 interviews for the WebTV of the event, one in french and another one in english explaining Open Source at HP. That’s what you get by sponsoring events So now, my kids can compare me with every other product seen on the TV !! By the way, it allowed me to understand that the professional video world is still unable to use FLOSS only technologies to perform such activities, and if you have spare time, and no defined project to work on, that would be a great addition to our community.

Then it was time to attend the sessions of the day and take some notes to share with you.

Lua and its ecosystem by François Perrad.

Lua is developed in Brazil using a closed development approach (just tar balls), even if it is Open Source since 1993, under the MIT license (which François underlined as important for its adoption by the embedded applications).
It’s a scripting language, fast, powerful and lightweight.

v5.1.4 is the stable version (from 2008)
v5.2 is now ongoing
(Some people still use v4 for its C API and the apps developped with it)

The language is concise but powerful: EBNF Grammar of 1 page and a Reference manual of 100 Pages (with C API, and libs included). It is 12.5 kLOCs and the binary size is 150 kB.
There are only 8 types and 21 keywords.
Lua has a table type like a hash, which allows to make everything.
It also has a userdata type to manipulate C structs/objects e.g.

Lua is portable (written only in C89 – but C++ compatible)
It exists for Linux, OSX, Windows, Android, iOS and Lego (eLUa). Other implementations exist: LuaJIT, LLVM-Lua, MochaLua, Jill (Mobile)

Lua is small but powerful (such as perl, python, ruby, subject of this conf) – It supports regex
François then showed examples of factorial and fibonacci programs written with it.
It is very easy to embed with C programs.

Lua modules are heterogeneous (no CPAN), so build, doc, test and packages are multiple.
Lua has Modules such as sockets, POSIX, XML, DB, wxLua, GTK, JSON, Corba, …

François gave then some application examples: textadept (text editor, 2kLOC C + 6 kLOC Lua), Wireshark (added scripting support with Lua), awesome (Window Manager), redis (key/value tool, server side scripting with Lua)
Lua has too few sponsors: Adobe (Lightroom), SocialMediaPress, CSTUG, OCé.

François complained that there is not a lot of return to the community in term of code from these sponsors. But after all, my take on this is that this is certainly due to the choice of license (permissive) and of development model which doesn’t invite or force contributions to be brought back. One of the reason I chose the GPLv2 for my projects. Clearly it seems interesting if you need to add scripting support to a software. now, IMHO not done to develop applications directly.

Colin Charles (Monty Program AB)
Chief Evangelist on MariaDB (he left when Sun bought MySQL)

Charles gave 3 conferences during the day to paint n state of the Art of MySQL and its ecosysem.
He started by presenting the MySQL Diaspora

Charles started by some MySQL history.
The MysQL release policy is: alpha, beta and GA. 5.0 was released in 2005.
InnoDB was bought by Oracle in 2005, which created problems. The Falcon engine, which was targetted for 6.0, was the solution but it never saw the light.
5.1 was released in 2008. Lots of bugs were present due to an early release and lack of tests. 5.2 was canceled (It should have been open to community contribs)
5.4 was the next release in the roadmap in 2008.
In 2009, Sun was bought by Oracle. As a consequence 5.4 & 6.0 were canceled.
5.5 was realeased in 2010.
5.6 is available at labs.mysql.com (2011).

InnoDB + memcached in front of it, is the NoSQL approach taken (even if Oracle doesn’t believe in NoSQL, surprise, surprise !!)
Half of the downloads at mysql.com were for Windows. But most people on Linux, OSX, BSD use the packaged MySQL from their OS which thus reduces the number of downloads for these platforms and gives a wrong impression of the usage. Secondlife e.g. used MySQL out of Debian.
However, distributions tend to include only a small subset of MySQL features. There are only 5 engines compiled in Fedora/RHEL !

Charles underlined that the real uniqueness of MySQL is its ability to use multiple engines very easily (more on that in a dedicated conference below).
He detailed the commercial ecosystem around MySQL for add-ons (InfoBright, TokuDB, ScaleDB, Nitro, SolidDB, RitmarkFS) as well as HardWare vendors too (Kickfire, Virident, Schooner – going away with Oracle) as well as 3rd party engines (that Oracle doesn’t like either)

Charles then gave definition of terms: Fork == do not merge back (Drizzle e.g.) – Branch: rebase all the time (MariaDB, Percona e.g.). He then expained the different MySQL children.

• Percona started from 5.0 (it took Google/Facebook code on top of MySQL). So Percona is MySQL with InnoDB enhancements (coming from FaceBook/Google) and minimal server changes. They have 1000+ customers in the USA. They provide more fine grained diagnostics. (Google uses MySQL/Percona for Adwords). It provides FlashCache which makes it a great offering for SSD support. They also provide the Percona Toolikit (lots of perl scripts, it is the former maatkit UDFs) to make admin life easier. Their business model is mainly around consulting, and some support.
• Drizzle is based on MySQL 6.0 whiwh was never released. It took them 3 years to release the GA, just recently, aimed to be the RDBMs for OpenStack. However, due to the time it took, OpenSTack now uses SQL Alchemy which supports multiple RDBMs. Drizzle is Unix only – No Windows. Drizzle has a Micro-kernel architecture so everything is a plugin (including optimizer, query caches, …). Fully written in C++ now (no C anymore), it targets Cloud. It uses a lot of other FLOSS tools (including Google Protocol Buffer). Patches from MariaDB are difficult to integrate as it’s a fork.
• MariaDB started from 5.1, and is now with 5.5/5.6 also including Percona patches and abandonned patches. MariaDB is Percona and more storage engines and lots of improvements (including µs precision, extended stats for slow query log, threads pool support (eBay), pluggable authentication, virtual columns, NoSQL, subqueries (finally in 5.3 !!), GUI for Windows).
• Other branches are OurDelta (pet project – died), MepSQL (just Facebook patches, died), XAMPP (Windows)

For NOSQL in MySQL, use MariaDB ! Its HandlerSocket provides 700000 queries/s vs 400000 queries/s for standard MySQL. It also provides Dynamic Columns, memcached interface to InnoDB and NDB (only in the labs of MySQL, MariaDB is waiting for publication to include it – droped in 5.1, maybe back in 5.6/5.7)

MySQL cluster is an external product, which will probably be integrated in main tree with version 5.6/5.7.

From a support perspective, many companies can deliver it: Oracle (MySqL ent), Percona (MySQL, Percona, MariaDB, Drizzle), SkySQL (MySQL, MariaDB, Drizzle, distros), and many others.

Charles announced that pap.fr just issued a press release about their migration to MariaDB.

He then underlined the difficulty to work with Oracle: There is no public roadmap for MySQL. MySQL is now OpenCore: threads (connection) pool is closed source (original patches from eBay !) with only API for community dev. PAM (AD plugin) code came from MariaDB and is now also closed source. InnoDB Hot Backup however has always been closed source. But xtrabackup is FLOSS. InnoDB has never been Open Source, and they are rebasing from what is published monthly.
These problems comes from the contribution agreement that Oracle forces MySQL contributors to adopt to integrate patches upstream, which gives them the possibility to close the code instead of integrating it inside the public version. Oracle is once again making foes here.

From a performance perspective, XtraDB in 5.5 > InnoDB in 5.5 > XtraDB 5.1.
Connectors remains 100% the same whatever the backend.

Scaling is the issue. Hopefully, working with HP on this we can help Monty programs make progresses on this side.
To a question I asked around Monty Programs, Charles explained that Monty Program is only developing around MySQL. Their business model is to sell 3rd level support and dedicated consulting around development. They have no interface with ordinary customers.

The ecosystem is now in fact split between multiple actors, compared to when MySQL AB was doing everything: Monty Program as the “Lab”, SkySQL for the support and Percona for the consulting, even if the last two overlap a bit.

I was lucky to lunch with Charles, so we could discuss into more details around the community, and some of the points upper were not presented during the formal presentation but are the result of my understanding during our talk. All mistakes are of course mine

During the next session, I had a discussion with Charles Schulz, administrator of the LibreOffice.org foundation to get some news of this community, and they have lots of plans, some announces that will happen in October, but that I can’t reveal for now. Stay tuned however as it’ll be great !!

Then I attended the second session from Charles Colin around Storage Engines of MySQL/MariaDB

The main engines are: MyISAM, InnoDB, NDB (aka Cluster), Falcon (dead), Archive, Federated(X), Merge, Memory, and the one from Partners, Community, Customers…
The Engines list also includes AWS, OpenLDAP, CSV, Blackhole, Q4M (queuing mecanism), …

Charles then explained the default choices:
MySQL 5.5 => InnoDB by default, MyISAM for temp files
MariaDB 5.5 => XtraDB by default, Aria for temp files (more secure, as crash safe)
Percona 5.5 => XtraDB by default, MyISAM for temp files
Drizzle => HailDB (the only engine), MyISAM for temp files

The value proposition of MySQL/MariaDB around the various engines is unique. It brings flexibilty very easily, from ACID compliant InnoDB, to memory engine for performance, to archive which compresses data up to 80%
There are of course differences between engines with regards to storage (side note advise; use XFS), indexes, memory usage, transactions support (ACID), …
To see what engines are supported in your version, use show engines (12 present in MySQL, 10 in MariaDB)
Engines are plugins handled as loadable modules after 5.1.
There are some commercial engines: InfoBright (Data warehouse), TokuDB (cloud)
To use a specific engine with a table use: create/alter table … engine=XtraDB
Distributions tend to remove engines in their delivery (half only in Fedora/RHEL).

Indexes use all type of algorithms available: B+trees (InnoDB), Red-black trees (memory), R-trees (MyISAM) + hash indexes (memory, NDB, InnoDB).
For MySQL benchmarks, Charles recommended to use mysqlslap, to which I’d also tend to add the excellent HammerOra, which will allow you to make comparison with other RDBMs very easily !

Charles then explained the specifics of some engines:

• MyISAM: excellent insert perf, small footprint, full text support. No transactions, no foreign key. Usage: log, audit, Datawarehouse. Was preferred earlier. Not usable with DRBD. Tuning hint: Use key_cache_segments = 1 in MariaDB.
• InnoDB: for OLTP, raw disk support (useful for Windows). Tuning is also required. Use innodb_file_per_table good when locking, innodb_buffer_pool_size to memory*0.8 if dedicated. show engine innodb status gives more stats on MariaDB. InnoDB supports row & statement replication. Hint: Use a primary key (if not, it will be done for you)
• Archive: good for auditing (can’t delete in it, only SELECT and INSERT), uses libz
• Federated was disabled in MySQL but FederatedX has replaced it in MariaDB.
• FederatedX: Used by Cisco to store centrally from routers. Good for synchronous replication.
• Memory: loosing popularity in favour of memcache (BTW there is also a memcache engine !) Used for lookup, session, temp or calculation tables. Of course, when server goes down, rows are lost.
• Aria: MariaDB only. Planned to be competing with InnoDB and was concurrent to Falcon (now dead). Crash safe MyISAM, may become ACID.
• PBXT: ACID. Transactional. Good for SSD storage. Popular in Germany. Supports efficient BLOB. Should be more known.

To install a new engine just issue: INSTALL PLUGIN myengine SONAME ‘engine.so’

Charles reminded that a mix and match of engines is frequent.
He underlined that backup is engine dependant. LVM snapshots are a way to solve this.
Monitoring is also engine dependant.
Wordpress uses MyISAm, Wikipedia/mediawiki prefers InnoDB except search, SugarCRM uses MySQL.

Was time for me to take the mic ! and talk about
Continuous packaging with Project-Builder.org by Bruno Cornec (HP)

Brilliant presentation as usual Find it online and hopefully at OSDC.fr where the Vidéo should appear soon.

Open Build Service by Vincent Untz (SuSE)

I stayed in the room to listen to Vincent, who insisted on the new name of the former OpenSuSE Build system which is now the Open Build Service (Vincent, I’ve now patched my presentation to be aligned )

Even if goals are similar (building more easily packages and supporting a continuous packaging approach, there are still some differences I think between the tools. First, of course, language used to develop them (perl for Project-builder.org and Ruby on Rails for obs), but also on pb side the willingness to share metadata across packages, in a VCS and providing a macro system to support it, the support of VMs *and* Virtual Environment* and now *Remote Machines*, with tools such as rpmbootstrap to help users managing them, the standalone aspect of the tool and it’s ease of deployment (just 3 perl packages + 2/3 deps max – fewer LOC). Also pb supports gentoo, and Solaris, and is very easy to extend as everything is now managed through configuration files.

Advantages on obs side are its great Web based interface, its reporting mecanisms, and its capability to rebuild projects based on modifications of other projects. obs announces a very large user base, and packages built, even if when you divide by the number of distro tuples it’s less of course. pb just supports 5 different projects so has a much less reduced installed base.

Vincent did a very good presentation, as usual, showing also how it had a positive impact on the OpenSuSE project by having all the build infrastructure rebuilt to be Open Source (GPL) in order to support the project. Another big plus of their infrastructure is their BaaS approach (Build system as a Service), whereas with pb you need to create it, even if it’s easy. However, Vincent underlined that OBS as BaaS is only accessible for Open SOurce projects. If you’re building other software types, then we are at parity as you need to deploy the tool in-house.

As I underlined at the end of my talk before welcoming Vincent, whatever the tool you use, what is the most important, is to promote continuous packaging, as a good practice of today for projects (Open Source or not). Everybody’s new mantra should become “package early, package always” !

Charles then made his last presentation of the day (busy guy !) on MariaDB.

He reminded that it is a branch of MySQL, community developped, feature enhanced and backward compatible.
Monty Program is the main sponsor of MariaDB (58% of patches come from Monty Program, 42% from commmunity members having commit access)

MySQL is now an Open Source/Core product not a project.
MariaDB is an Open Source project, with the goal to be 100% compatible and a drop-in replacement of MySQL.
It is stable (production ready) and GPLv2. XtraDB (vs InnoDB) is enabled by default.
Charles then covered the various versions published:

• 5.1: Spend lot of time on build system (Buildbot + VMs for 5.1. No compiler warnings (contributed back). Table Elimination (aligned with big RDBMs) improves perf. Pool of threads (or connection pool, good for short running queries).
• 5.2: Pluggable authentication. Easy to write in 1/2 days AD auth/PAM auth as Oracle published recently. Virtual columns. Includes ShpinxSE (Full Text Search Engine)
• 5.3: NoSQL (HandlerSocket – direct access to InnoDB in perl DeNA, Dynamic columns, Group Commit, Replication, mytop, optimizer, optimized subqueries). That code may not be given back to Oracle due to Oracle recent announcement.

Everything done in the open, ML, bugs, IRC, code, worklog, KB (1400 articles in 1 year, cc-by-sa) …

Future: GIS. Merging takes more and more time. Future may be to fork to solve the Open COre MySQL approach, which would be IMHO good news for the FLOSS world, as it was sith LibreOffice e.g.

Blocks in perl6 by Christian Aperghis-Tramoni

A real course on how perl6 manages some blocks (functions where particularly detailed). Not a lot of notes, here as everything is online, and this was mostly code explanations, so I prefered to follow, rather than taking useless notes that nobody would read anyway ! What I mostly retained is that perl 6 brings some huge differences with regards to perl 5, certainly in the good direction, but will need an adaptation from old perlers such as myself, not following closely perl 6 development as Christian, and also that it’s not yet ready for usage, as lots of features are not there yet, and no optimisation is available, so from a production perspective, it’s unusable now. Perl 5 has for sure a very long life in front of him

All in all, OWF 2011 was again a great event. However, I would prefer for next year that they concentrate mostly on the Think part, which is quite unique to this conference and makes all its value, whereas code and experiment are already well represented at RMLL or Fosdem. See you there next year hopefully !

Note: Some of my photos related to this event are now available at https://picasaweb.google.com/112434061686721373729/OWF2011