Posts Tagged ‘ProLiant’

When your CentOS 7 install fails, anaconda in CLI to the rescue

2018/03/08

Ok, I know I’m always trying to do stuff a bit non-standard (No, running Mageia everywhere IS standard !!)

So this time, I wanted to update my CentOS 6 deployment server in Geneva to CentOS 7 and automate its setup with ansible (I already have some playbooks here to help).

The server is in a DMZ, with pretty strict security control, which normally isn’t an issue, but when eveything else fails…

Ok, so first yum update on that machine didn’t work. Lot’s of issues with that. Most of the messages you find on the net recommend to install another server in parallel. Fine, but this is not a VM, but a real machine (HPE ProLiant DL 180 Gen9) and I don’t have another one ! So I tried to update some packages manually. To make a long story short, glibc updated, but after that nothing worked as you enter in a nightmare with incorrect paths (they put everything under /usr in CentOS 7, so all /bin or /sbin binaries were not working anymore). Ok, so I had to reinstall the machine as no other method could rescue the situation, hopefully keeping my /prj FS safe.

So you use the marvelous Virtual Media feature of your server in order to mount a local ISO image on your system as a local CD of your remotely managed machine. So you download your CentOS 7 DVD latest version and try that. But it doesn’t work. Once again systemd fails to do something (logind launch in that case), and after that nothing works 😦 Did I say already I hate systemd ? And even more the people behind it who think they are always right because it just works for them 😦

After trying multiple things, I found out that CentOS 7.3 was going further on that system, so worked with that. Further meaning I was able to get a shell, but the graphical interface never wanted to launch. And using inst.vnc is not an option due to firewalling restrictions I can’t change on my end. Using inst.sshd was one way to have a working remote access. So I used that and tried to launch anaconda manually.

I used the text mode version, but it wasn’t able to reuse my existing partioning schema, as I found out. Yes, because I’m picky (I know !) on partitioning schema, and that machine had to use md RAID devices in parallel with normal partitions, and LVM on top + ext4 FS as that’s what I want (why do you want to use Linux, if the system dictates to you what to use !! then use Windows instead :-). And I wanted to keep some existing partitions safe. But launching anaconda manually didn’t take in account my existing partitioning schema (or I didn’t find how, in text mode you don’t get much help) so I lost everything on the server, without any warning concerning partitions 😦 So back to backups then.

Ok so now I really have a new install to make, but still with the same partitioning schema. So I decided to create a kickstart for the machine, which would contain that partitioning schema so anaconda could use it (and get the rest of the conf at the same time).

The partitioning shema is as follow (nothing fancy, but nos completely standard either):

part /boot/efi --fstype=efi --ondisk=sda --size=200
part /boot --fstype=ext4 --ondisk=sda --size=500
part /tmp --fstype=ext4 --ondisk=sdb --size=700
part raid.01 --ondisk=sda --grow --size=200
part raid.02 --ondisk=sdb --grow --size=200

raid pv.01 --fstype ext4 --device pv.01 --level=RAID1 raid.01 raid.02
volgroup vg_dl180 --pesize=4096 pv.01
logvol / --fstype=ext4 --name=lv_root --vgname=vg_dl180 --size=40000
logvol /var --fstype=ext4 --name=lv_var --vgname=vg_dl180 --size=10000
logvol swap --name=lv_swap --vgname=vg_dl180 --size=4000
logvol /prj --fstype=ext4 --name=lv_prj --vgname=vg_dl180 --size=520 --grow

Now I don’t want to burn a new DVD to put the ks file on it, so I thought, let’s use the network.

But the instructions which tell you that you can use an instruction to set up the network statically (I have no DHCP server as this server will become the DHCP server of this LAN !) on your system forget one point. If you use ip=IPaddr::GW:NM:… and also inst.ks=http://somewhere/x.ks then the IP is put in DHCP mode automagically :-((

Ok, so now looking at the boot logs, I see that it tries to download the ks file, but can’t because it didn’t get the IP address through DHCP which is why I wanted to pass a static IP conf to it !!!

So you can’t use the inst.ks option, if you need static IP. In fact this is written but in the ip documentation. So let’s do that. You boot again (on my ADSL connection, it’s 30′ each time you try something so we’re 2 days after the start of course) and get to a shell with the IP setup correctly, but no ks file.

And I don’t have anything on that LAN of course, to share with http or NFS. So let’s find a way. Use https://file.io ! You can upload there a text file like the ks file, and get a URL so you can download it with wget in your shell ! (you need to setup the nameserver with the nameserver= option at boot time, or adapt your /etc/resolv.conf locally).

So now I have an IP conf, a local repo (seen as a CD) and the ks file !! I just now need to find a way to say to anaconda to use it. The useful doc can be found on the Fedora Wiki. Ok, not very detailed, but enough for me to try !

I needed to launch it with:
anaconda --text --kickstartfile=file.ks

I had to interactively change the source repo, as I had given a URL in the ks file, but for a reason it didn’t seem to work, so I pointed to the local CD instead (slower as the Virtual media was getting the content of CentOS through my ADSL link, and in that sense, it’s slow ! Luckily I only had 300 packages to install). I also had to setup a simple account (just in case)

And then it worked ! it took time, but worked ! It built my partition schema the way I wanted, and installed everything, including the boot loader correctly (was a UEFI conf).

So next time I’ll be stuck by anaconda not starting graphically, I know what to do, and hopefully you too 🙂

Next step is to automate its configuration, and capture that so I can present it during the upcoming HPE TSS event in Den Haag !

Advertisements

Time to Meet at Hewlett Packard Enterprise Technology & Solutions Summit 2017 !

2017/01/24

It’s time again for me to prepare the HPE TSS designed for our presales and channel partners, with some of our the best experts to transfer knowledge on everything HPE !

We have again this year a track dedicated to Open Source and Linux !).

So I’ll be in Cannes from the 6th to the 10th of February to cover:

  • B90 HPE’s Open Source Strategy – Auditorium K – Tuesday 7th of February 2017 – 5:30PM-6:20PM
  • L30-R1 Docker 101 Tutorial – Lab 1 – Wednesday 8th of February 2017 – 8:30AM-10:20AM
  • B91 Docker @ HPE – Ambassadeurs B – Wednesday 8th of February 2017 – 12:00AM-12:50AM
  • L31 Docker Orchestration Tutorial – Lab 1 – Thursday 9th of February 2017 – 8:30AM-10:20AM
  • B94 Docker: technology aspects and tools – Redaction 1 – Thursday 9th of February 2017 – 3:00PM-3:50PM
  • L30-R2 Docker 101 Tutorial – Lab 3 – Thursday 9th of February 2017 – 4:30PM-6:20PM
  • B92 Linux Distributions roadmap – Redaction 3 – Friday 10th of February 2017 – 8:30AM-9:20AM
  • B6 HPE RESTful API and the Redfish standard – Espace Californie – Friday 10th of February 2017 – 12:00AM-12:50AM

Pretty busy week no ?! And you should be able to replay the labs/tutorials on Wednesday afternoon during the OpenLab session in Lab 1.

A lot of sessions are covering Docker related topics this year, and I’m happy with thatas it’s a great techno, even more with the recently announced 1.13 bringing service support to docker-compose !

For those of you wondering why project-builder.org and MondoRescue versions are not published more regularly, that gives you again a hint 😉 Not speaking of the travel for 2 weeks starting the week after this one, and the next event in March in Grenoble, TES, where I’m part of the Event team and that we’ll organize for the 8th time.

See you there to talk of these subjects or something else as you see fit. I’m around the whole week.

Network install of Debian 8 on a HPE ProLiant Gen9 server

2016/06/29

There are articles to do this in a pure Debian environment, such as the excellent one I used as a base from Raphaël Hertzog.

But my deployment server in that case is a CentOS 6 one, so I needed some adaptations to make it work.

My target system is a HPE ProLiant BL 460 Gen9, equiped with bnx2x NICs (Broadcom BCM 57840 10/20 Gb/s cards). And during a network boot install, I have had messages indicating that the firmware was missing (failed to load bnx2x/bnx2x-…)

So in order for my system to network boot, I needed to adapt my initrd and add the missing firmware in it (Debian’s policy prevents the distribution of
non-free firmware, which these are). The Firmware in my case is available as a standalone package at http://ftp.fr.debian.org/debian/pool/non-free/f/firmware-nonfree/firmware-bnx2x_0.43_all.deb (hint from this article)

So on your non-debian deployment server, you can do the following:

mkdir /tmp/fw
cd /tmp/fw
# This is the network boot environment required for booting Jessie
wget http://ftp.debian.org/debian/dists/jessie/main/installer-amd64/current/images/netboot/netboot.tar.gz
[...]
tar xvfz netboot.tar.gz
[...]
# Keep the kernel in your deployment infra
mv debian-installer/amd64/linux /where/your/tftpboot/k/debian-8
# Extract the initrd to modify it
wget http://ftp.debian.org/debian/pool/non-free/f/firmware-nonfree/firmware-bnx2x_0.43_all.deb
[...]
mkdir initrd
cd initrd
gzip -cd ../debian-installer/amd64/initrd.gz | cpio -ivdum
[...]
# You need the dpkg commdand to make your life easier (part of CentOS, Fedora or Mageia)
dpkg -x ../firmware-bnx2x_0.43_all.deb  .
ls lib/firmware/
[...]
# Should show you the firmware in the right place
find . -print | cpio -o -H newc | gzip -c9 > /where/your/tftpboot/i/debian-8.img
[...]

Now you can add an entry to your boot loader (grub2 in my case) for the network boot part (more info on Network boot and Debian at https://wiki.debian.org/PXEBootInstall)

menuentry 'Debian 8' {
  insmod gzio
  insmod part_gpt
  insmod ext2
  insmod iso9660
  linux /k/debian-8 ip=dhcp --
  initrd /i/debian-8.img
}

If you have another firmware to deal with, refer to the dedicated Debian page at https://wiki.debian.org/fr/firmware

After that your system should be able to network boot and install your Debian Jessie distribution on it.

The next step is to automate the installation, which consists just in changing the linux line in the grug conf file:

linux /k/debian-8 ip=dhcp auto priority=critical url=http://deploy-server/ks/debian-8 --

where the debian-8 file under the ks directory is just the preseed file you’ll build for your automatic installation.

python-redfish reached 0.3, first interesting version for others

2016/05/12

Back in 2015, during our internal TES event in Grenoble, a group of people gathered to talk about the new Redfish DMTF standard and how we could add its support into Ironic.

Since that, we have worked first on a new Open Source python library to interact programmatically with Redfish 1.0 based systems.

We tried to promote it during OpenStack Summits but except in Vancouver when I did a short video explaining that, not enough interest from the community. Of course we had very few to show at that time.

One year after, we have now reached version 0.3 of the project. And while that doesn’t sound a lot, in fact the libray is now usable by others to interact with Redfish based systems.

What it provides:

Our next steps are in multiple directions:

Hopefully, as you’re reading this, you’re interested by the topic, and we’d be happy to get your feedback, and more over your patches in order to improve our work, and speed up the development !

And don’t forget the HPE Open Source project available as well on github which provides much more capabilities, due to its support of the iLO and iLOCM extensions to the Redfish standard.

Everything you want to learn about UEFI will be discussed at LinuxCon EMEA

2015/09/07

During the upcoming LinuxCon Event HP’s VP and Fellow Dong Wei, Chief Executive of the UEFI Forum will host a UEFI mini-Summit the 7th of October (in Dublin) from 9:00AM to 6:00PM. This is free to attend and will cover “how to” guidance, implementation options, and firmware tools and resources available for successful ARM and x86 platform integration using the UEFI ecosystem. Details are available at http://events.linuxfoundation.org/events/linuxcon-europe/extend-the-experience/co-located-events

Feel free to come for this technical deep dive with experts from HP, Insyde, Intel which will help with your transformation to adopt that key HP ProLiant Gen9 technology.

Contact myself if you want to organize specific additional sessions (especially with HP customers) around UEFI or Redfish with Dong Wei. And feel free to drop me a mail if you’re around and want to talk of these topics, Linux on ProLiant, MondoRescue, or whatever you would find interesting to discuss with me while I’m attending the event.

I hope to meet you there.

Mondoescue 3.2.1 is now available … with effort and pain !

2015/08/31

Believe it or not, it has been one of the most challenging version to produce ! And one which was really awaited by lots of people.

First all those of you who had problems restoring big files with 3.2.0 which had issues with that. Now that’s solved (was already with the beta previously).

Then those of you who had a RHEL 5 system. 3.2.0 wasn’t working, and 3.2.1 should be much better. Similarly for the people using RHEL 7. This is the real first version working for RHEL 7 available of MondoRescue. I’ve made most of my tests with a RHEL 7 VM in order to track all the errors which were remaining for that env (also some smaller but annoying ones). The root cause is that I had a customer wanting to move to that version for RHEL 7, and I wanted them to be happy with the result. Doesn’t mean they won’t encounter issues, despite what I did, but I’m much more comfortable to handle the cases in their support contract (Yes this is a paying customer).

And finally those of you who were trying to have a working environment for a UEFI based machine with RHEL 6. UEFI wasn’t working at all with 3.2.0 (and previous versions) even if EFI was working on Itanium systems. So this time I used a real system to make again lots of different tests, and solving again lots of issues linked to the support of UEFI, and now I’m happy with the result. The media made is recognized by the UEFI shell or the HP ProLiant guided boot menu. And a full backup / restore works also on it. So that other customer (an HP one, but not a paying one from a MondoRescue perspective… yet) should as well be happy and be able to use MondoRescue to perform its clones and installs correctly.

Could it be better ? Of course. RHEL 7 + UEFI are not yet friends. That will be for 3.2.2, as I want to keep my job ;-). Few bugs in trac have been addressed during these last 8 months, sorry for that. But a lot not mentioned there have been solved, and that was most of the work. And a lot of work has been put (commit hours are real !) into solving the customers needs and deliver that version on time. That’s probably why I hate time based releases, as it forces you to deliver (which is always good), but not with the serenity which should be part of an OpenSource development. Of course, that’s my fault ! I should have anticipated these requests (UEFI and RHEL 7 are there since a long time). But hey, this is still not my day job at HP. So I can only do my best as time permits. And I have a lot of other hobbies that I’ll be back to now so they can also know some progress. That’s also why I do not make more regular releases. But patches are always welcome.

Anyway, pain is over, a new version is here. Test it, report feedback, try to not break it too much, and remember to do regular backups whatever the tools you use.

Download at

Redfish 1.0 is now out and soon used by our projects

2015/08/20

Recently the DMTF announced that the Redfish standard had reached version 1.0, the first usable for production.

If you don’t know Redfish yet, and don’t want to read the full spec, I can summarize it very roughly as a cross-manufacturer RESTful API to configure Hardware systems and recover info from them. Pretty cool in fact as this was a missing brick for a full Open Source based multi-platform Software Defined Infrasctrucute.

Now remains for us to update our python-redfish with it, and start supporting more features now it’s stable. That the first step will allow Ironic, the bare metal component of OpenStack, to have Redfish support, which is IMHO a very highly desirable feature. Not for Liberty, hopefully the spec will be ready for M and code sufficiently developed to allow us to win a slot in the Summit !

Then it will also allow us to advance on the other project we’d like to promote: Alexandria, which is a clever (of course, I’m part of the design team ;-)) way to interface CMDBs (well the one who are themselves clever enough to have a parametrized data model such as iTop) and Hardware information providers such as Redfish.

Imagine that you rack a new Refish compliant server (HP ProLiant Gen9 are among the first to provide already a subset of it), cable power and management board, integrate some info on that machine in your CMDB (iLO credntial and IP address e.g.), and ask through Alexandria that the rest is filled by Redfish calls. And kept updated as time passes. Well, that’s exactly what we try to achieve, as this is one of the need we have to maintain our Solution Center CMDB up to date as easily as possible.

Lots of good ideas, but not enough code to show for now. Hopefully, we should be able to have an Alexandria demo with somple drivers ready for the upcoming OpenStack Summit… and be able to travel to exchange our views with other OpenStacker around all this.

Time to meet again at HP ExpertOne Technology & Solutions Summit !

2015/06/12

For those of you working for an HP value added reseller (VAR) and attending the HP ExpertOne Technology & Solutions Summit in Dublin from the 16thth to the 19th of June, feel free to meet during one of the session I’ll (co-)deliver:

  • UEFI: what, why, when, how and the rest Breakout 49 – Tuesday 16th of June 2015 – 5:45PM-6:35PM
  • HP Helion OpenStack in action – Breakout 175 – Wednesday 17th of June 2015 – 8:30AM-9:20AM
  • UEFI Hands-On Labs (François Donzé) – Lab 4 – Wednesday 17th of June 2015 – 11:00AM-12:50AM
  • HP ProLiant Gen9 value add on Linux (Hands-on Lab) – Lab 16 – Thursday 18th of June 2015 – 2:00PM-3:50PM
  • HP REST API for iLO Lab (François Donzé) – Lab 6 – Thursday 18th of June 2015 – 4:30PM-6:20PM
  • HP ProLiant Gen9 value add on Linux – Breakout 48 – Friday 19th of June 2015 – 12:00AM-12:50AM

And you should be able to meet the Geeks on Wednesday afternoon, myself included.

For those of you wondering why project-builder.org and MondoRescue versions are not published more regularly, that gives you a hint 😉 Even if I made some progresses recently I also need to document !

See you there to talk of these subjects or something else as you see fit. I’m around the whole week.

Software Defined Compute: Redfish and Ironic at the next OpenStack Summit

2015/04/24

The future is already here !

In fact here ! This is a new specification which is gaining traction to allow hardware configuration of systems through a RESTful interface. All new HP ProLiant Gen9 servers from HP are providing a first version of what is not yet the standard, but looks very near to it, and provide already a rich interface.

This is pretty exciting, as it will allow people like me working on Disaster Recovery e.g., or people like Devananda van der Veen who is the Ironic PTL (OpenSTack bare metal deployment component) or many others to have, when it’s ready, a single interface to configure the servers from different manufacturers following the Redfish Specification (as of now HP, Dell, Emerson, Intel). And it’s much more feature rich than IPMI, so you would be able to go well farer with it and reach a real Software Defined Compute node.

In order to make it usable for Ironic we’ve started to work on a low level python library (python-redfish) on which we’ll be able to build the Ironic driver for Redfish later on. This is work for Liberty and after of course. And as Ironic can be also use alone, I’m more and more thinking of using it as a way to provide MondoRescue images for bare metal deployment as an additional feature over media boot. Especially if there are plans to have a standalone web interface for Ironic !

If you want to learn more on the HP implementation of the RESTful API, there is a webinar organized Wednesday, April 29, 2015 from 10:00 am CDT to 11:00 am CDT. Register from https://h71044.www7.hp.com/campaigns/2013/events/linkandcoordinate/index.php

And if you want to learn a bit more of where we are on this, read the code (easy for now) or come at Vancouver for the upcoming OpenStack Summit to discover it.

Time to Meet at HP ExpertOne Technology & Solutions Summit !

2015/01/21

For those of you working for an HP value added reseller (VAR) and attending the HP ExpertOne Technology & Solutions Summit on Barcelona from the 2nd to the 6th of February, feel free to meet during one of the session I’ll (co-)deliver:

  • UEFI: what, why, when, how and the rest Breakout 101 – Wed 2014-02-04 9:30-10:20 AM
  • HP Helion OpenStack in action – Breakout 124 – Wed 2014-02-04 2:00-2:50PM
  • HP ProLiant Gen9 value add on Linux – Breakout 99 – Thu 2014-02-05 9:30-10:20 AM
  • HP ProLiant Gen9 value add on Linux (Hands-on Lab) – Lab 16 – Thu 2014-02-05 2:00-3:50 PM
  • UEFI Hands-On Labs – Lab 19 – Thu 2014-02-05 4:30-6:20 PM
  • Running Enterprise Linux distributions on HP ProLiant Gen9 – Breakout 100 – Fri 2014-02-06 9:30-10:20 AM

And you should be able to meet the Geeks on Wednesday afternoon, myself included.

For those of you wondering why project-builder.org and MondoRescue versions are not published more regularly, that gives you a hint 😉 Not speaking of the next event in March, TES, where I’m part of the Event team this time.

See you there to talk of these subjects or something else as you see fit. I’m around the whole week.