Monday, July 20, 2015

Using Snappy Ubuntu Core to test-boot anything

Lab-As-A-Service Inception OS

Today morning I realized that it is quite possible to use chain-loading to boot into a test OS from within a Snappy Ubuntu Core system. I've decided to try to implement that and you can give it a try now. LAAS inception 0.1 [snap].

Update: I uploaded an older, broken version, please make sure you have this version installed:
a1bc56b7bc114daf2bfac3f8f5345b84  laas-inception_0.1_all.snap

Inception OS is a small Ubuntu Snappy Core-based system with a one more snap for programmatic control over boot process. Using the inception OS, any x86 or amd64 system (both in UEFI and legacy bios mode) can be converted into a remotely controlled web API that lets anyone reflash the OS and reboot into the fresh image.

The system always reboots into the Inception OS so this can be used to run unreliable software as long as the machine can be power-cycled remotely (which seems to be a solved problem with off-the-shelf networked power strips).

Installing the Inception OS

  1. Get a laptop, desktop or server that you can run Snappy Ubuntu Core on enough so that you have working networking (including wifi if that is what you wish to use) and working storage (so that you can see the primary disk. In general, the pre-built Snappy Ubuntu Core for amd64 can be used on many machines without any modification.
  2. Copy the image to a separate boot device. This can be a USB hard drive or flash memory of some sort. In my case I just dd'ed the uncompressed image to a 8GB flash drive (the image was 4GB but that's okay).
  3. Plug the USB device into your test device.
  4. In the boot loader, set the device to always boot from the USB device you just plugged in. Save the setting and reboot to test this.
  5. Use snappy-remote --url=ssh:// install-remote laas-inception*.snap to install the Inception snap.
  6. SSH to the test system and ensure that laas-inception.laas-inception command exists. Run it with the install argument. This will modify the boot configuration to make sure that the inception features are available.
  7. At this stage, the device can be power-cycled, rebooted, etc. Thanks to Snappy Ubuntu Core it should be resilient to many types of damage that can be caused by rebooting at the wrong moment.

Installing Test OSes

To install any new OS image for testing follow those steps.
  1. Get a pre-installed system image. This is perfect for snappy (snappy-device-install core can be used to create one) and many traditional operating systems can be converted to pre-installed images one can just copy to the hard drive directly.
  2. Use ssh to connect to the Inception OS. From there you can download and copy the OS image onto the primary storage (hard drive or SSD) of the machine. Currently this is not implemented but later versions of the Inception OS will simply offer this as a web API with simple tools for remote administration from any platform.
  3. Use the laas-inception.laas-inception boot command to reboot into the test OS. This will restart the machine and boot from the primary storage exactly once. As soon as the machine restarts or is power cycled you will regain control as inception OS will boot again.

How it works

Inception OS is pretty simple. It uses GRUB chain-loading to boot anything that is installed on the primary storage. It uses a few tricks to set everything in motion but the general idea is simple enough that it should work on any machine that can be booted with GRUB. The target OS can be non-Linux (inception can boot Windows, for example, though reboots will kick back into the Inception OS).

Monday, July 6, 2015

Backporting python-launchpadlib3 to Ubuntu 14.04

tl;dr; You can now use python 3 version of launchpadlib from ppa:zyga/launchpadlib-backports. Read on if you want to find out how I did it.

Today a colleague of mine asked for a small hint on how to do something with launchpadlib. I asked for the code sample and immediately stopped, seeing this is python 2 code. As python 2 is really not the way to start anything new in Ubuntu nowadays I looked at what's stopping us from using python 3.

It turns out, my colleague was using the LTS release (14.04) and python 3 version of launchpadlib just wasn't available at that time. Bummer.

Having a quick look at the dependencies I decided to help everyone out and create a PPA with the backported packages. Since this is a common process I though I would share my approach to both let others know and give more knowledgeable Debian developers a chance to correct me if I'm wrong.

The whole process starts with getting the source of the package you want to build. I wanted to get the latest and greatest packages so I grabbed the source package from Ubuntu 15.10 (wily). To do that I just go to and search for the right package. Here I wanted the python3-launchpadlib package. On the right-hand-side you can see the link to the .dsc file. You want that link so copy it now.

The right way to download each Debian source package is to use dget. Using a temporary directory as a workspace execute this command (if you read this later, the source package may not be available any more, you'd have to adjust the version numbers to reproduce the process).


With that package unpacked you want to change into the sub-directory with the unpackaged code. At this stage, you need to have a sbuild for Ubuntu 14.04. If you don't have one, it's time to make one now. You want to follow the excellent article on the Debian wiki for this. Many parts are just copy-paste but the final command you need to run is this:

sudo sbuild-createchroot --make-sbuild-tarball=/var/lib/sbuild/trusty-amd64.tar.gz trusty `mktemp -d`

Note that you cannot _just_ run it as there are some new groups you have to add and have available. Go read the article for that.

So with the sbuild ready to build our trusty packages, let's see what we get. Note, that in general, the process involves just those four steps.

  1. sbuild -d trusty
  2. dch # edit the changelog
  3. dpkg-buildpackage -S -sa
  4. dput ppa:zyga/launchpadlib-backports ../*.source.changes
After doing step one you'll see that you have missing build dependencies. Those are python-keyring, lazr.restfulclient, python-oauth and python-wadllib. We need to backport those too!

At this time the process recursively continues. You grab a .dsc file with dget it, and try to sbuild it right there. Luckily, you will find that nothing here has more dependencies and that each of the four packages builds cleanly.

At this time, you want to create a new PPA. Just go to your launchpad profile page and look for the link to create it. The PPA will serve as a buffer for all the packages so that we can finally build the package we are after. Without the PPA we'd have to build a local apt repository which is just mildly more difficult and not needed since we want to share our packages anyway.

With the PPA in place you can now start preparing each package for upload. As a habit I bump the version number and change the target distribution version from wily / unstable to trusty. I also add a changelog entry that explains this is a backport and mentions the name of the PPA. The version number is a bit more tricky. You want your packages to be different from any packages in the Ubuntu archive so that eventual upgrades work okay. The way I do that is to use a (x-).1 version which is always smaller than the corresponding x Ubuntu version. Let's see how this works for each of the packages we have here.
  • lazr.restfulclient has the Debian version 0.13.4-5 which I change to 0.13.4-5ubuntu0.1. This way both Ubuntu can upload 0.13.4-5ubuntu1 and Debian can upload 0.13.4-6 and users will get the correct update, nice.
  • python-keyring has the Ubuntu version 4.0-1ubuntu1 which I change to 4.0-1ubuntu1.1 so that the subsequent 4.0-1ubuntu2 can be uploaded to Ubuntu without any conflicts.
  • python-oauth has the Debian version 1.0.1-4 which I change to 1.0.1-4ubuntu0.1 to let Ubuntu update to -ubuntu1 eventually, if needed.
  • python-wadllib has the Debian version 1.3.2-3 which I change to 1.3.2-3ubuntu0.1 in exactly the same way.
Have a look at an example changelog to get a feel of how this all works together.

Now all the dependencies are ready and I can do the final test build of launchpadlib itself. Since I always test-build everything I will now need to expose access to my PPA so that my sbuild (which knows nothing about it otherwise) can get the missing packages and build everything. This is the magic line that does it:

sbuild -d trusty --chroot-setup-commands='apt-key adv --keyserver --recv-key E62E6AAB' --extra-repository "deb trusty main"

Here, we have two new arguments to sbuild. First, we use --chroot-setup-commands to import the public key that launchpad is using to sign packages in my archive. Note that the key identifier is unique to my PPA (and probably to my launchpad account). You want to check the key listed on the PPA page you got. The second argument --extra-repository just makes our PPA visible to the apt installation inside the chroot so that we can resolve all the dependencies. On more recent versions of Ubuntu you can also use [trusted=yes] suffix but this doesn't work for Ubuntu 14.04.

After all the uploads are done you should wait and see that all the packages are built and published. This is clearly visible in the "package details" link on the PPA. If you see a spinning stopwatch then the package is building. If you see a green cogwheel then the package has built but is not yet published into the PPA (those are separate steps, like make and make install, kind of). When all the packages are ready I copied all the binary packages (without rebuilding) from trusty to utopic, vivid and wily so that anyone can use the PPA. The wily copy is a bit useless but it should let users use the same instructions even if they don't need anything, without getting weird errors they might not understand.

So there you have it. The whole process took around an hour and a half (including writing this blog post). Most of the time was spent waiting on particular test builds and on the launchpad package publisher.

If you have any comments, hints or suggestions please leave them in the commends section below. Thanks.