Friday, April 11, 2014

Checkbox challenges for 2015

Having a less packed day for the first time in a few weeks I was thinking about the next steps for the Checkbox project. There are a few separate big tasks that I think should happen over the next 6-18 months.

First of all, our large collection of tests needs maintenance. We need to keep adapting it to changing requirements and new hardware. We need to fix bugs and make it more robust. We also need to add some level of polish to the user interface. To make sure all our test programs are behaving in an uniform way, use correct wording, can be localized, etc. Those are all important to keep the project healthy. We also have a big challenge ahead of us, with the whole touch world entering the Ubuntu ecosystem. We will have to revisit some decisions, decide which libraries, tools and layers to use to test certain features and make sure we don't leave anything behind. This is very challenging as we really have a lot of existing tests. We also need to make them work the same way regardless of how they are started (classic Ubuntu, touch Ubuntu, remote Ubuntu server).

The core tools got an amazing boost over the past 12 months. Starting from pretty old technology that was very flexible but hard to understand and modify to something that is probably just as flexible but far easier to understand and work with. Still, it's not all roses. The Ubuntu SDK UI needs a lot of work to get right. It has usability issues, it has architecture design issues. We also have a big disconnect between the core technology (python3) used by and Qt+QML C++ codebase, talking over D-Bus with the rest of the stack. That brings friction and is 10x harder to modify than an all-python solution. Ideally we'd like to switch to PyQt but how that fares with the future Touch world is hard to say. I suspect that our remote testing story will help us have a smooth transition that won't compromise our existing effort and equally won't collide with the direction set by the first Ubuntu touch release.

Perhaps not in the spotlight but definitely we need to work on "whitelists" (aka test plans). We need to learn how our users take our stack and remix it to solve their problems. Our test plan technology is ancient and shows its weaknesses. We need a 2.0 test plans that allow us to express the problems we need to solve clearly, unambiguously and efficiently. We need to improve our per-device-instance test support. We need to provide rich meta-data for user interfaces. We need better vocabulary to create true test plans that can react to results in a way unconstrained by the design of the legacy checkbox first written over seven years ago. We also need to execute those changes in a way that has no flag days or burnt bridges. Nobody likes to build on moving sand and we're here to provide a solid foundation for other teams at Canonical and everyone in the free software ecosystem.

Lastly we have the elephant in the room called deployment. Checkbox doesn't by itself handle deploying system images and configuration onto bare metal (we have a very old and support project for doing that) and the metal is changing very rapidly. Severs are quite unlike desktops, laptops (Ethernet-less ultrabooks?) and most importantly tablets and the whole touch-device ecosystem behind them. In the next 12 months we need a very good story and a solid plan on how to execute the transition from what we have now onto something that keeps us going for the next few years, at least. Canonical luckily has such a project already, MAAS. MAAS was envisioned for big iron hardware but if you look at it from our point of view we really want to have uniform API for all hardware. From that big-ass server in a Data Centre somewhere across the globe to that development board on your desk, which will be the next tablet or phone product. We want to do the same set of operations on all of the devices in this spectrum, manage, control, track, re-image. The means and technology to do that differ widely and from experience I can tell you this is a zoo with all the queer animals you can think of but I'm confident we can make it work.

So there you have it. Checkbox over the next 12+ months, as seen through my eyes.