Analyzing Lantern Submissions

I wasn't working on Lantern today much (there are no new features yet) but I want to say thank you to everyone that has sent the second version of the test. I realize that it's more complicated to prepare and that it does include manual components but it is invaluable.

You can see all of the submissions in the linked repository. They are all JSON files so it should be simple to process in any language. There is no schema yet (now that I realized this I will probably write one) but it's rather simple to follow and understand what's inside.

The only part that is non-obvious is treatment of attachments. JSON is defined as all-unicode so to be able to preserve any binary blob that we may have to (a screen-shot, some random binary or mostly-text-with-GARBAGE-in-the-middle) we just base64-encode them.

Each JSON file (example) has a few top-level elements: results, resources and attachments. Plainbox, which generates the file, can do a bit more but that's all that we need in Lantern. Looking at the result map we can see that keys are just test identifiers and values are little structures with a few bits of data. The most interesting part there is the outcome which encodes if the test passed, failed or was skipped.

There are two kinds of tests that are interesting in the v2 lantern submissions. The first looks like 2015.pl.zygoon.lantern::test/intel_backlight/software-control note that the intel_backligth part is variable and can be acpi_video0 or nv_backlight or anything else really. This test checks if the software control is working through that device. The second test, which is only executed if software control works, is 2015.pl.zygoon.lantern::test/intel_backlight/brightness-0-is-visible. This test checks if setting brightness to zero actually turns the panel off.

Now I wrote this second batch of Lantern tests to check a theory:

firmware-based brightness control device, one that is based on /sys/class/backlight/*/type being equal to firmware, keeps the panel dim but lit when brightness zero is requested a raw driver will happily turn the panel backlight off

We now have the first essential piece of he puzzle, we know if the panel was turned off or not. The only missing bit is to know what kind of backlight control device we had, raw, firmware or platform. This data is saved in two places. The most natural way to access it is to look at a resource. Resources are a plainbox concept of allowing tests to generate structured data and keep this data inside the testing session. Plainbox uses this to probe the system and later on determine which tests to run. In Lantern we use this in a few places (1, 2). Since we also store this and the data is structured and easy to access we can simply look at it there. The interesting job identifier is 2015.pl.zygoon.lantern::probe/backlight_device. It can be found in the resource_map element and it is always an array. Each element is an object with fields defined by the resource job itself. Here it has the sysfs_type field which is exactly what we wanted to know.

So how to analyze a v2 submissions? Simple:

1. Load each JSON file and then
2. For each backlight device listed in resource_map["2015.pl.zygoon.lantern::probe/backlight_device"]
3. Memorize ["sysfs_type"] and ["name"].
4. If sysfs_type is equal to "firmware" then look at result_map["2015.pl.zygoon.lantern::test/" + name + "/brightness-0-is-visible"]["outcome"] to see if it is "pass".
5. If sysfs_type is equal to "raw" then look at result_map["2015.pl.zygoon.lantern::test/" + name + "/brightness-0-is-visible"]["outcome"] to see if it is "fail".
6. Each device that matches points 4 or 5 confirms out theory.

Note: I don't know much about platform devices yet so we can filter them out for now. Also note that we won't have the brigthness-0-is-visible entry at all if software-control test (which is a dependency) has failed.

I preferred to write this description rather than the actual script to familiarize everyone with the Plainbox result format. It's a simple and intuitive format to store all kinds of test results. If you want to contribute and write this analysis script just fork the code on github and start hacking. It should be simple and I will gladly mentor anyone that wants to start coding or wants to contribute something simple to the project.

Lantern update

Hey

Lantern is progressing nicely today. I've received a number of submissions and it seems that brightness ranges are all over the place. Thanks to everyone that has contributed. This data will be very useful for analysis and the more you can send, the better.

I wanted to test a theory, that firmware-based brightness control device, one that is based on /sys/class/backlight/*/type being equal to firmware, keeps the panel dim but lit when brightness zero is requested a raw driver will happily turn the panel backlight off. This is a common issue with many laptops. There is no consistent behavior. Users don't know what to expect when they hit the brightness control all the way down.

To test that theory I've created a test provider for plainbox (which is the project I'm hacking at work most of the time). It's a big change from a tiny script that fits on one's screen to a large (it's large!) body of code feeding of a pretty big set of data files and scripts.

I did this to ensure that the solution is scalable. We can now do interactive tests, we can use i18n, we can do lots of complicated things that are useful as we expand the library of tests. Using a toolkit simply helps us along the way in ways that simple shell scripts cannot hope to.

Currently I've added two interactive tests:

• test that checks if software brightness control works at all
• the brightness zero test I've outlined above

There's also a number of supporting jobs that detect and enumerate devices, collect logs and put this all together. You can see all of the details here.

So once again, I'd like to ask for your help. Look at the instructions and run the test suite. It takes about a minute on my laptop. Less if you already know all the instructions and don't have to follow along. As before, send submissions to my email  address at zygmunt.krynicki<at>canonical.com. If you come across any problems please report them.

You can also contribute translations (see the po/ directory for the familiar stuff), tests and discussions.

Thanks
ZK

Is max_brigthness coming from a random number generator?

First off, thank you for sending contributions to Lantern. Please keep them coming, we really need more data for meaningful statistics.

Now for the main dish: I wonder what's the cause of the seemingly random values of /sys/class/backlight/*/max_brightness as seen in the wild.

Currently I have instances of each of those:

[7, 10, 15, 89, 100, 255, 312, 494, 825, 852, 937, 976, 1808, 2632, 3828, 4437, 4438, 4882]

So one laptop has 7 steps of backlight intensity, another has 312, 976 and some have 4882. What is the cause of such a wide range of values? Can the hardware be the cause? But then again, are engineers that built this really so careful to expose, say 852 values instead of 825, or 100.

Given how most backlight control from user's point of view works, apart from smooth transitions that some Windows 8 laptops do, 25 would be way more than enough values.

If anyone has some input on what may be causing this, I'd love to hear that. I'll start working on improving the analysis script to do some correlation between the CPU type, GPU type and observed values.

Lantern

So you've probably seen my earlier post asking for data contributions.

This quick post is a follow-up to that, to say Thank You to everyone that contributed data, shared my post or replied with useful feedback

This is just the beginning of the nascent project I've called Lantern. I have a few extra tools in the works and I will describe them properly when I'm ready. The culmination of this process will be an attempt to determine:

• If the kernel interface works
• If brigthness==0 is "dim but visible" or totally off
• If brigthness control via hardware keys is reflected in what the kernel sees
• If brigthness control via software is confusing the firmware (when manipulated by the hardware keys)
• If X and vt* are behaving differently (most of the time that is the case)

In the end the results might impact the Ubuntu certification process (which is what I do at Canonical). I will share all the tools I've made and data I've collected.

Crowdsourcing help needed! Share your /sys/class/backlight please!

I'm working on a little mini-project that deals with back-light.
I've read the kernel documentation I could find [1], [2], [3], [4] and I poked all my systems to see what values are produced. I only have a number of intel and intel/nvidia systems. I don't have any AMD portables at home. I would really like to see how they behave.

Towards that end I wrote a little tool that collects the most common properties (type and brigthness ranges) and dumps that to a tarball, along with the output of uname, lspci and a few others (nothing sensitive or personally-identifiable though). You can grab the tool from [5].

If you want to help me out please run the script and send the results to my email address  at zygmunt<dot>krynicki<at>canonical.com.

Thanks!