Using RGB and indexed color modes in terminal applications

Recently I was working on finalizing a small command line application for Canonical Hardware Certification. The application here is of no great importance but I wanted to give it a better look than what our applications traditionally look. I made sure to pick the right colors from the Canonical color palette. I used appropriate paragraph layout and spacing (this is all text mode console, remember). It all looked great.

This is, in part, possible thanks to the RGB color support offered by Gnome Terminal. I tested it extensively and it was really quite gorgeous. Then, I ran it in screen. Ugh. The application was barely readable! Screen doesn't understand the escape codes I was sending and just happily ignored them. This was not what I was after. I didn't want to scrap everything either. I wanted this to work as best as the software around this permits.

Fast forward to today. I spend a while iterating on the basic idea and experimenting with a few ways to expose it. The application I wrote was based on python3-guacamole. A simple framework that helps to write command line applications, sitting basically one level above argparse and main(). Guacamole has support for ANSI color codes and exposes that neatly to the application via high-level interface. Instead of calling print(), the application can call aprint() which understands a few more keyword arguments. Among those arguments are fg and bg (for foreground and background color, respectively).

Guacamole also gained a color controller, an ingredient acting as a middleware for all color handling inside the application. I created it to explore the impact of usability-enhancing color use on users with various types of color blindness. It is an accessibility tool but making it required adding quite a few color transformation operations to Guacamole. If you haven't seen that yet it's quite interesting. The rainbow demo, invoked with the right options, is beautiful, in all its 24bit terminal emulator glory.

Today I expanded on that idea and introduced two new concepts: a color mixer and a the palette.

The mixer came first. It is a new property of the color controller (that applications have access to). Its goal is to act as a optional downmix of 24 bit RGB values to the special 6 * 6 * 6 palette supported by most terminal emulators on the planet. Implementing it was actually quite straightforward though the problem of coming up with an indexed image for a given true-color image is an interesting research topic. Here most "difficult" things don't really apply as we have a fixed palette that we cannot change, we cannot do dithering really and "pixels" are huge. This turned out to work great. I patched my application to enable the indexed 8-bit mode mixer when a given TERM or SSH_CONNECTION value is used and it worked like a charm.

Looking at the code though, I wanted to avoid the inefficiency of doing the conversion for each "image". At the same time I wanted to allow applications to offer optimized, hand picked indexed color equivalents of some RGB colors. This all materialized as generalized, extensible named-color palette. Now, the application has calls like:
aprint("something", fg="canonical.aubergine") instead of aprint("something", fg=(constant-for-canonical-aubergine)). The named color is resolved in exactly the same way as other constants like "black" or "red" would. What is also cool is that the color mixer can now expose the preference for PREFER_RGB, PREFER_INDEXED_256 or PREFER_INDEXED_8 and the color lookups from the palette may fetch optimized values if the application provides any.

All in all, this gives me a way to run my true-color terminal application via screen, putty or OS X's terminal application. It behaves good in all of those environments now.

The only problem left, is accurate terminal emulator detection. I want guacamole to handle this internally and let applications be blissfully unaware of particular limitations of particular programs. This is an open problem, sadly. TERM is useless. Everyone says "xterm" or something similar. People hack around it in their startup scripts since various programs hard-code detection to "xterm-256color" or similar. Many different programs use the same TERM value while offering widely different feature set. TERMCAP is equally useless. It might be better but nobody uses it anymore, with the notable exception of screen. All the stuff there is just old legacy crap from around the start of UNIX time. What I would love to have is something like what Apple's Terminal.app does. It sets two environment variables that contain the name and version of the terminal emulator. With this knowledge libraries can map out features, known issues and all the other stuff with far greater ease, simplicity and speed than any new standardized definitions (TERMCAP) would allow. Now if those also got sent across SSH then my life would be much easier.
For now I will see what can be done to get this just right in the Ubuntu ecosystem. I can probably patch gnome-terminal, ssh (to forward those variable) and perhaps screen (to set them).

I'll send a follow-up post when this lands in python-guacamole. I plan on doing a release later today.