The Embedded OS

abstract

A about four years ago, in the March 2002 issue of Linux Journal, I wrote a column titled "A tale of three cultures". The first culture was the technical community that brought us free software, Linux, open source and the free and open marketplace we call the Net.

On last Fall's Linux Lunacy Geek Cruise, I had the pleasure of getting some very educational hang time with 2.6 kernel maintainer Andrew Morton. I came away not only with enormous respect for Andrew and the kernel crew, but with an even deeper realization that Linux is not an OS in the sense we've been taught to think about one by the commercial marketplace.

Meaning, Linux isn't a product.

Linux is a project, of course. Specifically, it's an open source development project But that's a technical explanation, a noun compound. As a description, it doesn't convey the vast difference in nature between Linux and commercial platforms such as Microsoft's Windows and Apple's MacOS. Even when Linux is packaged into a distro, it's still radically different. And that difference is greater than the one between product and project, commercial and noncommercial, closed and open. It's a difference in nature itself.

In the past, I've characterized Linux' nature as that of a building material. Furthermore, I've characterized it as natural building material — like wood or granite. I've said it occurs in human nature, produced by human thought and work. I still believe all that is true. But now I believe it goes deeper than that.

This is what I learned from Andrew and Ted Ts'o, the file system maintainer who was also on the cruise. After sitting in on their sessions (which ran many hours — the cruise was something of a crash course at sea), I realized that Linux's nature, as a building material, is akin to that of a species.

As I write this, the creators of commercial OSes are laboring to anticipate forms of work that nobody is doing yet. Even OS X, which sits on a base of BSD, is being altered constantly to anticipate functionalities nobody is enjoying yet. Same with Windows developers working on Vista (the new Windows successor due for release this year). Theirs are products that practice Marketing 101 for consumer electronics: create something that mothers a need. For Apple it might be HD video editing. For Microsoft it might be multi-layered Web services. For Linux it might be...

"Stamping out bugs".

That was Andrew Morton's answer to a "What gets you excited?" question on the boat. It was a huge clue for me. Because I realized Andrew's job is almost entirely reactive, rather than proactive.

I used to wonder why Linus, when he spoke on earlier cruises, would answer so many questions with "I don't care" -- and then explain why by saying "That's user space". At the time I thought it was just a way of confining the scope of his interest, and keeping distractions away from the hard and necessary work at the Bottom of Things. Which it was. But it was also something much sharper than that, which is becoming clearer to me now.

User space is where commercial products live. Right now, on the desktop and the laptop, that's Windows and OS X, mostly. Eventually it will be Linux. But only after Linux evolves to a point where it's ready to support commercial products that compete.

Andrew Morton doesn't doubt that we'll get there. He also isn't holding his breath wating. Because he knows evolutinon moves at an infuriatingly slow pace.

Kernel development is not about Moore's Law. It's about natural selection, which is reactive, not proactive. Every patch to the kernel is adaptive, responding to changes in the environment, as well as to internal imperatives toward general improvements on what the species is and does.

We might look at each patch, each new kernel version, even the smallest incremental ones, as a generation slightly better equipped for the world than its predecessors. Look at each patch submission -- or each demand from a vendor that the kernel adapt to suit their needs in some way -- as input from the environment to which the kernel might adapt.

We might look at the growth of Linux as that of a successful species that does a good job of adapting, thanks to a reproductive cycle that shames fruit flies. Operating systems, like other digital life forms, reproduce exhuberantly. One cp command or ctrl-d and you've got a copy, ready to go -- possibly into an environment where the species might be improved by a patch. As the population of the species grows, and more patches come in, the kernel adapts and improves.

These adaptations are far more often reactive than proactive. This is even (or perhaps especially) true for changes that large companies such as IBM and HP, which might like to see proactive changes made to the kernel, to better support their commercial applications.

Several years ago I had a conversation with a Microsoft executive who told me that Linux had become a project of large commercial vendors, because so many of kernel maintainers and contributors were employed by those vendors. Yet Andrew went out of his way to make clear, without irony, that the symbiosis between large vendors and the Linux kernel puts no commercial pressure on the kernel whatsoever. Each symbiote has its own responsibilities. What's more, those responsibilities to themselves has a natural selection mechanism of its own.

To illustrate, he gave the case of one IBM application: DB2:

Some DB2 people .. the file system based AIA. The DB2 team don't want to implement it until it's available in the kernel. One of the reasons I'd be reluctant to implement it in the kernel is that they haven't demonstrated that it's a significant benefit to serious applications. They haven't done the work to demonstrat that it will benefit applications. They're saying "We're not going to do the work if it's not in the kernel". And I'm saying "I want to see it will benefit the kernel if we put it in".

He adds, "On the kernel team we are concerned about the long-term viability and integrity of the code base. We're reluctant to put stuff in for specific reasons where a commercial company might do that." There is an "organic process" involved in vendor participation in the kernel, he explains; but it is not one that is analagous to commercial development. In fact, it is radically different, yet in some ways far more natural:

Thing is that, if you look for example at the IBM engineers that do work on the kernel. These people, they understand that now. They are no longer IBM engineers that work on the kernel. They're kernel developers that work for IBM. My theory here is that if IBM management came up to one of the kernel developers and said "Look, we need to do that", the IBM engineer would not say, "Oh, the kernel team won't accept that". He'd say "We won't accept that". Because now they get it. They understand the overarching concern we have for the coherency and longevity of the code base.

Given that now these companies have been at it sufficiently long, they understand what our concerns are about the kernel code base, if IBM need a particular feature, they can get down and put it in the kernel. Just as they would for AIX. There are some constraints about how they do that, however, and they understand that.

Kernel space is where the Linux species lives. User space is where Linux gets put to use, along with lots of other natural building materials. The division between kernel space and user space is like the division between natural materials and stuff humans make out of those materials.

During a Q&A, somebody asked Andrew what got him excited. "Stamping out bugs" was his reply.