There's no doubt that Linux and open source are changing IT — Information Technology — at companies of every size. But how? When you read the IT magazines, go to the IT conferences, listen to the IT analysts, you get the same basic message you got ten or twenty years ago: vendors are in charge.
Of the sixty-two stories in Computerworld.com's current Week In Review, fifty-one, or 82%, are about vendors ("Red Hat goes Live with Fedora"), or refer to vendors in their headlines ("IBM's Palmisano says U.S. must innovate to keep jobs"). Non-vendor stories that should speak directly to open source concerns ("Cost containment may bolster '04 IT budgets") don't even mention the subject.
The market model is simple: it consists of vendors selling stuff to customers. That's it. When you talk about the "app server market", you're talking about the sum of all the app server goods and services vendors sell to customers. In graphical terms, it looks like this:
But in reality both vendors' and customers' IT worlds are steeped in a variety of development communities. Both vendors and customers develop goods for themselves, as well as for sale and for use by the rest of the world. As Eric S. Raymond pointed out in The Magic Cauldron, "there is empirical evidence that approximately 95% of code is still written in-house". So the real market actually looks a lot more like this:
In this ecology there is still plenty for vendors to sell, of course. But selling and buying gets put in a larger context — one that's rapidly changing as open source gains acceptance in IT shops everywhere.
Over the past year I've been on assignment by Linux Journal to study what's really happening in the IT marketplace, and the deeper roles played by Linux and open source development in that marketplace. My first report was How Linux Makes Companies Smarter, in last July's Linux Journal. This second report focuses on changes in IT itself..
What I've found is an increasing reliance on personal and development community initiative, and the freedom and trust making that possible. In sum, what I'm seeing is a do-it-yourself movement in companies everywhere: a growth in self-reliance I'm calling DIY-IT, for Do-It-Yourself IT.
Phil Moore, the Executive Director of Enterprise Application Infrastructure for Morgan Stanley and Company, explains:
Open source has lowered the threshold at which do-it-yourself is possible. You can't do everything with building blocks from vendors. They pretend they're selling you a prefab building and they're not. They're selling you pipes and fittings and stuff to put it together.
In reality, to build an enterprise, you have to have a set of experts in your IT shop who can put it all together. Certainly historically you need a lot of expertise to get anything done, because this stuff really isn't easy to put together. But if you've been led by the vendors to believe that everything dovetails together nicely, like you see in the .Net ads, or in any major marketing campaign that promises nirvana, you've got a problem.
You always need a certain amount of do-it-yourselfness. Consultants don't walk in, deliver an enterprise and walk out, saying "call me in six months for an upgrade." It's organic. An enterprise is constantly changing. Even the walls in your house right now are on their way to needing another paint job.
While DIY-IT involves a reduction in dependency on vendors, it doesn't mean vendors are bad, or that they don't play extremely important — often leading — roles in the marketplace. It does mean that the marketplace no longer belongs to them. It means there is a new balance of power between supply and demand, and a new division of responsibilities between vendor, customer and development communities. In sum, it seems the business is growing up, and becoming very much like other mature categories. Especially construction, from which it borrows much of its vocabulary (architect, design, build, tools, etc.).
In this report we'll look at several key factors involved: what leadership really means; the role of the Net; the rewards of courage; the cost savings imperative, valuing talent, where we stand, untold stories and perspective.
Follow which leader?
It's natural to look to leaders for leadership. But what about leadership of developments that have no leader? — developments where the leadership comes largely from within, from shared conviction and the practices that express it? What about leadership of movements that are tectonic in size and importance, yet go largely unnoticed? — where quakes along fault lines attract all the attention from those on the surface while vast hefts move silently and constantly everywhere but at the edges?
That's what we have with Linux, with free software, with open source. Linux is a development project, not a company. It is not contained by a corporate structure, Like a tectonic plate, it is held together by cohesive more than organizational forces. Free software and open source are value systems and development methodologies. To treat them strictly as populations, or as classes of goods, is to miss the nature and scope of what they're about. Plus the importance of every individual who contributes to them, or who takes initiative in putting them to use.
In the vendor-vs.-vendor sports coverage that comprises much of today's IT news flow, Linus is a red herring. "Gates vs. Torvalds" is a celebrity boxing match with only one guy in the ring. "That's user space," Linus says. "I don't do user space." Thus, with five short words, Linus dismisses the Dominion of Gates we see on the faces of 97% of the world's visible computers. The overlap of the two men's worlds is akin to that of land and lava. Gates' world is surface habitat over a thin layer of supportive bedrock. Torvalds' world is mantle and core. Which supplies most of the gravity?
Kernel space. That's where Linus says he lives. It's also behind 100% of the computers that live on the Net, itself a leaderless creation. Today the majority of the Web is delivered to browsers by open source servers running on open source operating systems. As of November 2003, Apache ran on more than two thirds of the 30+ million Web servers surveyed by Netcraft. And Linux is the leading OS for Apache.
The Net not only supports much of what we now take for granted in the technical world; it puts everybody and everything in a position to get more connected, more informed, more intelligent. That goes for people as well as machines. The cohesive force in tectonic movements like Linux is the Net itself. And the connectedness of the intelligence that lives there only increases with exposure and use. Public bits outsmart secret ones, even if secret ones still have economic and other forms of value. Same goes for the people who create those bits and put them to use.
Sean Moriarty, Executive Vice President of Technology at of Ticketmaster, puts it this way in his talk at OSCON last year:
The best thing about the Internet, to me, is that it mitigates tremendously the friction imposed by time and distance. You can take people who have individual passions and great talents, and unite them, and eliminate many of the obstacles to communications.
The open source community is truly global, and that matters an awful lot to us. We're an international company and we like access to very smart people worldwide as we grow — who are still accessible and relatively close-knit.
With an email or a post on a message board we can get really good information from people who are not only experienced with the technology, but may have created it themselves. If you have a valid issue to raise, and you know the right forum, you can get a response extremely quickly from someone extremely talented. Which is an amazing thing.
The notion that you can take part in this community, and do great things, and be supported, and foster this whole breeding ground of innovation, is absolutely incredible.
All you have to do is look at all the open source projects underway at any point in time, look at who the principals are, and look at where they're working. And you see some of what's going on. We're motivated by how much we can benefit from not only drawing from it, but by participating in it as well.
The Networked IT World
It is nearly impossible to imagine a civilized corporate or govenmental organization today that is not sustained by the Net, and therefore also by open source software. Organizations everywhere are therefore also coming to realize that they depend on open source values as well as talent and code. In just the last year it became clear to governments around the world that their computing infrastructure needed to be built on stuff that is open, that has no secrets. They want to be able to inspect it, to make sure it's sound, reliable and open to improvement — by anybody.
Case in point. For the last year or so, Diebold has been fighting a losing battle to justify secret bits in the brains of voting machines. While open source values may not apply everywhere, it is becoming increasingly clear, to both governments and citizens, that they do apply in this case. It can't be long before many of us ask, What other cases? The open source movement grows with every affirmative answer.
We've come a long way in a short time. Six years ago, when the entertainment lobby pushed the Digital Millennium Copyright Act through Congress, open source values were nowhere in sight. The DMCA passed by a unanimous vote. Today the same legislation wouldn't get out of committee. The turnaround is close to 180 degrees.
Who led that turnaround? Was it IBM? Oracle? HP? Novell? Those are your big "Linux companies" today. Does their Linux advocacy cause anything other than income and headlines? What if their enthusiasm for Linux were less a matter of cause than effect? If so, what's causing that effect? What's behind this shift in the magnetic poles that arc out through the networked world from kernel space below?
The answer is personal.
The rewards of courage
Take Sterling Ball, the leader and namesake of Ernie Ball, the guitar string maker. You can't hit SCAN on the car radio without hearing the tones of an Ernie Ball string.
What makes Sterling Ball an open source revolutionary isn't his technical chops. It's his independence. His guts. Those became evident after Ernie Ball was raided in 2000 by federal marshals as part of an "unannounced software audit " by the Business Software Alliance, which found (surprise!) unauthorized "pirated" software on some of Ernie Ball's computers. The BSA still brags about the raid on its Web site, and how it made an example of its target:
The legal action began with a call to BSA's anti-piracy hotline 1-888 NO PIRACY. BSA filed a complaint for copyright infringement in federal district court in Los Angeles, and the court ordered the unannounced audit of Ernie Ball's computers. The court also entered a temporary restraining order preventing Ernie Ball from deleting software from its computers.
Here's how Sterling Ball put it when I interviewed him at Linux World last August:
A disgruntled ex-employee saw a nail-your-butt opportunity, so he called the BSA. I was sued under federal seal. There was no warning. We were raided at ten o'clock on a Friday. We were shut down and ordered not to touch our computers. There were armed marshals. Our employees were sitting there going 'What's the matter? Is our company criminal? Are we crooks?' Then they sent out press releases... It's coincidental that they always send these out after business is closed.
We're the number one employer in terms of manufacturing in San Luis Obispo. We're a big fish in a little pond. The headline reads, "Ernie Ball Raided for Piracy." And the story says "Company officials unavailable for comment." Well, no shit. I was at home. And I never say "No comment". So, when it came time to tell my story, I said "They came for bear and got squirrel."
Ernie Ball went to court and paid a fine; but that didn't end the matter:
The worst thing was when Microsoft printed a four-color reproduction of that newspaper article on an executive's desk, sent it to every registered Microsoft user, and said "Don't get caught like Ernie Ball — a fine company that found out just how hard it is to stay compliant. Call us. We'll give you a free audit and sell you software at 20% off." Keep in mind that we had already downloaded the BSA self-auditing software and it didn't work. This was fear-based marketing, with government help.
Sterling Ball didn't just get mad. Or get even. He got out:
I didn't care if we used ten thousand abacuses. We were not going to do business with Microsoft.
Everybody thought I was crazy. The IT people thought they were going to get fired. I said 'no'. Because I've never seen any greater programming in the world than "You can't do business unless you've got an office suite on your desk." Hey, I'm talking here at Linux World because I changed my word processor! The solution everybody (at our company) uses is a cocktail of open source stuff. Nobody showed us how to do it. We had to figure it out ourselves.
Today Ernie Ball's servers run Red Hat Linux. Its desktops are Gnome on thin Sun clients, with apps that run off a Linux server. The company time clock and security software run on Linux. The company email is Ximian's Evolution Their office suite is OpenOffice. Everything is open source. Nothing is Microsoft.
Sterling Ball does give Microsoft credit where due: "I grew my business with Microsoft. Now I run it with Linux."
The lesson here isn't about technology. Just about every Linux Journal reader knows how to build a Microsoft-free office. The lesson is about independence, integrity, and the courage to break free of mental programming so deep and pervasive that it rivals The Matrix in detail and complexity. It's a lesson about the souls of individuals, of organizations, and of a marketplace that still thinks vendors are in charge, even though the success of the Net and the open source movement prove they are not.
The cost savings imperative
As with The Matrix, breaking free is akin to awakening. And it doesn't just happen for companies the size of Ernie Ball. Take Ticketmaster, for example. Here's Sean Moriarty again:
Ticketmaster provides a service for live event ticketers — the people who actually own the tickets: artists, promoters, venue owners, professional sports teams. We provide the infrastructure and the systems by which they sell those tickets to the public. We support 8000 clients in 10 countries. In 2002 we sold 95 million tickets through channels that represented over four billion U.S. dollars. This puts us in the top 25 of all retail Web properties — actually #2 between Dell and Amazon. So we're doing a lot of business, but on behalf of other people who have entrusted us with their business.
We have 3500 outlets, 19 call centers, and the Ticketmaster.com Web site, which does about 50% of our business. We also provide box office solutions. If you've ever purchased a ticket at a box office, that's also a Ticketmaster system, with the same inventory bucket.
Our product and technology organization is the cornerstone of Ticketmaster as a company. We've got 250 people devoted to product and technology in an organization that's about 2000 full-time employees.
We provide solutions and systems, but we also support those 8000 clients. In many cases, because of the nature of the business — highly customized, highly variable traffic, and all kinds of strange configurations that are actually much different than any other retail businesses — there are no commercial solutions available for what we need to do.
In fact, we are one of the first application service providers: extending the service to thousands of clients, actually writing the code, hosting the system, providing the customer service, and charging a fee per unit sold. We have to build high volume systems for very specific and peculiar businesses. Open source allows us to do as well or better — at least in our experience — at half the price of commercial solutions.
That's why Ticketmaster converted the Ticketmaster.com site primarily to open source technologies over the course of the last 18 months.
He showed a small spreadsheet:
| Ticketmaster.com | Open Source | Commercial | |
| 400 PC Based Systems | $1MM | $1MM | |
|
$0 | 600,000 | |
| Web Server Software | $0 | $120,000 | |
| Database Software | $0 | $240,000 | |
| Total | $1MM | $1,960,000 |
It's really all about licensing costs. We buy the same class of machine, same configuration, from the same vendor. But we're using all open source technologies in these areas, and we pay nothing for licensing. So you can see that we save 50 cents on every dollar we invest, and get the same or better performance.
And we see better support from amongst the community than we typically get from commercial vendors. There aren't that many other vendors who have the same solution set, the same spikiness, the same aggregate peak traffic that we do. And it's nice to have others using the same tools in the same way.
We've been able not only to build a site that works very well day-to-day, but because we have access to source, perform those optimizations for traffic profiles that don't really exist anywhere else.
The Website costs, in hardware and capital development, a couple million bucks. So it's a small fractional part. It doesn't materially change the business from Wall Street's perspective, or from the CFO's perspective. But it matters to us because the money that's made available we can use to employ more smart people. And that's really the key.
Valuing talent
To Sean Moriarty, open source human resources are collective as well as individual. To explain, he quotes T.S. Elliot: No poet, no artist of any art, has his complete meaning alone.
It's about the power of the individual talent. I fundamentally believe that individuals are capable of producing many wonderful things if they have passion, aptitude and the opportunity to do it.
It's not about philosophy, politics or allies. It's about people. Open source gives us access to a specific type of individual talent, which I believe is the most important ingredient in the success that we've had. We're driven by common sense. And by providing really robust tools to our people, and by pulling in more people who are interested in solving problems, who can work autonomously, but who like to have the benefit of a larger community while they're working...
(People like) Stas Beckman, doing core work for us... mod_perl 2.0. (He's) doing a lot of the work that will benefit the community, which benefits us. He's also working on database connection pooling... So we get to take the lead on doing some things that have broad general benefits for everybody. Geoffrey Young is going to start working with us. He's going to be working on things that can be done autonomously, that benefit the community but also provide immediate benefit for our business when deployed in very specific ways.
Our teams have ownership over their tools. They also don't have an excuse. They can't say that a vendor doesn't have an answer or isn't getting back on the phone. Because everybody knows that there's a community out there and you have access to the source. Everything is in front of you. So there's an accountability that's reinforced, when you have source code and a community that knows so much and is so willing to respond.
You're also more motivated. When people pick their tools, the work invariably makes sense to them. They're also working side-by-side with the people creating the tools that we're using day-by-day. You can't get that anywhere else. This has been incredibly powerful for us.
Once again, a company gets smart and saves money by aligning itself with its own smart engineers — along with the development communities to which they belong.
What's different between now and ten or twenty years ago? Moriarty says it's "the amplifying effects of the Internet on the power of the individual, adding "Only a fool would ignore that."
As for commercial suppliers, he adds, "The support of the major vendors, whether it's in word or in kind, makes no difference to me. This bears witness to the power of the model."
Where we stand
Things are moving fast. Two years ago, I had to go scrounging around to find IT managers who would even admit that they had Linux operating in their companies. As for CIOs and CTOs, forget it. "Not even SAMBA servers?" I'd say. "What's a SAMBA server?" they'd reply.
Last year one SAMBA developer told me there might be fourteen million SAMBA servers in the world. That's the only number I've been able to find. "Most of them are dark matter," he said. "But people keep putting them out there."
The question is, What people?
The top guys? We read about IT brass going gaga for Linux almost every day. Ken Harris, CIO of Gap Corp., recently said he's in favor of "Anything touching Linux." Emea Harris of Lehman Brothers said, "We're very aggressive around migrating to Linux."
Doug Kaye, an author and Web services guru, puts it in context:
Open Source is now also the new source for "vision" for IT managers and CIOs. How do you know what's real and not just vaporware? Go to the bookstore and look at the O'Reilly end cap display. There you'll see the LAMP titles: Linux, Apache, mySQL, and the Ps. These are the technologies from which one can quickly and inexpensively build industrial-strength applications. Real work doesn't begin until technologies reach the early adoption phase. The O'Reilly books appear, standards are finalized, and one can at least hire a consultant who knows a little bit about the technology. This is also the point at which technologies first appear in open source. Hence, it's open source — not the earlier vendor hype — that provides the what-you-can-do-now vision for IT managers today.
Vision is nice, but there's not much how-to involved. And it's around how-to stuff that rank & file IT workers tend to clam up. Phil Moore, who prides himself in "saving the company money by kicking vendors out the door", explains corporate reticence about talking open source turkey:
The reason you don't hear more from companies on the customer side is that people aren't allowed to talk about what they're doing. After I spoke at O'Reilly's Open Source Convention two years ago, I got in trouble for a lot of the things I said. But (the talk) got a lot of attention on our use of open source. It actually got our competitors to start calling up and saying "You guys are doing open source? Really?"
It was a story that people weren't talking about. And what disturbs me the most is that we're still not getting those stories out there. Attention is still on marketing messages from companies like IBM, which have jumped on the open source bandwagon. Their stories tell only one side of what's going on.
Yet there is progress on trails that guys like Phil Moore have already blazed. When I spoke about DIY-IT at the O'Reilly Open Source Convention last July, the majority of those attending the talk (about a hundred in the room) were rank & file IT guys, mostly from large companies. Phil was one of them. I asked the room, "How many people here feel they're not allowed to talk?" Very few hands went up. One audience member said some companies feel that their own open source developments give them a competitive edge, and don't want to talk about it for that reason. "They don't want their competitors to know how they do it faster and cheaper," she said. But others in the room dismissed the issue. Mostly they didn't talk because, as Phil said, talk is forbidden. That's still marketing's job.
Untold stories
So what other kinds of stories are we not hearing, then? Here are a few, in no particular order:
Phil Moore again:
Open source has lowered the threshold at which do-it-yourself is possible. You can't do everything with building blocks from vendors. They pretend they're selling you a prefab building and they're not. They're selling you pipes and fittings and stuff to put it together.
In reality, to build an enterprise, you have to have a set of experts in your IT shop who can put it all together. Certainly historically you need a lot of expertise to get anything done, because this stuff really isn't easy to put together. But if you've been led by the vendors to believe that everything dovetails together nicely, like you see in the .Net ads, or in any major marketing campaign that promises nirvana, you've got a problem.
You always need a certain amount of do-it-yourselfness. Consultants don't walk in, deliver an enterprise and walk out, saying "call me in six months for an upgrade." It's organic. An enterprise is constantly changing. Even the walls in your house right now are on their way to needing another paint job.
Perspective
It's impossible to ignore the influence of marketing on the news it pays for. Without vendors, we wouldn't have magazines or trade shows, to name two of my favorite things. It's important to the market's ecology for vendors to push their goods and tell their stories. The problem we've had — and still have — is a long lag between what's happening in the marketplace and how we cover the subject. And I believe that lag derives from the young ages of the industries involved.
The computer industry is about fifty years old. The software industry is half that age. The Internet — which changed everything — began supporting business only about nine years ago. Linux and open source, which are impossible to separate from the growth and success of the Internet, have only found widespread acceptance in the last couple of years, and both are still considered "threats" by the largest software company on the planet.
What we need are more stories from the demand side of the marketplace, and more courage by those in position to tell them. We also need publications that welcome those stories, with authors and editors and analysts to help tell them.
My friend Christopher Lydon, a former reporter for the New York Times and host of NPR's Connections, believes what's happening in our industry — this DIY-IT movement — is profoundly Emersonian. He points to this encouraging prose from the author's seminal essay, Self-Reliance:
To believe your own thought, to believe that what is true for you in your private heart is true for all men, — that is genius. Speak your latent conviction, and it shall be the universal sense; for the inmost in due time becomes the outmost...There is a time in every man's education when he arrives at the conviction that envy is ignorance; that imitation is suicide; that he must take himself for better, for worse, as his portion; that though the wide universe is full of good, no kernel of nourishing corn can come to him but through his toil bestowed on that plot of ground which is given to him to till. The power which resides in him is new in nature, and none but he knows what that is which he can do, nor does he know until he has tried.
We'll be doing our part here at Linux Journal. And, as always, we rely your help as well.
Doc Searls is Senior Editor of Linux Journal. His monthly column is Linux For Suits, and his bi-weekly newsletter is SuitWatch.