Missing pieces


Robert L. "r0ml" Lefkowitz (that's a zero, not an o...you figure it out) is a tall, smart, charismatic and good-humored man with a shaved head and a red clown nose in his pocket that he sometimes wears to explain, visually, that he's "just a bozo." Except for the nose, he's not. Lefkowitz is an IT veteran who stepped from the frying pan of Wall Street to the fire of the Telco business -- specifically from Merrill Lynch to his current job as Chief Technical Architect and VP Information Technology for AT&T Wireless. He calls himself "geek-in-chief."

R0ml is a long-time advocate of open source. But he's also the rare open source advocate who will tell you all the ways open source is either inadequate as a development methodology, or still lacking in the tools and solutions large IT organizations require. He makes it his job both to prod the open source community to produce the missing goods, and to find new goods for possible adoption by his company. He likes when his own engineers have already found them.

Last July at the O'Reilly Open Source Convention, Lefkowitz gave a talk titled "Six missing open source projects." It was as much a primer on The Real World of IT as a call to action for the open source programmers who packed the room. "Relationship Management" was one example. "Relationships are more important to most companies than code," he said. Which is why they spend more money on marketing than on programming. He said there is plenty of CRM -- Customer Resource Management -- software out there for big companies; but none yet from the open source world. Also none for following relationships between IT departments and development communities. There were so many questions such a system could answer: Which industries have the highest patch submission to running copies ratio? What percentage of bug reports come from financial services firms? How about from other industries? Who's working on what, anyway? And for how long?

He also called for help with asset management, distributed cron, change management, messaging, single-sign on and source termination. He said we should work to create a definitive literature for problem domains, to make it easier to integrate open source systems. He called for open source developers to start thinking about operations: accounting and financial systems, help desk automation, customer relations, marketing. "All the stuff businesses care about."

This March at the Open Source Business Conference in San Francisco, R0ml gave a talk titled (More) Missing Open Source Projects. In it he proposed four criteria that must be met for open source projects to take root:

He opened his talk by holding up two CDs and explaining that these were what was left of a telecommunications billing system called Flexcell®, that was orphaned when AT&T Wireless ate Vanguard Wireless, the North Carolina company that wrote it. "How cool would it be if there were an open source company in North Carolina?" R0ml quipped, tongue planted deeply in his cheek. "That would be fortuitous. Particularly if they were interested in enterprise systems."

He went on to explain that telecommunications billing systems were terrific ways to show off micropayment chops. Phone billing is personalized, goes down the second, and keeps track of many variables all at once. "Very kewl," he said. Then, the pitch:

So we are interested in testing the hypothesis. How will you take something this cool and turning it into an open source project. Would anybody actually be interested in working on that? We won't invest a heap of money. But if other people are willing to invest some time or energy or money, then certainly that would encourage us to work with them a little bit.

So the offer I'm making, if anybody is interested, is doing the world's largest and coolest micropayment system, I've got the source code. (And) I do have authorization to look into how to open source such a thing. I'll be happy to take any licenses or whatnot. It's all open for discussion.

Then he pitched CRM: "Since I was so successful at convincing you that billing systems are really cool, I'd like to give it to you that CRM systems are really cool." Then he detailed a peer-to-peer CRM system he called "Carester." He went on to pitch projects in visual programming, business process integration (BPI), messaging, business intelligence (data warehousing).

"Other than "kewlness", is there any other reason why open source doesn't tackle billing, CRM, BPI and business intelligence?" he asked. Is it scale? Performance? Lack of a market? "Maybe this just isn't a value network that open source can ensconce itself in. I don't know. I'm just an open source guy, and a bozo in IT." Then he brought out the nose.

It was a downbeat talk. Kind of a Swiss cheese treat where all they serve is the holes.

Later in a telephone interview, I asked R0ml to name some examples of cases where open source was succeeding, even in the categories he had mentioned. He said,

"Take the data warehouse space. You're dealing with expensive, specialized hardware. Teradata NCR. Old style hardware-software bundles that do large specialized databases. Now they have a competitor: Netezza. When a query comes into a Netezza box, it hits a quad-CPU that's running Red Hat Linux and PostgreSQL. Since PostgresSQL has a BSD license, they hacked it to do all this funky parallelization, so they can run it on their platform, which is kind of a blade thing with these souped-up disks. The executive, if you will, that does all the dispatching to all the subnodes and collects all the responses and sends them back to make these special-purpose queries, are Linux-based. If I buy them, I'm buying special-purpose hardware that's able to come into the market to undercut the established players. And they can do that because they build their stuff up from open source basics like Linux and PostgreSQL, which they can take and adapt to their needs. They're doing DIY: Do-It-Yourself. They're being smart and resourceful. I like that."

The DIY-IT environment, R0ml says, is a complicated place. And it will never submit to simplistic DIY efforts. Least of all those that limit their interest to open source:

"Most environments are mixed. In some mixed environments, especially those using large enterprise class software, you have some projects that will probably never be handled by open source, and probably never come from anybody other than a big vendor.... Here at AT&T Wireless, none of the large enterprise class software we're using is available in open source, outside of databases. Our billing system uses Siebel, on Sun. It also runs on HP-UX, AIX or Windows. But not Linux. Our billing system runs on something other than Linux. We also use Vitria businessware, which just added Linux support."

On the other hand, "There are two classes of vendors in this space now: those that say they'll deliver products on Linux any day now and those that are thinking about whether there's a market. Meanwhile, there's a market." In other words, it's catch-up time.

Meanwhile, your bread & butter enterprise systems are going to be provided by vendors. Not by customers. Sometimes a vendor will drive things. For example, The Wall Street Linux Roundtable was sponsored by Intel. There were representatives of various vendors such as Reuters, which does a lot of business on Wall Street. "When you have representatives from Deutche Bank, Morgan Stanley, Merrily Lynch, Goldman Sachs, all sitting in the same room saying, ""We want stuff on Linux," the vendors are going to go back to the office and say "We underestimated the size of the market."

Back at the Open Source Business Conference, I led a panel on DIY-IT that included R0ml, Kevin Foreman of Real Networks, Win Coekaerts of Oracle and Ted Shelton of CallTrex. Everybody made interesting points, each of which said something important about what each kind of company contributes to the market ecosystem. Wim Coekaerts, for example, made this point about testing:

It's not just about contributing source code. It's about having the resources in hardware. Big iron hardware that your run test suites on and you do stuff with. There is the communication that you have internally. That's how a big part of what we contribute is testing. We have to do that for our customers. They expect a product to be reasonably well-tested, and to meet minimal criteria. Also, if you look other Unix operating system, or Windows, hardware and software come from the same company: Solaris on SPARC, AIX on PowerPC and so on. The companies themselves that built those systems have the hardware to do the testing. They've had tons of people dedicated to working with Oracle to run the database on their platforms. The testing involves a very small community. Now if you look at Linux, none of the distribution developers can afford the big iron hardware. But what we're doing now is working with the distributions to get the hardware vendors to participate in the testing. So we're trying to set up this virtual Linux test environment where...we'll deal with Dell, HP and other companies to say, "Here is a subset of tests you should run at the vendor side, in your early cycles. So that when they ship a hardware product, it has been pre-tested. In the past they didn't do that. You have the hardware specification list, you get a CD, and good luck."

Ted Shelton offered a very interesting answer to R0ml's question about what it would take for open source appear in some of these large-scale enterprise categories:

I've heard that things like CRM and ERP are not going to be addressed to by open source. If you only look at open source as a very broad based group of otherwise disconnected people coming together to do something, that's probably true. But sometimes you get a company that builds a big solution in a vertical market, and decides to take it out in the open source market, because they realize their profits are in selling hardware, not software. And they've now developed a big community that provide bug fixes and applications running n top of it. That's Asterisk (the open source Linux PBX).

Then Ted turned to Wim, and said, "So, when Oracle gets done buying Peoplesoft, why not open source all their stuff?" Ted did it for laughs, but you could see some heads in the audience nodding along.


Doc Searls is Senior Editor of Linux Journal. He also writes SuitWatch, a bi-weekly Linux Journal newsletter. He also presides over Doc Searls' IT Garage, a sister site to Linux Journal on the Web.