The Rise of the Underground Engineer

By Larry Lange
Hobbit, Mudge and Yobie refer to it simply as "The Dinner.'' At one level the meal was an attempt by two Microsoft Corp. executives to get inside the heads of three hackers who had spent most of the previous seven months ripping apart the security underpinnings of their Windows NT operating system, a product on which much of Microsoft's future rides. The meal was also a confrontation between two clashing cultures: the business-minded managers from Redmond and a trio of long-haired techno-radicals sporting T-shirts and pseudonyms; yuppie and hippie. Made all the more strange by its somewhat surreal setting in an upscale restaurant on the Las Vegas Strip, The Dinner was also an emblem of a broader meeting of the minds: In an age of pervasive distributed computing, the corporate design community is reaching out to hackers as a kind of unofficial vanguard of security, the underground engineers of the Internet era.

But if The Dinner is any indication, the embrace is an awkward one at best. The Microsoft meeting had plenty of cue for tension, coming as it did on the heels of the Black Hat Briefings. It was at this security conference last July that Hobbit delivered a treatise on the fundamental flaws in NT's Common Internet File System (CIFS) to a mixed audience of some 80 hackers and security professionals representing straitlaced concerns like Cisco Systems, the Department of Defense, the National Security Agency, Price Waterhouse and Toyota. Hobbit's presentation was punctuated by catcalls-"That's wrong!" and "No, no, no!"-from one of the hosts of The Dinner, Paul Leach, Microsoft's director of NT architecture.

Not usually prone to such outbursts, Leach is an accomplished engineer better known as one of the founders of Apollo Computer, the Chelmsford, Mass., workstation maker that is now a part of Hewlett-Packard Co. Besides working for Microsoft, he also sits on a number of Internet-related industry work groups and is an auxiliary professor in the department of computer science and engineering at the University of Washington.

Whatever the tensions, The Dinner with senior managers of, arguably, the most powerful technology company in the United States was a mark of vindication, a badge of legitimacy, for the three hackers. "It was the proudest moment of my professional life," says Yobie Benjamin. And that's saying a lot.

Recently promoted to chief knowledge officer at Cambridge Technology Solutions (Cambridge, Mass.), Benjamin has traveled from self-described "off-the-boat immigrant" from Manila to respected Java hacker to big-salaried security spokesman at a $2 billion company. He also happens to be something of an unofficial mouthpiece for a hacking community that began revealing specific vulnerabilities in NT and posting them to the Internet earlier this year.

"It was bound to happen for Microsoft and NT," says Benjamin, a fresh-faced Filipino with long, jet-black hair that bobs from his shoulders as he excitedly makes a point. "They're a victim of their own success, and the OS is completely hampered by backward compatibility. It's so easy to break." Still, he says, "this [meeting] was a good first effort from them. I think that more cooperation will follow now."

The waif-like Mudge, who has a day job as a security expert at a major networking firm in the Boston area, shakes his head vehemently. "No, no, no. I got the distinct impression that they were forced to come here," he says, his piercing blue eyes shining with anger. "About seven minutes into it, I was about to get up and walk out. They were so not getting it."

Mudge, who declines to disclose his real name, abruptly stands and begins preaching in a manner reminiscent of a '60s radical. (In fact, Mudge later jokes that if he were to write an autobiography, he would title it Steal This Network, in homage to the '60s screed by his hero, Abbie Hoffman, Steal This Book.) "Microsoft is so ingrained in its ideology," he rants. "I mean, they were even talking about Unix systems that had ancient security holes from 20 years ago. That's not the response I want from Microsoft spokespeople on security, especially face to face."

As a founder of the institution knownas "The L0pht," Mudge feels he has a right to some straight answers. The often-misunderstood band of Boston-based hackers has pulled apart Unix-in both Solaris and FreeBSD versions-Novell Netware, IBM's Lotus Domino and most recently NT. Over the years, their work has attracted unwanted and vaguely threatening visits from "suits" at the NSA and FBI.

Mudge was a focal point of the Las Vegas dinner because his L0phtcrack programs, which descramble and give access to NT passwords, were among the core hacks Microsoft had faced in recent months. With patches and upgraded NT Service Packs, the Redmond, Wash., software giant has since made Mudge's work unwieldy to use. Still, this underground engineer has no intention of stopping. He promises an enhanced and more-robust version of L0phtcrack soon.

Mudge and Yobie look to their acknowledged leader, Hobbit-nee Al Walker, who in real life runs his own consulting company-for a definitive take on The Dinner. "I thought it was a cordial conversation," said Hobbit in his soft monotone, "although I wasn't quite sure what their motives were. Probably just to loosen us up a little and see what we're like. If they take home some of the ideas we gave them and it gets under their skin enough to make them actually go and fix what clearly are problems, that's just peachy. But I wouldn't count on it."

History of a hack
This soft-spoken, frail-looking man clad in a tie-dye T-shirt, faded cutoffs and ragged sandals-his scraggly hair pushed back in a ponytail-was the one who kicked off the NT hackathon. Last January, quite out of the blue, he published a 20,000-word technical treatise on his Avian Research Web site, complete with an abstract. Wryly titled "CIFS: Common Insecurities Fail Scrutiny"-a takeoff on Microsoft's Common Internet File System-the paper rang like a virtual rifle shot through the hacker community, which responded by swarming around Windows NT. "I had to read [the paper] three times," said Yobie Benjamin, "just to begin to understand it, but then I understood perfectly" Hobbit's breakthrough analysis of NT security weakness cut to the heart of the operating system's vulnerabilities, the hackers said. CIFS is a file-access protocol designed for use over the Internet and based on Microsoft's Server Message Block protocol, used for file and printer access in Windows. Hobbit claimed to have found ways to use TCP/IP file-sharing protocols with CIFS in a way that could compromise the integrity of NT.

Less than a month after Hobbit's work appeared, another important NT security paper hit the Net, a document called "A Weakness in CIFS Authentication." It was posted on the Usenet by another speaker at the Black Hat confab, a Washington-state-based NT security consultant and former Microsoft programmer who called himself Nihil.

On its heels, in March, came the NT password-cracking tool "Pwdump," written by Jeremy Allison of Cygnus Solutions, a Sunnyvale, Calif., provider of software security tools. When run on an NT server, Pwdump could basically descramble all the encrypted passwords in the registry and dump them out in a file to plain text.

Next, Yobie Benjamin added more fuel to the virtual fire by galloping onto the Net with a Trojan horse. His program, when shipped to an unsuspecting e-mail addressee, would find its way onto an NT server at the person's organization and run Allison's Pwdump. Pwdump would grab the server's passwords out of the registry and ship them back to the sender.

Mudge then added his L0phtcrack program to the mix. By April it was clear that the hacks-and press reports of the bugs they found-were in high gear, with no end in sight. And by the second week of July, Leach and a senior Windows NT marketing executive, Carl Karanan, were on a plane to Las Vegas to hear these underground engineers speak at the Black Hat Briefings and then meet with them face to face.

Microsoft's goals in inviting the three to dinner "were to understand the issues they have, and I think that setting up a working relationship is really the first step in that," said Mike Nash, director of Microsoft's NT Server group, who did not attend The Dinner but clearly was briefed about it. "The results of that relationship will bring a valuable service that together we can deliver to Microsoft and to our customers." Nash pauses a moment and adds: "It's a joint effort between the hackers and Microsoft."

Nash seems to be searching about for an analogy for this unusual collaboration. "Most of the success of Microsoft has been in our ability to deal with technical people in the industry, and usually that's been with the ISVs [independent software vendors]," he said. "In a sense the hackers have been kind of like that, in that they're technical people and they are valuable in giving us feedback to make our products better."

Microsoft is not alone in its attempts to come to terms with the hackers. Indeed, The Dinner was just the latest, and perhaps highest-profile, encounter of many meetings going on across the industry as companies as large as Novell and Sun Microsystems make efforts to get the hackers on their side.

"Hackers are almost part of our engineering staff," said Patrick Taylor, a vice president at the Atlanta-based Internet Security Systems Inc. "All of them are fundamentally trying to make the digital world a better place in which to live." ISS, which provides software products that analyze Unix or NT for security holes, has long embraced the hacking community, Taylor said.

"I like the fact that we have sort of a freelance engineering team out there," said Michael Simpson, director of network services at Novell Inc. (Provo, Utah). "There's no way that I could have a staff sit around and pontificate about every possible way that you can hack into our environment, so we have to talk to the hackers-we can't ignore them. And we appreciate what they do."

Simpson does concede that the relationship is a "strange" one, however. "I won't sugarcoat this. Every one of these [hacking] situations is uncomfortable for us-really uncomfortable. You could question their motives, their intentions or the methods by which they communicate these hacks [to the public], but the end result is that our customers always get a better product." Simpson adds in a hushed tone, "We're kept in check. We can't put our guard down."

The NT attack is far from being the first assault on a commercial operating system; the OS is only the latest in a long string of products torn apart by underground engineers. The schedule of the Black Hat Briefings provided ample evidence of this history, covering the A to Z of network-security problems for every flavor of Unix, Novell's Netware and everything Internet, from redecorating Web pages to flooding e-mail boxes.

Indeed, in the age of the Internet, hacking is part of the fabric of business, said Kuljeet Kalkat, Internet engineering group manager at the Sunsoft subsidiary that develops the Solaris operating system for parent Sun Microsystems Inc. (Mountain View, Calif.). "Security is something that the mainframe world knew well and did well, but they did it in a way of being a strong policeman. You could not get through their glass house," Kalkat said. "But now, with distributed computing-where the network is the computer-security has become a very large issue."

Along with Microsoft, both Novell and Sun have felt compelled recently to step up their security interests in myriad ways. For starters, the three have relatively new security-related Web pages (, and

"We've spun off an entire part of the company to address security," said Kalkat. "We're trying to attack security at every level-from a networking level, product level, user level, server level and also at the levels of Internet security, transactions security and commercial security. We're all in the middle of something that's going to be the predominant topic for the next three years at least."

For its part, Novell now has a full-time engineer exclusively dedicated to plugging security holes, as well as a marketing person specifically responsible for handling communications about any vulnerabilities in Novell products or even in those of its competitors. "Sometimes even if a Microsoft product has a vulnerability, in a mixed environment some of our products may be dependent on their products," explained Simpson.

Taylor at Internet Security Systems thinks it was only a matter of time before Microsoft got yanked into the fray. "I think we've seen a dramatic shift for Microsoft," he said. "It used to be that they wouldn't even have acknowledged these kind of bug postings on the Net. Maybe they would have put a fix in the next service pack or something. But what we've seen lately is that they're more actively acknowledging their problems and implementing fixes in a pretty short period of time. They've converted over to the modes that the Unix vendors have traditionally operated in: if there's a problem, we'll address it right away."

Benevolent anarchists
As defenders of the digital realm the hackers are a curious lot, to be sure. Most of them gave up on formal education just short of finishing technical degrees at large universities-often, it seems, out of boredom. They discovered computers and modems early, usually as young boys (virtually the entire hacker community is male) or as teenagers. Few of them seem to care about money, and none seem to mind the occasional notoriety that comes with breaking into a Web site or posting a major security bug on a supposedly secure product. And all of them have a wicked streak of anarchism.

The press often terms them-incorrectly-"crackers," but within their own hacking community they are the "white hats," the good guys. This is the breed of hacker that likes to rip technology apart to see how it works and then show the world how they did it-not for personal gain but for the common good, like cyber Robin Hoods. Reaching that broader audience is easier now, thanks to the personal home publishing machine called the Internet.

The strange pseudonyms these underground engineers give themselves is a revealing detail. At one level it shows a deep concern with guarding their privacy from the networks they know are vulnerable. At another level it is simply part of the mystique-one with hints of role-playing games, fantasy literature and conspiracy theories plus, perhaps, a dollop of self-importance.

The way this loosely allied community suddenly jumped on NT suggests that it has risen from being a corporate gadfly to something more substantial-a technical counterculture, a digital hippie militia. "Security is an area of computer science that has gotten limited attention in the past, partly because security has only been available on Unix," said Microsoft's Nash. "But now that NT has become a mass-market product and security a mass-market function, these guys suddenly have a forum to talk."

But some think the hackers' angst goes deeper. "These guys like Mudge and Hobbit are old-school hackers," said Jeff Moss, a 27-year-old former hacker and the founder and president of the Black Hat and DefCon conventions, the largest annual gatherings of hackers. "They detest the slick, glossy spin-doctor lies that are coming out of corporate America in general, recycling old crap as new."

Perhaps because of Microsoft's eminence, NT has become their bête noire. "Some of the most devastating stuff going on out there now, and to come out of this [Black Hat] conference, centers on NT," Moss confirms.

Indeed, hacker fascination with NT may represent an inevitable backlash to the success Microsoft is having penetrating the corporate network with it. Figures published by International Data Corp. (Framingham, Mass.) show new sales of Windows NT Server growing by 85 percent last year, outselling new units of Novell's Netware 4.x by more than double. By contrast, all flavors of Unix enjoyed a combined compound growth rate of only 15 percent.

But to the winner belong the hassles. "If there's going to be wide deployment of an operating system-say, like a Morgan Stanley-they're going to want to know more than just what's written in the book that they got in the box," said Sunsoft's Kalkat. "I don't think Microsoft has much of a choice" but to submit to its trial by fire at the hands of the hackers.

Six burly guys
While some generalizations hold true, every hacker is unique. For Yobie Benjamin, any corporate pressure experienced in his role as an NT hacker pales next to what he endured as a student activist in his homeland of the Philippines. Benjamin attended the Philippines University, considered that nation's Harvard, during the Marcos martial-law years. At the age of 17 he and his compatriots decided to take on the dictatorial regime.

"We decided we would start with marches, which were met with support [from other students] over the newest tuition rates," Benjamin recalled not long ago. Soon the student group was establishing alliances with other universities, and had organized a nationwide boycott that drew half a million students. Then, suddenly, Benjamin found himself a political prisoner.

"One day I was just hanging out in Manila, and I was picked up by six burly guys," he said. Benjamin reports that he was strapped into a dentist chair, tortured and thrown into solitary confinement for several weeks. "My family thought I was dead- 'salvaged' as it was called."

Benjamin spent the long hours of incarceration going over in his mind the text of the Encyclopedia Britannica, which he had read cover to cover as a child. One day, just as suddenly as he had been detained, Benjamin was released. "It happened as curiously as I was arrested," he said now. "They came in and said, 'you can go.' And I asked, 'Why?' " He got no answer.

Some years later Benjamin traveled to Europe, where he married. In the early 1980s, he and his wife came to the United States, where Benjamin found himself drawn to the new world of personal-computer technology. "My first computers were the Tandy TRS-80, Morrow Designs' 26-Mbyte Discus and a Sinclair Research ZX80-all from the early '80s," he said. A quick study, Benjamin soon landed a job at the then-fledgling Lotus Development Corp. in Cambridge, Mass. "I very quickly realized that the engineers at Lotus were the ones who were making the money, so I quickly learned to do that too."

Somewhere along the line Benjamin got into hacking. "When I first started I loved to break the encryption on copy protection for games," he said. "I couldn't afford [to buy] them, so I had to." Following his stint at Lotus, he worked independently as a software engineer before signing on at Cambridge Technology Solutions a few years ago.

Political activism is a thread that runs through Mudge's life as well, though not in so dramatic a way as in Benjamin's. "I guess you could say I'm a leftover left-wing liberal," he said. "I'm concerned about my privacy. I'm concerned about my personal information being stolen and posted on a Usenet group with 500,000 people reading it. You're putting your faith in people who make technology, and you shouldn't. They don't care; they lie to get the money."

Mudge's hacking work is unsystematic, driven by a kind of anarchistic desire to flout authority. In his eyes a commercial product comes with an unacceptable proposition: "Here's something that you own that you can't modify or do anything to, and we've gone to lengths to make sure that's the case," he said. "But that's not right. I paid you the money. It's mine now. If I buy the Mona Lisa, I have the right to draw on it; and if I buy software I have the right to delete it-chop it in half-do anything I want to with it."

Mudge said his first technical mentor as a boy growing up in Alabama was his father, a chemist. Together the two would pore over their Apple II+ and the Tektronix 4051, and even build new computers to aid in his dad's work. Mudge is also a musician with two degrees in classical composition. After gravitating toward the Massachusetts Institute of Technology and the Boston area, he taught computer science at Northeastern University and Emerson College and recorded a CD for a record company. "Don't look for it-it didn't sell," he said with a chuckle.

He and six others founded the L0pht pretty much out of necessity in the early 1990s. "Everybody had apartments or rooms, but the 'significant others' were complaining that there were computers in the bathtub, software strewn all over the place and reams of computer paper all over," Mudge relates. "So we decided if we all chipped in we could afford a loft space."

The group soon began running a well-known BBS called "Black Crawling Systems." They were also earning a reputation as quality hackers, finding weaknesses in Novell's and Bellcore's security products.

Like Mudge, Hobbit's fascination with engineering began when he was a boy working alongside his father-by coincidence, another chemistry researcher. His dad, too, imparted a love for understanding how things work. "I like to see how someone else solves a problem, and then become better at solving problems myself," said Hobbit.

Hobbit spent some 18 months working toward an EE degree (he won't say where), but "bailed because I wasn't learning anything really useful or practical there." Eventually, he too settled in the Boston area, where he runs his consulting business, operates his Avian Web site and hangs out at MIT. "The culture around there is a big reason I moved to the area," he said, "not to mention that there are lots of folks who like hacking computers, locks, electronics or even buildings."

Hobbit's extensive experience in computing and networking has lent him an aura among the community of being a true hacker guru. "I cut my teeth mostly on mainframe-class machines," Hobbit said. "Tops-10, Tops-20, VMS 20 [Digital Equipment Corp. operating systems], and eventually to Unix." He learned enough to get around on DOS and Windows PCs at FTP Software, an early Internet company, but "began to clue in about how lame these [systems] were as I had to help people maintain and fix them." In 1979 he discovered the Arpanet, via MIT.

Hobbit's work ethic is quite rigid, if unconventional. He laughs when asked if he works from midnight to morning in stereotypical hacker fashion. "It's more like 11 a.m. to 4 a.m., punctuated by coffee and occasional bursts of attempting to have a life," he said.

Across the country, Washington-state hacker Nihil has a day job too. "Quite a few of us are security professionals," said Nihil, who will not divulge his real name or employer. "We gravitate toward companies that are willing to listen to the security mantra. Since I work in an engineering environment, my day usually starts around 10:30 a.m. I have been known to work long hours, but that's a byproduct of the fact that I like what I do. My work allows me to explore a lot of what I am interested in, so I don't have to separate work from my personal interests too much."

Jeff Moss, the Black Hat convention organizer, exemplifies how many hackers become an uneasy mix of engineer and anarchist. "During my high school years my buddies and I practically became engineers," said Moss, who grew up in the San Francisco Bay area amid the Berkeley scene. "We got into phone networks and phone phreaking [hacking the telephone network], and discovered how the phone networks work, and how the infrastructure works, and how the copper companies work and how computers are put together."

Asked to describe the hacking passion, Moss speaks of an urge to get beneath the surface of things, to tease out the secrets of how the world really works. "You're driving down the street, and you wonder how the electrical system is working, how the power grids work," he said. "You're seeing one more layer deep all the time. Pretty soon your fundamental way of looking at the world changes."

Not so different, perhaps, from how an aboveground engineer sees the world-with one exception. Engineers long to understand how things are made so they can re-create them; hackers yearn for the same knowledge so they can tear things apart. The latter view goes hand in hand with a deep-seated belief in the vulnerability of all things man-made.

Everything is hackable, said "A.J." Effin Reznor, a well-known Netware hacker. "When breaking into something, if you can't quickly pick the lock on the door, go around the door," he said. "Check the back door, check the windows, check the roof. Tunnel if you have to." Reznor cites a friend who as a teenager was placed in a juvenile detention center. He was "out within a few hours. While air-locked doors may have presented a problem for him, popping the ceiling tiles obviously didn't."

Sun's Kalkat compares this mentality to a squirrel with "nothing to do but to get into a birdcage to get a seed. If it spends all day, it will find a way"-no matter how cleverly squirrel-proof your birdcage design.

Mudge believes that hacking is no passing fad but rather, an ongoing force that will grow and change as technology itself changes direction. "People are pulling their physical lives into the digital world," he said. "There's going to be a cross of hardware and software hacking coming out in the near future on smart cards [and] on biometric security devices."

For instance, "operating systems will one day make the switch from passwords to special security access cards," said Mudge. "We've already been ripping apart cards like that at the L0pht for some time. We've also set up a microwave network we built out of trash parts to make 'feed horns' [components of a satellite dish]. We're able to intercept stuff and decode it-we're even able to encode it-and bypass the security in these things.

"Software, hardware, network," Mudge sums up. "Whatever they throw at us-we're all set."