Whitedust Security Interview: Paul Watson By Mark Hinge and Peter Prickett (Thu, 28 Jul 2005 20:09:29+0100) Paul 'Tony' Watson is best know for his discovery of a flaw in TCP/IP last year. Once working for the US Air Force and then for the US government doing 'computer counter-espionage', he now works for the leading and most popular search engine, Google. WD> First off, there seems to be some confusion, is your name Tony or Paul? And why not stick with one of the other? I am asked that question all the time. My real name is Paul Anthony Watson. Growing up all my friends and family called me 'Tony'. However, in business meetings and professional settings I typically use my real name Paul. The easiest answer might be, after a person has tossed down a few coronas with me at CanSecWest or DefCon, they call me Tony. WD> How long have you been involved in internet security? How did you get started? My introduction to computer security came around 1983-84. My first computer was a Commodore-64. I loved game programming, but I was also pretty 'financially challenged'. I got involved in the early 80's warez scene in order to get better tools for programming. Since most people traded games, I had to rip games in order to trade for other software like Graphics editors, Machine Language monitors, assemblers, and stuff like that. I also did a little red boxing and blue boxing in order to call BBSs over in Europe where the scene tended to have a better tools-to-games ratio. Then one day a friend told me about the C programming language. The C compiler for the Commodore was outrageously expensive and I couldn't find a warez copy, but I wanted to learn the language. I made a deal with a friends mom who was in school for Computers at Purdue; I would help her write her programs for her computer classes if she let me have use her Unix account so I could learn Unix and C. I fell in love with Unix immediately. In 1993, I joined the U.S. Air Force as a computer programmer helped develop command and control systems for NORAD. That's when I started serious 'recreational' hacking since I finally had full-time Internet access. Eventually, the Air Force OSI took notice of my 'recreational activities' and recruited me do computer counter-espionage work. I used to joke with friends years later that it was like a 'license to hack' (a reference to James Bond's 'license to kill'). I stopped all my 'recreational hacking' at that point, because I could see that things were changing. Hacking was no longer just a nuisance in they eyes of Government, it had become a threat. I had several friends end up with criminal records because they were unable to recognize this or didn't take it seriously enough. That is a very abbreviated version of my start in Computer Security. I could talk forever on the early days of hacking and the Internet, but I think it would probably fit better in a book than an interview. WD> How long have you worked at Google? Is it is a cool as everyone thinks it would be? I came to work at Google late last summer. It gets a lot of media buzz about being geek-sheik and super cool. I have worked at some really cool places before Google, but Google is so much more incredible than any media article or Slashdot post could ever describe. The best phrase I can think of would be nerd-nirvana (or should it be nerdvana?) WD> How did you get the Google gig? Exactly what is the nature of your gig? I submitted my resume on a fluke around June of 2004. The interview process consisted of two hour-long phone screens followed by two days on on-site interviews. I'll never forget the first phone screen. It was schedule while I was in Las Vegas at DefCon, and I had been drinking all morning. I was in the pool with some friends when they asked about where I worked. Suddenly I remembered that Google was going to be calling me for my phone screen at any second, and I bolted from the pool and ran into my room to find my cell phone. The phone rang just as I got to it, and for the next hour I had the toughest phone screen of my life (while mostly intoxicated). My focus at Google is Network Security, but at any given time I may find myself configuring firewalls and routers, pen-testing new applications, evaluating encryption and hashing methods, or writing code to make my job easier. It's a very demanding job that really requires a broad knowledge of every possible aspect of the security field. I can honestly say that the Security Engineers that I work with at Google are the most skilled and talented people I have ever worked with. WD> Who did you work for pre-Google? I worked for a great company called Rockwell in Milwaukee, Wisconsin. Most of the time when people hear Rockwell, they think of the Space Shuttle, but they are a really large company that is involved in a lot of different things. It was a really great place to work and I really miss the team that I worked with up there. WD> Apart from Slipping in the window: TCP reset attacks (which we shall come to shortly), which of your projects are you proudest of? Which is of the most use to the masses? I am really proud of my work in the Air Force. I worked on many projects, but one project in particular was called CTIS (Command Tactical Information System). It collected data feeds from many sources and did correlation and other back-end work and provided that data to Mac terminals that displayed it to end users and also on huge screens in the RAOCs (Regional Air Operations Centers). It was a lot like you saw in `War Games' but with much better graphics. Our group won the MIT Brunnell award for Macro-engineering for that effort. While not open source, I think a program that contributes to the National Defense is certainly useful to the masses, even if they never get to see it. WD> What was the role of the Hammersmith project? How does Cygnus relate to it and when will it be ready? Cygnus is just a new name for Hammersmith really. When I worked at my previous employer, they had several really serious virus/worm infections that quite literally shut the entire network down. Hammersmith was basically a darknet/blackhole system that I wrote from scratch. The thought I had, was that if I can observe 'normal background static' on the network when everything was normal, I could more easily detect worm infections as soon as the first couple of systems were infected. One of the problems, however, was that the network was huge, and spanned the globe. So I created a database of all the networks, critical systems, where they were located, who managed each system, their contact information, and many other bits of information. When the system detected suspicious activity from a host, it would collect lots of critical details such as MAC address and NetBIOS information, and combine that with the static information in the database and then notify the administrator of that particular system. The notification provided the administrators with detailed information containing everything needed to rapidly respond. As a result, we were able gain a huge time advantage in our reaction to these events, and I don't recall ever having another event bringing down our network after it was implemented. It also proved almost equally useful in the detection of mis-configured software and systems. Since the system was very customized to my previous employer, I wanted to rewrite all the code into something more generic and usable by anybody, which is when it was renamed Cygnus. But since I have been at Google I have been really busy with other projects that have consumed a large portion of my time. I hope to get around to working on it again soon. WD> You had your fifteen minutes of fame exposing Cisco Systems flaws. Exactly were the problems? Have they now been solved to your satisfaction? It wasn't any one problem. It was a system of problems that really created a much more dangerous problem. I am still amazed how few people really understand all the issues that were involved in the paper. 1. The assumption that TCP Sequence numbers need to be exactly on the mark to make a TCP injection attack work. In reality, as long as you are within the TCP Window size it will be accepted. 2. Then there was the fact that RST packets are acted upon immediately, even if they are out of sequence with the other TCP data being received. 3. Next, source port number selection was incredibly predictable but nobody had ever done much an audit of this. When I found that different versions of Cisco IOS all had a different starting source port after boot, and that they increased by 1 for each new connection... well... ouch. Since routers have very few outbound TCP connections (except BGP) it was easy to reliably predict what source port would be used. 4. Bandwidth had become cheap and available to the masses. 5. Finally, it is far too easy to find out what routers peer to each other. Some ISPs publish network diagrams online. Route reflectors make it easy to peer into the BGP backbone. Traceroutes and DNS reverse records help a lot as well. As far as the issue being solved, its not really important if I am satisfied that the problem is solved. The decision to use MD5 hashing is what the community implemented to work around this. Fine. Of course, its now possible to attack these same routers with resource starvation attacks by forcing them to compute a flood spoofed packets with bad hashes. They can address that issue if and when it hits them I guess. I was really shocked about Cisco's patent for their `fix' of the issue. It was a clever idea for a fix but it really broke my trust in them. The vendors (like Cisco) want Security Researchers to notify them about vulnerabilities before going public. But Cisco took advantage of this `blackout' period when they should have been fixing the issue, they decide to issue a patent. It is almost like insider trading on Wall Street and it has bugged me more and more as time goes on. WD> How difficult was it to get the people in a position to repair the problem to take you seriously? The US CERT team was a train wreck. They complete ignored emails from both Cisco and myself. When I say ignore, I really mean ignore. No response for months. They later claimed it was due to some re-organization where the paperwork got lost in the shuffle or something like that. That's a horrible excuse. I personally think that some reviewer looked at the title of my paper `TCP Reset attacks' and just assumed it was old issues being repeated and tossed it away without actually reading it. The NISCC in the UK was entirely different. After US CERT didn't respond, Cisco suggested I send the paper to NISCC. They read the paper, understood the impact, responded to me within hours and coordinated with me throughout the whole process. Those guys kick ass. WD> The Cisco fiasco received a large amount of media coverage in North America, but minimal coverage in Western Europe. Why do you think that was? It didn't receive much coverage in Western Europe? I didn't know that. I thought it was funny when I had friends sending me newspapers from Pakistan and other countries that had front-page articles mentioning me (even through I couldn't read the article because of the language difference). WD> April 2004 saw your profile rise massively. Do you still receive media attention? Do you find yourself looking for bigger security issues than you used to, in an attempt to outdo yourself? Well I am doing this interview, so I guess you have answered the question for me. Actually, I do get occasional calls to provide quotes for an article that some reporter is writing, or to be a guest on radio talk shows that have Computer security topics. In regards to all the media attention, I think that by far the coolest thing to come from all that attention was when I was Slashdot'd. That was like getting the key to the city from the Mayor of Geekville. Also the fact that there are hundreds of thousands of people named Paul Watson, and my Google ranking rose (and still is) to the top spot, even ahead of the Paul Watson who helped found Greenpeace. That is seriously cool. I was top ranked in Google for `Tony Watson' for a long time, but I am now second behind some musician in Los Angeles. The media attention was cool, but being Slashdot'd and top ranked in Google was much more flattering to me. I have no desire or need to try and "outdo" myself. My wife Cyndi enjoyed seeing all the press more than I did. To be honest, I found all the media attention to be interesting, but also very distracting. By interesting, I mean that I learned a lot about how the press works and how media cycles operate. I have an idea to write a system to index news articles from hundreds of sites, and try to identify how various stories like the `TCP Reset' story propagate. There seemed to be a lot of "word for word" sentence fragments that appeared in dozens and dozens of articles with no credit or attribution. It would seem that many reporters are nasty little plagiarists that only rephrase what the `top tier' reporters are saying with no original content of their own. I think it would be interesting to have a system that could recognize where various facts in a particular story originated and how they flowed outward to other subsequent articles. WD> How did Cisco systems react to your telling them that what they said would take 140 years would in fact take 15 seconds? I sent my paper to Matthew Franz and Sean Convery who wrote the original paper for Blackhat that started my work and asked them to peer review it. Matt was really helpful and responded back immediately. His response was something like, "oh, we really missed the boat on that one..." I felt bad Matt and Sean, as they really presented a great paper, but all the attention seemed to be focused on the fact that they got one issue wrong, rather than the other dozen issues they covered that were right. WD> When and why did you purchase terrorist.net? It used to be that anyone could query the Network Solutions WHOIS database directly without limit. I wrote a small program that ran every night immediately after the daily DNS updates were issued. It scanned about a thousand key words that I thought would be cool domains to own, or that I wanted to grab if some poor company forgot to re-register. Every time it found one of these domains open, it would page me with the domain name, and I could then just reply to the page and it would go out and register it for me. I got a lot of domain names that way, such as www.billgates.org, which is now a parody of Bill Gate's homepage that I put up in about 5 minutes just for a laugh. I registered terrorist.net sometime around 2000, and it used to be a spoof of a terrorist recruiting website. I had to take that site down in the summer of 2001 as I was moving from Washington DC to Chicago. Before I could get it re-hosted, the terrorist attacks of September 11, 2001 occurred and I decided not to put the original content back online. I later just pointed the domain to my own homepage. I think the name of my website contributed to the media interest in my research... It is much easier to get readers interested in a story when you mention words like terrorist, killing the Internet, and stuff like that. WD> Do you receive any unwelcome interest/visitors? When the site used to be (prior to 9/11) a parody site for recruiting and hiring terrorists, I had several people who didn't really get the joke and some actually sent me real resumes. On guy was a former Marine who claimed to have just retired and was an expert in explosives, and a sharpshooter/sniper. That was a little weird. WD> Has being the owner of a site with such obvious overtones caused you problems? The domain really freaks some people out, and I have had people who refuse to send email to me at my @terrorist.net address or afraid to type www.terrorist.net into a browser for fear the Government will come kick in their door and haul them off to Guantanamo. So I have other aliases for the site such as my name and my initials; www.paulwatson.org and www.paw.org. It is the same site, but with a different URL and Banner graphic to appease the tin-foil hat crowd.