Web Server Discovery Tool by Boris Loza This project started when I decided to find all the web servers on my network. One can do this by running 'nmap' to identify all open HTTP/S related ports: 80, 8000, 8080, or 443. But 'nmap' is known for crashing servers (just a couple of misbehaves to mention: killing 'syslogd' on Solaris, Cisco's DOS, etc.) Therefore it is not allowed in some organizations. Moreover, even if the ports in question are open, 'nmap' doesn't give you the type and the version of the web server listening to it. 'nmap' can also trigger the IDS and page the information security group! Using commercial tools like ISS Network Scanner or CyberCop to find all web servers on the network is cumbersome, time consuming, and IDS detectable. Taking all this into consideration, I decided to write my own tool for discovering all web servers on the network. I wanted this tool to be easy to run, not to use "crafted" TCP packets, be efficient, quick, and provide as much information about discovered web servers as possible. We intended to run this tool periodically, like a war dialer, and to do this even during business hours (before users shut down their workstations to go home). I wanted to create a tool as efficient as possible with minimum network and server impact. In this article you'll see what I eventually came up with. The Tool First, let's understand a little bit about how a web server and a browser communicate. The browser or client generates request headers and sends them to the web server. The server receives the request headers, translates them, and generates the response headers. These response headers have to include information specific for the web server that will allow both the browser and the server to communicate. I decided to use this information to create the tool. In the heart of the tool is the following Perl Code: 1. use HTTP::Response (Encapsulate HTTP responses) 2. use LWP::UserAgent (Dispatch WWW requests) 3. $ua = new LWP::UserAgent (User agent object is created) 4. $ua->agent('Mozilla/5.0') (Using Mozilla/5.0 as agent's name) 5. $req = new HTTP::Request(GET,"http://$ARGV[0]") (Encapsulate a request using GET method) 6. print $headers = $ua->request($req)->headers_as_string (Read response from the web server) I use Perl's 'libwww-perl' library for WWW access (#1 and #2). This library will provide the API for writing my own WWW clients. First I need to create a request header (#3 and #4) by specifying the name of the web browser the request comes from. Now I can send the request to the server using the GET method (#5). Strictly speaking, I can use any agent's name here, for example agent('Foo'). This doesn't matter, since I need just one response from the server and I am not going to continue the session. Now I can print everything that comes from the server (#6). After naming this little Perl script as 'ws.pl' and running it against one know web server, I've got the following output: C:\>ws.pl 192.168.0.40 Date: Thu, 04 Apr 2002 15:27:06 GMT Accept-Ranges: bytes Server: Microsoft-IIS/4.0 Content-Length: 56 Content-Location: http://192.168.0.40/Default.htm Content-Type: text/html ETag: "f82f8972cf9ac01:5ee8" Last-Modified: Mon, 19 Feb 2001 23:55:33 GMT Client-Date: Thu, 04 Apr 2002 15:28:43 GMT Client-Peer: 192.168.0.40:80 X-Meta-Postinfo: /scripts/postinfo.asp As I expected, the web server strikes back by sending all necessary information that will be needed for the session. If no HTTP web server is listening on port 80, the output will be: C:\>ws.pl 10.56.53.27 Client-Date: Thu, 04 Apr 2002 18:38:39 GMT In this article I am not going to explain all response headers from the output. For anybody who is interested, please refer to RFC2616. For the purpose of the script, I am interested only in one: Server: Microsoft-IIS/4.0. This is the name of the web server I connected to. So I can modify #6 of the script to display only this response header: print $headers = $ua->request($req)->header('Server'); C:\>ws.pl 192.168.0.40 192.168.0.40 Microsoft-IIS/4.0 After understanding the concept, I started working on something more useful. Below is a listing of the complete tool. This tool will discover a single web server or all web servers on a given subnet. The default port to scan is 80, but you can specify any port you wish. To run the 'ws.pl' tool against a single host type: C:\>ws.pl 192.168.0.2 Microsoft-IIS/4.0 Or specify a different port (port 80 is the default): C:\>ws.pl -p 8000 192.168.0.58 192.168.0.58 HP-Web-Server-3.00.1696 You can use both an IP address and a DNS name here. For the verbose mode use the '-v' option. The following command will print all response headers for the host 192.168.0.2: C:\>ws.pl -v 192.168.0.40 Date: Thu, 04 Apr 2002 15:27:06 GMT Accept-Ranges: bytes Server: Microsoft-IIS/4.0 Content-Length: 56 Content-Location: http://192.168.0.40/Default.htm Content-Type: text/html ETag: "f82f8972cf9ac01:5ee8" Last-Modified: Mon, 19 Feb 2001 23:55:33 GMT Client-Date: Thu, 04 Apr 2002 15:28:43 GMT Client-Peer: 192.168.0.40:80 X-Meta-Postinfo: /scripts/postinfo.asp To scan the whole class-C network use the -C option. For example, to discover all web servers running on port 80 on the subnet 192.168.0, type: C:\>ws.pl -C 192.168.0 192.168.0.10 Microsoft-IIS/5.0 192.168.0.21 HTTP/1.0 192.168.0.33 IBM_HTTP_Server/1.3.6.4 Apache/1.37.7-dev (Unix) 192.168.0.34 IBM_HTTP_Server/1.3.12 Apache/1.3.12 (Unix) 192.168.0.40 Microsoft-IIS/4.0 192.168.0.45 Netscape-Enterprise/4.1 192.168.0.82 ApsyServer 1.0 HTTP/1.0 on the host 192.168.0.21 is a web interface for HP printer. To scan the same subnet looking for web servers listening on port 8000, type: C:\>ws.pl -p 8000 -C 192.168.0 For help, type: C:\>ws.pl -h Use: ws.pl [-v] [-p port] hostname ws.pl [-p port] -C IPaddress Conclusion After running this tool for the first time I found three times more web servers that I had on my list of the "official" web servers. The 'ws.pl' has proved to be very efficient. Now I run it periodically to discover rogue web servers without any network or server impact. What else is important, I know what every line of this script is doing and can customize the script for my needs. --begin code-- #!/usr/bin/perl # # ws.pl # Web Server Discovery Tool. Boris Loza, 2002 # Discover Web Servers. # Hostname can be specified by an IP address or a DNS name. # # Options: -v :verbose # -p :specify a port (default 80) # -C :scan class-C subnet # # Examples: ws.pl -v 192.168.10.3 # ws.pl example.com # ws.pl -p 8000 example.com # ws.pl -C 192.168.0 # ws.pl -p 8000 -C 192.168.0 # # use HTTP::Response #Encapsulate HTTP responses # use LWP::UserAgent #Dispatch WWW requests # $ua = new LWP::UserAgent #User agent object is created # $ua->agent('Mozilla/5.0') #Using Mozilla/5.0 as agent's name # $req = new HTTP::Request(GET,"http://$ARGV[0]") #Encapsulate a request using GET method # $headers = $ua->request($req)->headers_as_string #Read response from the web server use HTTP::Response; use LWP::UserAgent; use Getopt::Std; $usage = "Use:\tws.pl [-v] [-p port] hostname\n\tws.pl [-p port] -C IPaddress\n"; getopts ("C:hp:v") || die $usage; print $usage if $opt_h; my $port = 80; # Default port to scan if ($opt_p) { $port = $opt_p; } my $host = $ARGV[0]; # Create Request Headers my $req = new HTTP::Request(GET, "http://$host:$port"); my $response = $ua->request($req); # Use verbose mode. For single host only! if ($opt_v) { print $response->headers_as_string; exit; } # Scan Class-C Network $count = 1; if ($opt_C) { (my $subnet, my $node) = ($opt_C =~ /(\d+\.\d+\.\d+)\.(\d+)/); if ($node) { print $usage; exit; } while ($count <= 254) { my $host = "$opt_C.$count"; # Skip unreachable hosts for speed (for Windows users only) # Comment out for UNIX! if (`ping $host` =~ m/(timed out)/) { $count++; next; } my $ua = new LWP::UserAgent; $ua->agent('Foo'); my $req = new HTTP::Request(GET, "http://$host:$port"); my $response = $ua->request($req); if ($response->header('Server')) { print $host, "\t", $response->header('Server'), "\n"; } elsif ($response->header('Proxy-Agent')) { print $host, "\t", $response->header('Proxy-Agent'), "\n"; } elsif ($response->header('Title')) { print $host, "\t", $response->header('Title'), "\n"; } elsif ($response->header('Client-Peer')) { print $host, "\t", "Web Server not found, but port $port is open.\n"; } $count++; } exit; } if ($response->header('Server')) { print $ARGV[0], "\t", $response-header('Server'); } elsif ($response->header('Proxy-Agent')) { print $ARGV[0], "\t", $response-header('Proxy-Agent'); } elsif ($response->header('Title')) { print $ARGV[0], "\t", $response-header('Title'); } elsif ($response->header('Client-Peer')) { print $ARGV[0], "\t", "Web Server not found, but port $port is open.\n"; } --end code-- Source Code
Return to $2600 Index