Anomaly Detection Systems - Part 2

by Thuull

In my last article, "Anomaly Detection Systems - Part 1" in 17:3, we explored the general concepts behind intrusion detection, a means of classifying Intrusion Detection Systems (IDS), and a brief outline of a simple passive/host-based intrusion detection system on a Linux platform.

This article will outline a couple of different ways to accomplish anomaly detection on large heterogeneous networks cheaply and efficiently, from the passive/network-based angle.  We'll also discuss signature-based IDS systems' usage in conjunction with anomaly detection to create a well-rounded overall intrusion detection solution.

I can't stress enough the necessity of understanding the traffic flow on your network.  If it is your mission to protect that network, how can you protect it if you don't understand what is there?  How many web servers do you have?  What are their IP addresses?  Do they use SSL (443/TCP)?  HTTP (80/TCP)?  Find out... only in knowing what belongs on your network can you spot what doesn't belong.  If you can't spot what doesn't belong, then what doesn't belong is just going to keep on not belonging, without you knowing about it.

I discussed in my last article the fundamental vulnerability that exists in all attack signature-based intrusion detection systems: they cannot "see" zero-day exploits.  Generally, there is a period of about one week to nine months between the time that a new exploit is created for a recently discovered vulnerability and the time that the attack signature for that vulnerability finds its way into your attack signature-based IDS.  So, until you have the signature, what will your IDS system tell you?  Absolutely nothing.  Won't even see it.

A solution to this fundamental problem?  Learn your network, know what belongs, highlight what doesn't.

Say your NNTP server has only two ports open: NNTP (119/TCP) and SSH (22/TCP).  An attacker doesn't know that those are the only two ports open on it until the attacker probes the machine.  If the attacker is smart, he'll hit the machine with one packet a day from a different IP address every day.  Will your attack signature-based IDS show a single SYN packet to port 23/TCP?  I don't think so.

Anyway, back to that solution... collect all traffic that crosses your network at a chokepoint, then bounce that traffic off of a filter set that siphons off all traffic that belongs.  What you have left is everything else.  You'll find in investigating this "everything else" that about 90 percent of it turns out to be system misconfigurations or what-not on either your end or the other end of the comms stream.

However, the remaining 10 percent are malicious.  In the above example with the NNTP server, write filters that ignore port 119 and port 22, and have the system show you everything else.  You might even want to only filter out incoming traffic to those ports that are from IP addresses that you know should be using those ports.  Everything else is suspect.

If you're paying attention, you're probably screaming right now: "What about an exploit against SSH or against NNTP?"

Well, two answers to that question.  Yes, incoming traffic that is malicious can match a filter that you put in as "normal" traffic, but 99 times out of 100, more than one port is going to be checked on the system before an actual exploit is launched.  That, and someone probing for port 119/TCP on your systems will most likely look for it on other systems as well, which should show up in your system because you're not filtering 119/TCP from other machines... only from your NNTP server.

The second answer: this is where attack signature-based systems come in.  If the exploit used is old enough, your IDS system will probably have a signature for it, and will flag the attack.  This covers the hole created when an attacker's traffic matches valid traffic that you would expect to see, to a certain point.  This does not provide a solution for when an attacker uses a zero-day exploit that matches expected traffic.  Still though, you will probably see traces of the activity on other machines.

Do you use firewalls?  I bet you probably do, unless you're running a small network at home where you can easily keep up with all the latest vulnerabilities.

An effective anomaly detection system can be "built" with the firewall(s) that you're currently using.  Leverage your firewalls to be your eyeballs into what's coming in and going out of your network, not just as a simple barrier.  Every firewall platform that I am aware of has the capability of not only logging traffic, but of filtering information that is displayed in the log files.

Generally, this is used for troubleshooting network issues... did the traffic ever reach the firewall?  Run a filter on the logfiles to look for that IP address, if it's not there, it didn't make it to the firewall, etc.  But, those filters can be used the other way too... instead of writing a filter to show a specific something, write a set of filters that hide a set of specific somethings... those specific somethings being all traffic that belongs on your network.

Filter out all traffic to port 80/TCP on your webservers (and 443/TCP if you're using SSL), port 20/TCP and 21/TCP on your FTP servers, 53/TCP and 53/UDP on your DNS servers, etc.

Remember, you'll want to be able to see port 53/TCP and 53/UDP connects to everything except for your DNS servers, so write your filters specifically for individual machines.  Normally, firewall systems will allow you to save filter sets... use them.

Check them every day.  Log the anomalies in a database, to look for trends later.  I once identified a very patient fellow this way, plugging away at the network with two or three packets a day against a different port from a different IP address every day.  All put together, they added up to a portscan... amazing.  By the way, on that one, ISS RealSecure never saw a thing... of course, you can't blame it; that's not what the IDS systems that are out there today are designed to find.

There are two other ways to accomplish this in passive/network-based mode.  You could put Linux machines out in front or behind your firewalls (at prominent chokepoints), or off of monitored switch ports running ipchains in accept all but log mode, run logcheck against your logfiles every hour and have it report anomalies to your email.  You could even write your ipchains rules to do the filtering for you... i.e., accept and don't log 80/TCP to the webservers, but accept and log all else.  That would keep log files down some.  Or, you could take the Shadow IDS system from the CIDR project and revamp it a little.

The Shadow IDS system is already designed to suck in all the traffic on the network via tcpdump and store it in massive logfiles for after the fact analysis.  Filters are then written using normal tcpdump syntax to grep out of those logfiles traffic which matches certain criteria... i.e., you can write a filter to run through and check specifically for individual attacks.

However, with a little modification, you can rearrange the system to instead of going in and pulling out the stuff that you want to see (which requires that you know what you're looking for before you look for it), you can have it go out and filter out all of the stuff that you know belongs on the network and report to stdout whatever is left.  Hello, anomaly detection.

Let's talk briefly about limitations.  Anomaly detection is not the end all answer here.  I strongly advise a combination system.  The methods that I've outlined do not include things like fragmentation reassembly, MTU size, low TTLs, etc.  However, I guarantee that with a combination system, you will see far more than you would with an attack signature-based system alone.

As far as attack signature-based IDS systems go, if you are looking for a system to use in conjunction with this sort of anomaly detection, my suggestion would be the Dragon IDS from Network Security Wizards.  I'm personally very impressed.not only with this system's ability to find and identify known attack signatures, but its usage of more all encompassing "built-in" broad-based filters that are based upon parameters that catch certain "classes" of attacks which share similarities with known attacks.

Essentially, this means that in some cases, new zero-day exploits that are modifications of known exploits, or work within similar parameters, will be at least highlighted for further analysis.  And that's just the built-in functions... you can write your own rulesets for it that turn Dragon IDS into an anomaly detection system per the style above, simply by having your rulesets ignore everything that you expect to see on the network.  Take a look at it, they're doing some neat things.

My point here I guess is simply this: You can't go into intrusion detection expecting that you know what to look for.  If your system(s) get compromised via a vulnerability in a service and not by some misconfiguration error that you've made, one of two things has happened.  Either you are stupid and didn't patch an announced vulnerability, or someone used a zero-day exploit against you.

(An academic note here: from statements earlier in this article, you should be able to surmise now that I believe that attack signature-based systems are only useful to stupid people [Caveat:  That's mostly a joke, there are valid uses for attack signature-based systems for smart people].)

If you are smart and have patched everything that needs patching, you're still not secure, but you can at least see the attack coming from the other smart guy sitting out there somewhere.  And if you're really smart, then your systems are probably tight enough that it's going to take that other smart person longer than he wanted to in order to compromise your network.

This gives you the opportunity to do something about it before anything ugly happens.  Let's face it, it's like a big game of chess... sometimes the other guy is smarter than you are, and you get to learn something.