July 23, 2003

You've been hacked, what do you do in the first hour? page 2 of 2

The next step is to review group memberships to ensure that no new users have been given more group memberships and therefore more privileges than they should have. Pay particular attention to the administrators group, since this group membership essentially allows users to do anything they want to the system. The easiest way to approach this is to go to each group and review its memberships. If a user account has a membership it shouldn't have, you should remove the membership and flag the account for further evaluation.

You'll also need to review the access control lists (ACLs) for the system. This is probably best done with a tool such as SomarSoft, which allows you to capture all the ACLs from the system and output them to an easily readable format. You're looking for entries that appear to give more access than is appropriate.

If you do find log entries that you aren't comfortable with, the system should be isolated until the source of those entries can be identified or until you can provide detailed observation of the system. Log entries can indicate successful and unsuccessful attempts to use security, but most frequently, they're set to capture only information pertaining to failed attempts. It's quite possible that you'll quit seeing failure events in the log because the hacker has gained access.

An important step is to verify that no Trojan horse programs are loaded and running on the systems. These are typically identified by antivirus programs, but they aren't always detected by every scan engine. If you want to check your systems with an alternate antivirus engine, try Trend Micro's online scanner.

Rebuild the compromised systems
After compromised systems have been discovered, your next challenge is to figure out what to do with them. You will have determined which systems are affected and reviewed security settings to ensure that the intruder hasn't created another opening. However, dealing with compromised systems is difficult because the best way to handle them is impractical for most environments.

Ideally, a compromised system would be rebuilt from scratch or restored from a backup that was made before the compromise happened. Rebuilding from scratch is exceedingly time consuming. However, determining which backup to use for a restore can be difficult because it may not be possible to pinpoint the exact time of the compromise. As a result, you must guess which backup is not compromised and then reevaluate it after the restoration is done. If the compromise is still evident after the restore, you must use an even earlier backup.

Rebuilding compromised systems is painful but necessary to complete before the systems are returned to service. Because of the urgency and the pain of a proper restoration, many organizations choose to patch a system, close vulnerabilities, and either hope that the system has been cleaned or come back to the system later.

Patch vulnerabilities
Before you reconnect the network to the Internet, you need to patch all the vulnerabilities you can. In most cases, it won't be clear what vulnerability was exploited to gain access to your network, so the goal is to patch them all.

In some environments, you'll need to have patches approved before they can be implemented. But in many other environments, patches can be installed as needed. In either case, you should load all the patches you can. In particular, you should apply security patches to all systems, making sure that your firewall is up to date, external systems are appropriately tightened, and that all the internal systems are patched.

If you skip this step thinking you'll get to it later, you may find that your network is compromised again shortly after you've cleaned everything up. The amount of effort required to really clean up from a security breach makes it imperative that you do everything that can be done to prevent further compromise.

Reconnect your systems
The physical process of reconnecting the systems to the Internet sounds trivial: You plug things back in and they start working. Of course, it's what happens as soon as the systems are logged back in that is important.

As soon as the devices are reconnected, they'll start making outbound connections through the firewall. You need to monitor those connections to see where they're going. Most people discover that a lot of applications and utilities on their workstations are talking to the outside world.

Every connection should be investigated to see whether it's valid. This is particularly true for connections that aren't destined for port 80 (the HTTP port). To investigate the destination IP address, you can use nslookup. Type NSLOOKUP at a command prompt. Next, type SET TYPE=PTR and press [Enter]. Finally, take the IP address, reverse it, and append in-addr.arpa. After you enter this whole thing, you can look for an answer. For example, if the outside address is 216.37.52.229, you would type 229.52.37.216.in-addr.arpa. If you don't get the answer you want, start dropping off the end of the IP address until you do. For example, our next command would be 52.37.216.in-addr.arpa.

Cleaning up pays off
When a network has been compromised, a lot of work is required to ensure that it doesn't happen again. But despite all the effort, it's not a good idea to cut corners. No cost can be placed on the problems that can result if you don't completely clean up compromised systems and monitor the results after they are reconnected.

Back to the July Newsletter