Enterprise

Appendix A: Discovering and Recovering from an Attack

Sooner or later, one of your machines will be cracked. It's happened to all of us. Perhaps you were on vacation for a week when a new vulnerability was discovered, and you didn't have time to update your system before the crackers did it for you. Or perhaps you turned on some buggy software for "just one minute" to get something done and forgot to turn it off. Maybe you gave a friend an account on your machine, and his machine was cracked, which gave the cracker a step into your machine by watching your friend's movements.

So it's time to face facts: One day a crack will happen to you. In this section, we will teach you what to do about it. You should read this chapter carefully. Good luck!

How to know when you've been cracked

One of the most important ways to keep your machine secure is to have alert systems in place that let you know when your machine has been broken into. The less time crackers have on your system, the less damage they can do, and the greater your chances of kicking them off and repairing the damage.

Keep in mind that the more sophisticated the cracker, the less likely you are to know that your machine has been compromised. Skillful crackers will cover their trails well, making it difficult to realize that they have made any changes, and they can hide the fact that they are on your machine even when you're looking at it. By hiding processes, open connections, file access, and system resource use, crackers can make their actions almost entirely invisible. If they have cracked the root account, they can do pretty much anything they want at the kernel level to hide their presence.

You can, however, ensure that various systems are in place to help you detect that you've been cracked. Following are some methods commonly used to detect a cracker's presence in a system.

Intrusion Alerts
Ideally, you should be warned about attempted and successful attacks by any alert systems you've set up. If you have installed paranoid log monitoring software, you should notice when something new or unexpected appears, such as a new daemon being started or a new and previously nonexistent user logging in. If you have full-blown Intrusion Detection Systems (IDS) in place, they will warn you that an attack may be occurring, and a good IDS will identify common postcompromise traffic, such as outbound attacks from your machine.

Web Page Defacement
A popular time waster of a newbie cracker (or one with an actual message) is to replace content on your web site to announce his successful crack. Such defacement usually occurs on the home page itself, where it would be most noticeable. If crackers want to maintain access, however, they will seldom announce their presence in this or other obvious ways.

Warez/Dramatic Decrease in Disk Space
Crackers will often use your machine to store warez (illegal or cracked versions of commercial software), cracking tools, porn, and other files they wish to have available or share with others. This "free" disk space tends to be eaten up quickly. Output from df will tell you your current disk usage.

High Network Usage
If your network activity seems high, even when you're not doing anything, someone may be using your machine to serve files (often warez), break into other machines over the network, or effect Denial of Service attacks. Check netstat -na or lsof -i output to see what connections exist.

Contact from Other Administrators
If your machine is being used to launch attacks against other machines, administrators who are being attacked may contact you and let you know. Mind you, they may suspect you are the actual cracker, so don't expect to be greeted happily.

Promiscuous Network Interfaces
If crackers want to sniff any of the networks available on your computer, they will put the interface into promiscuous (capture all packets) mode. Look for PROMISC in ifconfig -a output.

Wiped/Truncated Log Files
Experienced crackers will remove individual lines from log files that show their inappropriate access to your system. A newbie cracker may instead simply delete the logs entirely. Any log files that are missing chunks of time or are suspiciously erased may have been tampered with. A good way to assure that you can check these missing logs is to have logs stored on additional servers (via syslog, and so on), which you can compare against the suspicious log files.

Munged utmp/wtmp
A cracker may wipe out her login entries from the utmp and wtmp files (programs such as zap, wipe, and vanish2 do this quickly) or erase the files entirely to hide the fact she has logged on. If you notice truncated last results, it is likely that a cracker simply erased the files. Programs such as chkwtmp and chklastlog check these files for signs of tampering.

New Users on Your System
New users in the password file are sure signs that someone has compromised your system — most likely a newbie cracker, or one who doesn't think you'll notice. Newbies often use usernames that are similar to existing usernames to make them less noticeable, such as lpr instead of lp, or names that play on cracking lingo, such as t00r or 0wn3d.

Strange Processes Running
If you see processes running that you didn't start and that aren't part of the system, they may belong to a cracker. Many programs run out of cron, so verify that any suspicious process isn't merely a piece of the system itself. For example, slocate often causes concern because it uses a decent amount of CPU and disk access, but it's a legitimate (though optional) system resource.

Unexplained CPU Usage
Sophisticated crackers may hide their processes from view, or they may merely name them after legitimate system programs such as cron, inetd, or slocate, to avoid being easily noticed. If the machine has high CPU usage, or just seems slow, it could be that your machine is being used by a cracker. Often crackers will run password-cracking programs (generally CPU intensive) on cracked computers rather than their own, relieving their machines of the load.

Local Users Have Had Remote Accounts Cracked
A cracker will often crack from one machine to the next by following users as they access other machines. By cracking the first machine, the cracker could watch those outbound connections and compromise the account on the new machine. This means that a crack of a user on an external machine could indicate your machine may be a target soon, or that your machine was already successfully cracked. In general, when one account is compromised, it's a good idea to check the security of all other accounts and change passwords during the process.

Things Just "Seem Funny"
Most of the cracks that are discovered started when the administrator simply thought something was amiss and started searching. Sometimes this leads to noncracking-related problems, such as a failing disk, bad memory, or unannounced networking changes, but often it leads instead to the realization that the machine has been cracked. Put simply, if the machine is behaving other than normally, the cause should be identified. Hopefully, it's a hardware or software problem, rather than a crack, but there's only one way to be sure, and that's to check it out.

This excerpt: Copyright 2003 by The McGraw-Hill Companies. All rights reserved. No part of this article may be reproduced or distributed without the prior written permission of McGraw-Hill.

What to do after a break-in

Once you've discovered that your system has been broken into, various remedies are at your disposal. Theories about the best way to approach recovering a machine after a break-in differ widely, even in professional circles. The approach we present is the one we prefer, but it will not fit all environments or needs.

Stopping the Damage
The surest way to protect your machine from further harm by the cracker is drastic but effective:

1. Turn off all network interfaces (Ethernet, PPP, ISDN, and so on). This removes the ability of the cracker to do anything else interactive to your system, while running processes will continue. If you have physical access to it, you can just unplug the network cable.

2. Take the system into single-user mode. Turn off all official root processes and all user processes. Any processes left over may belong to the cracker.

3. Reboot the machine from a pristine Linux boot floppy. By booting a clean boot floppy (or CD-ROM), you are sure to be running a minimal and tamper-free version of the Linux kernel, and you can now roam through your system (mounted read-only, preferably) to see what changes have been made and determine how the cracker got in.

   Note: The attacker may have left backdoors or processes that will do more damage when a machine is shut down. A simple method would be for the cracker to add new rc scripts in /etc/rc6.d or /etc/rc0.d, for example. You may want to investigate the possibilities of this type of problem on your system before rebooting to make sure that the attacker doesn't have anything tricky in store for you.

4. Begin serious damage control.

Between each step, you have a chance to look at the system and see what changes have been made by the cracker, while splicing away at the cracker's available countermeasures.

Assessing the breach

Once you've booted your tamper-free Linux kernel, you can traverse the disks of your system knowing that nothing can be hidden by the cracker's use of kernel changes, modules, and so on. To prevent yourself from losing the ability to track the cracker's actions, you should mount all of your partitions in read-only mode. Make careful notes of everything you find so you can clean out problems later.

Find suspicious files/directories
Look for directories that contain password files, cracking tools, or anything that you didn't put on the system. These may not have been visible until you booted the floppy kernel.

Locate New sXid Programs
Any new suid or sgid programs (especially those owned by root) are extremely suspicious.

Check Timestamps
Although this is not a reliable test, check for any files modified after the suspected break-in to get an idea of what was the cracker was trying to do.

Read Log Files
Check all your log files for signs of the cracker's entry point. You may use your log analyzing tools, but you should probably do a manual once-over of them all (especially during the time you believe the cracker gained access and thereafter) in case your tools miss important log entries. If you have a second syslog server, compare the logs on the cracked machine with those on the syslog server.

Verify Checksums
Verify the checksums of all your installed programs. It's a good idea to compare these against checksum databases from before and after the suspected break-in.

Verify Package Installations
Verify the checksums of installed packages using the built-in verify options as well. Verify that you're running the correct versions. A cracker may have downgraded your software on your behalf, leaving you with insecure versions.

Verify Config Files by Hand
A quick glance at various configuration files may highlight inappropriate changes, such as a web server that is now configured to run as root, or additional services in /etc/inetd.conf.

Back Up Your Files
Back up your files to tape or CD-ROM, if you can. If not, bring up just enough networking to be able to copy them to another computer, making sure you don't have any network-accessible services running.

Use Special Tools
More tools are becoming available to help you examine your system. One recent intriguing suite is the Coroners Toolkit (http://www.fish.com/tct/) by Dan Farmer and Wietse Venema. It can generate tons of (difficult-to-weed-through) output about your system via the "grave-robber" script or help you look for deleted files by scanning the drive's "unused" sections for inodes.

Inform the Authorities
It is a good idea to let the authorities know that a breach occurred. By being able to calculate the occurrences of successful cracks, they can warn the community of problems on the rise.

This excerpt: Copyright 2003 by The McGraw-Hill Companies. All rights reserved. No part of this article may be reproduced or distributed without the prior written permission of McGraw-Hill.

Getting back online

After determining what was done to gain access to your system, you have two main options: plug the holes and bring the machine back up, or completely reinstall the system. The most secure option is always to reinstall the system from scratch.

It is always faster simply to plug the perceived hole and go on. However, you can never know exactly what a cracker has done to your system. The cracker may have left time bombs that won't go off for months. He may have changed system binaries, leaving the machine usable but less stable. Thus our suggestion for the "best" method to purge a cracker from a system is the following:

1. Make backups of your important files.
2. Wipe your machine's drive entirely clean. (This is also a good time to make any changes you need to make — for example, adding more disks or changing the sizes of partitions. Use the downtime to your advantage.)
3. Install your Linux distribution from scratch, including only what you absolutely need.
4. Install all updates for your installed packages.
5. Make checksums of your machine and store them in a safe place.
6. Make any necessary configuration file changes manually. Don't simply copy the files from backups; they may have been modified.
7. Copy needed files from backups.
8. Recheck the files you installed from backups for any signs of cracking.
9. Run another checksum of the filesystems.
10. Turn on the network for the first time.

This is definitely not the quickest way to get your machine back up and running after a break-in, but it is the best way to be sure of the security of the system.

Mitigating concerns

For many reasons, these suggestions may not be feasible as written and must be modified appropriately. In this section, we'll show you some appropriate modifications.

Unacceptable Downtime
Following the procedures listed in the preceding section will require that your machine be down for at least a day during the investigation, backups, installation, and restore processes. However, doing this can be completely unacceptable in today's world of high- availability requirements. Instead, you may wish to install a second machine to take over functionality of the compromised machine, switch to the new system, and then cleanse the compromised machine.

Finding the Perpetrator
The procedures do not take into account actually finding the attacker. Although some evidence on your machine could implicate the guilty party, it is much better (from a legal standpoint) to "catch the perpetrator in the act," — and that means keeping your machine on and accessible to the cracker while you get the authorities involved to help track the cracker. Most crackers will run if they believe that they have been or are being discovered — meaning you may not be able to gather enough information to track them down.

Unresolvable Insecurity
If you fail to determine the cause of the break-in, performing a reinstall may not do any good. It could be that the insecurity is not yet known by the security community, and you would only end up installing the same buggy software.

Disclosure Rules
Your company may have its own rules about what can and cannot be disclosed. For example, a large bank would likely prefer not to reveal that it had been compromised and would not want to release this information to security organizations. However, disclosure can also work in your favor by getting a full-blown team actively involved. On several occasions, companies that were attacked were able to avoid disclosing any information that could have left them looking foolish and unprepared, by stating that they were not allowed to do so, due to a pending FBI (or other) investigation.

Retaliation Attacks/Counterstrikes
Some people believe the best response to an attack is to find the source of attacks and retaliate. Sometimes the reverse attacks occur with administrator go-ahead, while often they are automatically triggered by security software.

The retaliation attacks are sometimes merely probes: simple port scans, finger attempts, or traceroutes. Often, however, they are automated suites of full-blown attack scripts used to gain access. While we don't have any particular fondness for the former category, because it seldom provides any useful information, we actively discourage the latter retaliations for various reasons:

• Misdirected attacks: The apparent source of the attacks may not be the actual host from which the attack is originating. If the cracker is using source address spoofing, she can be pretending to be any host, and you can never know for sure her true source address. Even if the cracker is not impersonating an unrelated machine, the machine from which the attacker came is likely not her own system. The source of the attack is likely a machine that has already been compromised by the attacker, not her actual host. Thus, your retaliation attacks are more likely to be directed at innocent third parties rather than the attacker.

• Legal ramifications: Cracking of various forms is illegal in many areas. Your retaliation strike, though perhaps well meant, is governed by the same laws as the original attack and could cause much more trouble than it is worth, especially if you end up attacking an innocent party.

• No legitimate gains: Say you've gotten root on the offending machine — What do you do now? Trash the place?

• More animosity: Attacking someone who has shown that she has the technical skills to break in is simply not a good idea. She is now likely to take your retribution personally, and she may escalate her activities against you. Where once your machine was just another box, now you and all your machines are direct targets.

• Bad karma: Rather than attempting unauthorized access (the reason you felt violated enough to counterstrike), the more legitimate method is to inform the administrators of the source and the network provider of the activity with as much logging as you have available. Then you can work together to purge the intruder.

Blackholes

One other common countermeasure to an actual or perceived attack is to remove the ability for the offending machine to communicate with you. This can be accomplished in a variety of ways:

• Use TCP Wrappers to deny connections from the IP address of the cracker.
• Employ ipchains/iptables rules to reject/deny packets from the IP address.
• Create reject routes such that your machine cannot communicate with the IP address. You still receive packets from the source, but you cannot respond, which destroys the communication.
• Create similar access lists on both network firewalls and hardware.

These can all be legitimate actions, but we suggest you be wary of the following pitfalls if you wish to have such responses automated:

• Lost connectivity to legitimate hosts: If the cracker is impersonating a legitimate host, you will no longer be able to communicate with it for the services you require. For example, a cracker who impersonates the root domain name servers and sets off automated blocks will render you unable to resolve forward and reverse domain names, and you will not only be unable to connect to Internet hosts by name, but you may also deny access to local services based on hostnames with TCP Wrappers.
• Too many rulesets: There is a limit to how many rules and routes you can have before your networking starts to slow down. A cracker impersonating many different addresses could fill up your tables and cause you to perform a Denial of Service attack on yourself.
• Unwieldy TCP Wrapper files: If adding lines to the /etc/hosts.deny file, be sure you're not adding hosts more than once, or you are likely to fill up the file quite quickly. Be sure that any programs appending to the file read it upon startup as well, or you will only exclude duplicates since the program started.

Summary

Even if you take all the precautions mentioned in this book, chances are that eventually you will become the victim of an attack. You may miss a critical software upgrade. Or one of your software packages may have a vulnerability known only to the blackhats. Or maybe someone is determined enough to attack your machine that he resorts to gaining access at the console — not terribly difficult if your machine is at a colocation facility. Eventually, something slips and your machine will be cracked.

Your best protective and proactive measures will make it easier to determine the method the cracker used to get in and make it easier for you to recover your data and get your machine running securely again.

This excerpt: Copyright 2003 by The McGraw-Hill Companies. All rights reserved. No part of this article may be reproduced or distributed without the prior written permission of McGraw-Hill.