Hal Roberts, Ethan Zuckerman [twitter:ethanz] , and Jillian York [twitter:jilliancyork] are doing a Berkman lunchtime talk on Distributed Denial of Service [DDoS] Attacks against Human Rights Sites, reporting on a paper they’ve posted.

NOTE: Live-blogging. Getting things wrong. Missing points. Omitting key information. Introducing artificial choppiness. Over-emphasizing small matters. Paraphrasing badly. Not running a spellpchecker. Mangling other people’s ideas and words. You are warned, people.

A DDoS is an attack that consumes the resources of the target machine so that that machine is not able to respond, Hal says. It is an old problem: there was a CERT Advisory about an IP spoofing attack in 1996. A distributed DoS attack uses lots of machines to attack the host, typically via botnets (armies of infected machines). Hal gives an example in which infected machines check Twitter once a minute looking for encoded commands to do nefarious tasks. Gambling sites have often been targets, in part because they are reluctant to report attacks; they’ve also been known to attack each other. In one case, this resulted in the Net going down for 9 hours for most of China. Hal points out that botnets are not the only way DDoS attacks are carried out. In addition, there have been political uses. Botnets have been used to spy as well as bring down sites.

One monitor (Arbor Networks) notes 5-1500 DDoS attacks per day, globally. Hal thinks this number is too low, in part because there are many small attacks.

An application attack “crashes the box.” E.g., a slowloris attack slows down the host’s response time, reducing the number of available TCP connections. App attacks can be clever. E.g., simply reloading a homepage draws upon cached data, but doing searches on random words can be much more effective.

A network attack “clogs the pipe.” It floods the target with as much traffic as it can. This often will take down all the sites hosted by the ISP, not just the target site. The powerful network attacks are almost all “amplification” attacks. E.g., you request a big chunk of data: a little data in requests a massive amount of data back.

To defend against DDoS, you can optimize your server and harden it; you can build in over capacity; you can create a system that adds more resources as required; you can do packet filtering or rate limitation; you can scrub the attacking packets by “outsourcing” them to highly experience sys admins who look for signs in the packets that distinguish good from bad; if flooded, you can do source mitigation, asking routers routing the flood to you to block the packets; or, you can tell your ISP to dynamically reroute the packets. But, none of these technique work well enough or are too expensive.

The study by Hal, Ethan, Jillian, et al., asked a few key questions about how this affects human rights sites: How prevalent are DDoS attacks? What types are used? What’s the impact? How can sites defend against them? To answer these, they aggregated all the media reports, they surved human rights and media organizations. They interviewed respondents. And they hosted a meeting at Harvard. They learned:

• Attacks are common

• Sites on the edge of the Net, such as indie media, are particularly vulnerable

• It’s not just DDoS attacks

• There are some good answers for application attacks, but fewer for network attacks

• Network attacks may provoke a move to the core

• It helps to connect local geeks with core sysadmins

In their media research, they found lots of attacks, but not a strong correlation between the attacks and the politics of the attacked sites. The data are hampered, however, by the difficulty of gathering the info. Not all sites know they’ve been DDoS’ed. And the study had to use large boolean queries to try to find coverage in the media.

Even though there are many attacks, the core (Tier 1 providers, plus their direct customers) does well against DDoS attacks. Those Tier 1 sysadmins work closely together. But, as you get out further from the center — a customer of a customer of a customer of a Tier 1 operator — people have little recourse. “Being at the edge in terms of DDoS is a really bad thing,” says Ethan. The core has dedicated staff and a ton of bandwidth. They typically respond to a DDoS within an hour, and probably within 15 mins. So, if you’re Google, it’s not that much of a problem for you.

But, if you’re a small human rights site, it’s much harder to defend yourself. E.g., Viet Tan has been attacked repeatedly, probably by the Vietnamese government. Worse, they’re not just being DDoS’ed. 72% of those who said they’ve been DDoS’ed are filtered by their governments. 62% have experience ddos attacks. 39% have had an intrusion. 32% have been defaced. Viet Tan was being attacked not just by a botnet, but by the Vietnamese around the world by people who had downloaded a keyboard driver that logged keystrokes and could issue attacks. The people attacking them were the people they were trying to reach. “It’s an incredibly sophisticated way of doing things,” says Ethan.

Arbor Networks says 45% are flood-based, and 26% are app based. Hal et al. sent Arbor the list of attacks his research had uncovered, but Arbor had only known of a small percentage of them, which is some small evidence that Arbor is under-reported.

Of the sites that eperience a DDoS attack last year, 56% had their sites shut down by their ISP, while 36% report that their ISPs successfully defended them. E.g., there was an attack on the Burmese dissident site, irrawaddy.org. This knocked not just that site out, but all of Thailand. Thailand has its own national ISP, which is Tier 2 or 3; a 1gb/sec attack will take down an ISP of that size. Irrawaddy moved ISPs, got hit with a 4gb attack and could not afford to pay for the additional bandwidth.

Hal points to the consolidation of content through fewer and fewer ASNs. In 2007, thousands of ASN’s cotribted 50% of content. In 2009, 150 ASNs contributed 50% of all Net traffic. This may be in part due to the rise of high def video (coming through a few providers), but there’s also fewer on the long tail providing content (e.g., using gmail instead of your own mail server, blogging on a cloud service, etc.). Small sites, not in the core, are at risk.

Should you build dedicated hosting services for human rights sites? That puts all your most at-risk sites in one pool. How do you figure the risk and thus the price? One free host for human rights sites does it for free because they’re a research group and want to watch the DDoS attacks.

The paper Hal et al wrote suggests that human rights sites move into the cloud. E.g., Google’s Blogger offers world class DDoS protection. But, this would mean exchanging the control of the DDoS attackers for the control of proprietary companies that might decide to shut them down. E.g., WikiLeaks moved onto Amazon’s cloud services, and then Amazon caved to Joe Lieberman and shut WikiLeaks down. The right lesson is that whenever you let someone else host your content, you are subject to intermediary censorship. It is an Internet architecture problem. We can respond to it architecturally — e.g., serve off of peer-to-peer networks — or form a consumer movement to demand non-censorship by hosts.

(The attacks by Anonymous were successful mainly against marketing sites. They don’t work against large sites.)

Recommendations:

• Plan ahead

• Minimize dynamic pages

• Have robust monitoring, mirroring, and failover

• Strongly consider hosting on blogger or something similar

• Do not use the cheapest hosting provider or dns registrar

Bigger picture recommendations: In the most successful communities, there is an identifiable, embedded, technical experts who can get on the phone to highly-connected core systems. Many of these core entities — Yahoo, Google, etc. — want to help but don’t know how. In the meantime, more will move to cloud hosting, which means there’s a need for a policy, public pressure approach to ensure private companies do the right thing.

Q: Shaming as a technique?
A: We need to do this. But it doesn’t work if you’re, say, a large social media service with 500M users. Human rights orgs are a tin percentage of their users. They tend to make the easy decisions for them, and they’re not very transparent. (Tunisia may turn out to be turning point for Facebook, in part because FB was under attack there, and because it was heavily used by Tunisians.)

Q: Public hosting by the government for human rights groups?
A: Three worries. 1. It’s hard to imagine the intermediary censorship being less aggressive than from commercial companies. 2. It’d be a honeypot for attacks. 3. I’m not sure the US govt has the best geeks. Also, there’s a scaling problem. Akamai carries 2TB/sec of legit traffic. It can absorb an attack But the US would have to create a service that can handle 200gb/sec, which would be very expensive.

Q: What sort of tech expertise do you need to mount an attack?
A: The malware market is highly diversified and commodified. Almost all the botnets are mercenary. Some are hosted by countries that in exchange ask the botnets be turned on enemies now and then.

Q: Denial of payment?
A: We have a case in the study called “denial of service by bureaucracy.” E.g., a domain name was hijacked, and it took 6 wks to resolve. A denial of service attack doesn’t have to attack the server software.

Q: Can botnets be reverse engineered?
A: Yes. Arbor Net listens to the traffic to and from infected computers.
A: You either have to shift the responsibility to the PCs, or put it on the ISP. Some say it’s crazy that ISPs do nothing about subscribers whose computers are running continuously, etc.

[Fabulous presentation: Amazing compression of difficult material into a 1.5 hour totally understandable package. Go to the Berkman site to get the webcast when it's ready.]

Read the original blog entry...