Dispersing a DDoS: Initial thoughts on DDoS protection

Distributed Denial of Service is a big deal—huge pools of Internet of Things (IoT) devices, such as security cameras, are compromised by botnets and being used for large scale DDoS attacks. What are the tools in hand to fend these attacks off? The first misconception is that you can actually fend off a DDoS attack. There is no magical tool you can deploy that will allow you to go to sleep every night thinking, “tonight my network will not be impacted by a DDoS attack.” There are tools and services that deploy various mechanisms that will do the engineering and work for you, but there is no solution for DDoS attacks.

One such reaction tool is spreading the attack. In the network below, the network under attack has six entry points.

Assume the attacker has IoT devices scattered throughout AS65002 which they are using to launch an attack. Due to policies within AS65002, the DDoS attack streams are being forwarded into AS65001, and thence to A and B. It would be easy to shut these two links down, forcing the traffic to disperse across five entries rather than two (B, C, D, E, and F). By splitting the traffic among five entry points, it may be possible to simply eat the traffic—each flow is now less than one half the size of the original DDoS attack, perhaps within the range of the servers at these entry points to discard the DDoS traffic.

However—this kind of response plays into the attacker’s hand, as well. Now any customer directly attached to AS65001, such as G, will need to pass through AS65002, from whence the attacker has launched the DDoS, and enter into the same five entry points. How happy do you think the customer at G would be in this situation? Probably not very…

Is there another option? Instead of shutting down these two links, it would make more sense to try to reduce the volume of traffic coming through the links and leave them up. To put it more shortly—if the DDoS attack is reducing the total amount of available bandwidth you have at the edge of your network, it does not make a lot of sense to reduce the available amount of bandwidth at your edge in response. What you want to do, instead, is reapportion the traffic coming in to each edge so you have a better chance of allowing the existing servers to simply discard the DDoS attack.

One possible solution is to prepend the AS path of the anycast address being advertised from one of the service instances. Here, you could add one prepend to the route advertisement from C, and check to see if the attack traffic is spread more evenly across the three sites. As we’ve seen in other posts, however, this isn’t always an effective solution (see these three posts). Of course, if this is an anycast service, we can’t really break up the address space into smaller bits. So what else can be done?

There is a way to do this with BGP—using communities to restrict the scope of the routes being advertised by A and B. For instance, you could begin by advertising the routes to the destinations under attack towards AS65001 with the NO_PEER community. Given that AS65002 is a transit AS (assume it is for the this exercise), AS65001 would accept the routes from A and B, but would not advertise them towards AS65002. This means G would still be able to reach the destinations behind A and B through AS65001, but the attack traffic would still be dispersed across five entry points, rather than two. There are other mechanisms you could use here; specifically, some providers allow you to set a community that tells them not to advertise a route towards a specific AS, whether than AS is a peer or a customer. You should consult with your provider about this, as every provider uses a different set of communities, formatted in slightly different ways—your provider will probably point you to a web page explaining their formatting.

If NO_PEER does not work, it is possible to use NO_ADVERTISE, which blocks the advertisement of the destinations under attack to any of AS65001’s connections of whatever kind. G may well still be able to use the connections to A and B from AS65001 if it is using a default route to reach the Internet at large.

It is, of course, to automate this reaction through a set of scripts—but as always, it is important to keep a short leash on such scripts. Humans need to be alerted to either make the decision to use these communities, or to continue using these communities; it is too easy for a false positive to lead to a real problem.

Of course, this sort of response is also not possible for networks with just one or two connection points to the Internet.

But in all cases, remember that shutting down links the face of DDoS is rarely ever a real solution. You do not want to be reducing your available bandwidth when you are under attack specifically designed to exhaust available bandwidth (or other resources). Rather, if you can, find a way to disperse the attack.

P.S. Yes, I have covered this material before—but I decided to rebuild this post with more in depth information, and to use to kick off a small series on DDoS protection.

Large Scale Network Design LiveLesson

Alvaro and I finished recording a new LiveLesson back in December; it should be available for pre-purchase at the end of January. For those folks interested in network design, this is going to be a great video series. We originally started out with the idea of updating Optimal Routing Design, but the project quickly morphed into its “own thing,” which means this video series is actually more of a compliment to ORD, rather than a replacement. Some pieces will be more up-to-date than the book, but there are a number of things covered in the book that are not covered in the video.

Large Scale Network Design LiveLessons takes you through the concepts behind stable, scalable, elegant network design, including modularity, resilience, layering, and security principles. This livelesson will focus on traditional distributed link state, distance vector, and path vector routing protocols, as well as the basic principles of centralized control planes (such as OpenFlow). A special point will be made of sorting out the relationship between policy and reachability, and where they can best be managed in a large scale network.

On the ‘net: Rockstar SE with Terry Kim

I was recently interviewed on Rockstar SEby Terry Kim—it’s a worthwhile listen!

Russ White has over 30 years of experience in network engineering; A Distinguished Architect during his time at Cisco, holds the highest certification Cisco has to offer as a CCAr and holds CCDE #1. He’s also a Published author, worldwide speaker at many tech conferences, and has over 40+ patents. This guy is a legend in the field of networking folks. In this episode we discuss SDN, Fog Computing, Disaggregation, and what network engineers should focus on for the future and more! You don’t want to miss this one!

The Great Domain Debacle

For those who are interested—this weekend I got into a “discussion” with my old/current DNS provider, Network Solutions. I’ve been using them for years, but there have apparently been recent changes at the company. Part of their “new” terms of service say—

We may, at any time, activate the auto-renew service for eligible services in your account. Further, we may provide you with an opportunity to “opt in” to our automatic renewal process in accordance with the instructions (and subject to your agreement to the terms and conditions pertaining to that process) on our Website. You agree that if you are enrolled in or otherwise utilizing our auto-renew service, we will attempt to renew your service at some point less than ninety (90) days prior to its expiration. Such automatic renewal for your service(s), if successful, may be for a shorter term than the term for which you originally purchased your service(s), but in no event shall such term be longer than the term then-currently in place for the service(s). Such automatic renewal for your service(s), if successful, shall be at the then-current price for the service(s). You further agree that, to turn off the auto-renew service for any of your services with Network Solutions, you must call 866.908.3450. You acknowledge and agree that the renewal price may be higher or lower than the price you paid for the then-current term of the service, and that we are authorized to charge your credit card or other payment method (such as PayPal®) on file for the renewal of the service(s).

Apparently they turned on auto-renewal for private registration for some domains I had intended to allow to expire. I opened a case about the charge, and was informed that I must call to stop auto-renewal from occurring, no matter what the check marks on the web page say, as is outlined in their terms of service (see above). You need to read that part of the user agreement several times to really get the impact of it—you essentially cannot stop using their services once you have signed up for them.

As their support person said in email—

There is no “allow to expire” with web.com you will still get charged since domains have a past shelf life of one to two years after dissconnected or “let to expire” as most people feel, and also you are bound into a contract that says that when you sign up. Please call.

The result was me shifting off of Network Solutions for all DNS services, and onto another provider. In the process, I realized the new provider does not support the .guru TLD, so I switched the blog over to rule11.us. I intend to add rule11.tech to the domain names that point here in a bit, whenever I get around to messing with the records and such. In the meantime, I also pointed the original .guru domain back here, but it’s not working. All the records look correct from the control panels, but they are not set up in the actual DNS records correctly. I’m not going to fight with it at this point; I think most folks have gotten the news that rule11.us is the primary domain at this point. Or at least I hope so.

There may be another “bump” when the domains are finally released by Network Solutions and to the new provider.

Probably more than most of my readers wanted to know.

SDxE: Engineer Focused

As an engineer, you’ve probably asked yourself a thousand times—what does all this software defined stuff mean for me? Answers are out there, of course; it seems like everyone is writing about it. Some of the answers out there are even useful, of course, but some of them are not. Most folks writing about the software defined craze are either unrealistic, or they’re focused on the large scale network you probably aren’t working on. Which leaves the question lingering: how does software defined apply to me?

SDxE—Software Defined Enterprise—is a new show designed to answer those questions for the engineer. I’ll be there; the full schedule isn’t in place, but I am currently pulling together a panel about the end of the (appliance) router. I plan to have folks from Cumulus, 6Wind, and at least one independent expert (Jeff Tantsura), sitting down to chat with me about disaggregation and the future of the router market. Specifically, are the tools in place that will allow you, the average engineer, running the “average” “enterprise” network, to take advantage of disaggregation?

Shawn Zandi will be there discussing the LinkedIn data center, and Pete Lumbis will be there talking about network automation. It promises to be a great new venue for engineers to learn, connect, and grow.