DDoS attacks – behind the scenes

— Adam Haertle —

I wanted to start by telling you something about DDoS attacks. I assume that most of you know those attacks well and I don’t have to explain technical aspects, but probably some of you may not be familiar with this video, which perfectly illustrates the essence of DDoS attacks. It shows how complicated defence against DDoS attacks can be – depending on how large the scale of the attack is. You can probably see every fifth frame of this video, but I hope it gives you a pretty good idea of what DDoS defenders actually have to deal with. If you ever have to explain to your board how a DDoS attack works, I think this video is the best ten-second-long explanation there is.

Why are we even talking about DDoS attacks? They have been around for years and are here to stay. Many different companies try to do the statistics, sometimes they are more successful, sometimes less. Here is data from the previous two years and as you can see from the growing bars, the results are pretty good. So, if you’re just selling anti-DDoS solutions, charts like this are always good news because there is only one trend in these attacks – they are and always will be on the rise. Line capacity and availability are increasing. Just a couple of years ago, four-bit capacity attacks were within the capabilities of large countries, whereas today any teenager with a botnet is able to achieve it. We have 100, 200, 500 Mbps links at our homes. Taking control over them and connecting a few thousand computers is enough to flood one service with traffic. Internet providers are in a rush to provide more and more speed and it has a side effect – more and more speed is also provided to the originators of DDoS attacks.

There used to be a time, last 5-6 years maybe, when anti-DDoS solution providers would regularly publish reports indicating the maximum capacities they had seen and there were actually those years where it was kind of a duel – “we spotted a 50 GB attack”, “and we spotted a 80 GB one”, “and we spotted a 100 GB one”, “and we – 200”… and in fact when I was trying to prepare the slides for those presentations and find the latest records – because as you know records are fun, they are widely used in media and they’ll look nice on the slides – the last record I found that was pretty impressive was from February 2018 and it was about an attack on GitHub that reported that capacity had exceeded over 1 terabit per second. It’s quite interesting to me that for the past year no one has announced a more successful story.

Does that mean there were no larger DDoS attacks? I doubt that. Does that mean there won’t be any larger DDoS attacks? I doubt that either. Perhaps it’s just that a company that has nothing to boast about or their marketing departments have decided that there’s no point in breaking these records because we’re talking about capacities that are beyond the imagination of most network engineers designing their solutions. Because who prepares for such a traffic on purpose? From a business perspective, handling terabits per second doesn’t seem reasonable unless you’re some kind of YouTube or Vimeo service provider. Then there was another attack – just a month later – reaching around 1.7 terabits, but in this case, we have no information about who was attacked. All we can say is that the number and scale of these attacks is growing. I’ll give you the figures I was able to find – in the first half of 2017, one of the anti-DDoS solution providers reported 7 attacks larger than 300 Gbps, while in the first half of 2018 there were already 47 such attacks. I wouldn’t be surprised to see more than 100 in the first half of 2019. That scale is constantly growing.

What are the most common targets of these attacks? Well, it depends on the scale. The “bulkiest” attacks affect the largest companies, of course, and the most typical targets of such attacks in recent years have been mainly online gaming providers. We used to play offline in our rooms. We would invite our friends, they would bring their computers, joysticks, and even monitors and we would connect to one network and nobody would DDoS us (unless someone tripped over tangled cables). Nowadays, we have online games, and, as a result, gaming providers are the targets of these kinds of attacks. Of course, where there are attacks, there are victims, and there is someone who wants to make money out of it. Who is most affected by a DDoS attack? Probably someone who provides a service on the Internet, and that kind of service provider is mainly hosting companies, which also suffer from these kinds of attacks. There was a period in Poland, it might last three months, during which hackers – we don’t know whether domestic or foreign – tried to force hosting companies into paying quite lavish ransoms. One example of this is the attack on Nazwa.pl that took place in June 2018. First, there was a small-scale attack, half an hour later there was a second, larger one that actually hit the clients hard. A few days later they posted an announcement. Actually, the title of this message “Nazwa.pl stops a DDoS attack” is quite characteristic, because a DDoS attack can be stopped simply by accepting it, that is, the website takes all this traffic and accepts it, its services don’t work, but it can be said that they stopped it because it wasn’t heading anywhere else – it was stopped because it was accepted. So, they stopped it so that client services stopped, so the PR had a lot of work to do.

Of course, Nazwa.pl is not the only provider of services of this type in Poland. We also have Home.pl, which is a large company and which by the nature of its business was another victim of this attack. In September Home.pl was also shut down. We are actually talking about attacks of such a scale that the largest hosting providers were challenged, despite the fact that, in principle, by 2018 they should have been well prepared to handle this type of incident and should have had infrastructure and solutions in place to prevent the effects of this type of emergency. But it turned out if a hacker has enough capacity, they are able to shut down even professional companies. At one point, Home.pl went quite far in terms of openness of its communication – which is a positive sign that such initiatives are happening. Someone clearly wanted to share the details of the attack, there was a breakdown of its stages, time of each event, what happened and what the consequences were. Unfortunately, you won’t find this post on this company’s profile anymore because someone else decided that this data was a little bit too much and the post was deleted. Fortunately, the Internet never forgets – in this case we have a screenshot. There is really nothing shameful in the fact that we were a victim of an attack. I truly hope that one day Polish companies will be able to honestly admit what happened and how the incident was handled. On the side – sharing this kind of information is certainly not desired by PR or legal departments, while for IT departments it is priceless knowledge, as we are usually attacked together and defend ourselves separately. I think that’s unfortunate, but maybe this will be a topic of some separate webinar in the future.

What are the implications of an attack on a large hosting company? It turns out that clients are affected, sometimes badly. You may be familiar with SkyCash. It operates a number of different payment systems – for parking spaces, tickets as well as other financial services. It used services of one of these hosting companies and unfortunately had to announce that its services were unavailable to customers. What did that mean? Let’s say that someone took the train and wanted to buy a ticket – they couldn’t. Someone parked their car and wanted to pay for the parking space – they couldn’t. Situations like those are really uncomfortable for customers. A hosting company is expected to provide its services around the clock, all year round, because when this does not happen, customers, depending on the profile of their business, may have some serious problems, because now they have to cascade this message about the outage to their customers and in fact they are helpless and may as well call their account manager and cry, but this will not result in service availability being restored any sooner. These hackers usually come back and take their chances – for better or for worse. So just a few weeks later Home.pl announced that there was another attack. On the other hand, we are clearly learning from the attacks, which is always good. The companies that have already been under a DDoS attack, gain knowledge and experience and in the case of subsequent attacks, deal with them in a much better and efficient way. It was apparent that the subsequent attack had in fact been properly addressed and its impact was diminished.

In Poland we had some interesting incidents related to DDoS attacks from outside the hosting sector. In May 2016, we there was an intriguing attack on Polish banks. Several banks were forced to face traffic in excess of 50 Gbps. Most of them did pretty well, some not so much, but then they started to do better. In any case, the customers of the said banks were mostly not affected by the attack, although it was visible on the capacity monitoring systems of the banks’ providers – those 15 Gbps of traffic was something to be noticed and could cause some damage if the banks had not been properly prepared.  Interestingly, this was very multi-sector traffic, as it was both a volumetric attack and an attack on individual protocols. It was quite interesting, because it looked like a collection of all possible botnets that were for rent and were infiltrated into individual Polish financial sector institutions. The funny thing is that the hackers impersonated the Kadyrovites, which is a group of Chechen guerrillas. Of course, we assume that they had nothing to do with Chechnya apart from the name of the group they adopted, but attacks of this kind occur periodically. Earlier, there were attacks on secure e-mail providers like ProtonMail or their competitor Tutanota, and the same – or allegedly the same – group started attacking banks. That is, someone rented botnets and then sought out the targets of those attacks.

What is interesting, groups claiming to be the Kadyrovites are still attacking for example Polish shopping malls – they say they planted a bomb in one of them and demand a ransom in bitcoins. The business model is very similar, these attacks are more or less fictional, or in the case of bombs, completely fictional. In the case of DDoS, such attacks actually took place, but their scale was not sufficient enough to threaten the bank infrastructure.
The Polish touch was also quite intriguing when data from one of the servers of DDoS solution providers leaked in 2015. Because if you want to send too much traffic to someone, you don’t have to do it yourself, there are providers. Everything is “as a service” nowadays, so we also have “DDoS as a service”. There’s a whole ecosystem of these kinds of providers. Once someone has a botnet, they want to monetise it the best they can, and one conceivable way to monetise such control over thousands of computers around the world is to use them for a DDoS attack. It was a server of a group called Lizard Squad, or at least that was the name they used and in fact the database of participants of this game leaked and filtering by domains of e-mail addresses, you could also find Polish users. A glance at their passwords actually indicates that not only were the e-mails in Polish domains, but these users may have actually had something to do with Poland. They didn’t carry out that many attacks, but the list of leaked data provided also the targets and among them there weren’t many Polish websites: an online shoe shop or some smaller companies, while from the more interesting things there were a dozen or so servers at HosTeam. I had the opportunity to talk to the head of the company, and he told me that these servers supported online games – mainly Tibia or Minecraft – so obviously someone was onto the players and used this botnet to launch a DDoS attack on Polish servers which hosted game servers.

Another very interesting story with DDoS attacks and Poland is connected with ClubWorld casino. In 2013, this casino was the target of DDoS attacks and, in fact, the target of extortion. This extortion was conducted by two Poles. We know this because the casino itself was based in Manchester, and under British law, the identities of the perpetrators of crimes – even suspects – are revealed along with their images. So, we were able to learn the names and pictures of the hackers who were arrested. They were two businessmen, entrepreneurs from Szczecin, board members of one of Szczecin’s IT companies, who decided to make money by blackmailing the owner of a British online casino that is worth several million pounds. They offered this owner to transfer half of the casino shares to their company in exchange for discontinuing DDoS attacks. To prove their determination, they launched successful DDoS attacks disabling casino operations. Interestingly, they foolishly arranged a meeting with their victim at a hotel, and the victim was smart enough to inform British prosecuting authorities about it and as a result, the meeting was documented on video. If you search YouTube, you can find a video of that meeting – parts of it, obviously – where our businessmen, sitting on the right, are negotiating hard with the British businessman to acquire shares in the business he had built. Those men were unaware that the meeting was being recorded and that they would encounter British police on their way out. The result of this ignorance was a sentence of over five years in prison. It was passed at the end of 2013, so it looks like if the men have served their well-deserved sentences, they are now free. Remember that those blackmailers went to a meeting at a hotel. What were they thinking? On one website I read that they had probably carried out supposedly successful operations of this kind and hence were convinced they were unstoppable and as a result, they had this slip-up with blackmailing the casino, which fortunately ended badly.

There were quite a few of these incidents in Poland, actually. On April 8, 2015, there was a DDoS attack on the policja.pl servers and a Polish Cyber Army was allegedly behind the attack. The attack was actually successful and the website was shut down. Here we must admit that the police reacted well and fast – the next day the perpetrator of this attack was already stopped. He was hiding in the attic of his parents’ house. It was an effective police action that showed that indeed, certain actions can be conducted pretty well. However, after several weeks in custody, the perpetrator was released, because his subsequent Twitter posts looked like he was looking for help (not sure what kind), while a small scandal arose that the police were oppressing Anonymous hacktivists. When someone launches a DDoS attack on a police server, they can, or rather should, expect the police to launch a DDoS attack on their personal freedom and the issue will not be left that way.

The next story, from 2014, is quite interesting as OVH servers were attacking OVH servers. How do you attack a company that has pretty good anti-DDoS systems for external traffic? Take control of the servers on the internal network and launch a DDoS attack on the internal network. What is interesting is how this DDoS was mitigated. At the very bottom of this post from the company’s blog, you can read that simply one server room was cut off from the other and the traffic went out by the public network, thus succeeding in unclogging the internal network. This is some sort of solution – if your internal network is attacked, let the traffic out, let others worry about it. Did it work? It did, pretty well actually. As the example shows, internal network traffic can be cut off in this way.

A very interesting incident happened in 2016 – GitHub stopped working. Many of you are probably familiar with this service, some of you also use it on a daily basis. A non-functioning GitHub is a problem for many development teams around the world. Not only GitHub, but also Shopify fell victim to this attack, and in fact it wasn’t until the next day that it was revealed that the attack didn’t affect GitHub, Shopify, Spotify or any of the other websites that went down that day, but their DNS service providers. This is very interesting because if it turns out that the site you’re attacking has good anti-DDoS protection, then perhaps it’s the DNS provider that is the weakest link. If one does not use several providers simultaneously, it may turn out that disabling this DNS provider disables the service because no client is able to reach its server when the domain name is not resolved to an IP address. So, this attack had quite a bit of reach and impact. Interestingly, one site that was a client of this DNS provider and resisted this attack was PornHub, which actually used six different DNS providers. As you can see when you have a serious business going, you care more about business continuity.

In the case of DDoS, we have another interesting story, in which the problem was caused by the default passwords, as DynDNS was attacked i.a. with the use of the Mirai Botnet, which was based on cameras. If you’re wondering when a toaster-based botnet will emerge, I’m afraid it never will, no matter how many toasters will be connected to the Internet. Why are botnets based on cameras, routers, and set-top boxes, but not toasters, TVs, and refrigerators? It’s not because of lack of Internet-connected toasters, TVs and refrigerators, but because of the network architecture of these solutions – if you connect a toaster to the Internet, it’s unlikely to be accessible from the Internet. It has access to the Internet, but not the other way round, because it’s standing somewhere behind a NAT device, whereas the devices that are redirected to the public Internet from the public Internet are the ones that you want to connect to when you’re away from home – e.g., at work you want to view the image from the camera that’s at home. That is why cameras, routers and video security recorders are most often enlisted in botnets, because they’re the ones that are accessible to connections from the outside, and this was the example of the Mirai Botnet, which took advantage of vulnerabilities or default passwords of many such devices and was used i.a. to attack DynDNS.

This is a map of Internet failures. As you can see, half of the U.S. couldn’t access some Internet services, and the reason for this attack – disabling GitHub, Shopify, Spotify and in some parts of the world also Facebook – was quite simple: someone had a fight with someone on some online game and the perpetrators, unable to disable the game server, which was hidden behind a good anti-DDoS service, disabled that server’s DNS service and thus disabled the game server. So don’t annoy gamers, they might turn off your internet.

The most known incident connected with DDoS in Poland and in the world, and one of the few incidents in Poland that gained interest of world media, is an infamous case of Polish Airlines LOT, which fell victim of an IT attack – or at least they said so at the very beginning – which caused cancellation of 10 or 11 flights. What do we know about this attack and what does it have to do with DDoS? Firstly, just a few seconds after problems appeared, LOT issued an announcement that it had been the target of an ICT attack on its ground IT systems. For anyone involved in IT security, this news was pretty surprising. How could an attack on ground-based IT systems disable the ability of aircraft to take off? It turned out that flight crews couldn’t download flight plans and there was some kind of network congestion. How could someone clog the LOT network? That was interesting. Later on, LOT announced that it had fallen victim to an attack on its network and they were talking about a massive attack, so it is automatically associated with a DDoS attack. Then it turned out that LOT used world-class systems and security, so this problem could affect the entire airline industry. It was getting really serious, and no wonder CNN was talking about it on their main newscast. Also, we’re guessing that someone from the Internet launched the DDoS attack on the LOT network and this what this incident was about. So, the topic was further analysed, more information was provided and at one point we read that LOT could have been grounded due to a human error and not a hacker attack. We were the victim of an attack and yet it was human error – that’s weird. And we’re talking about mistakes made by an outside company. This is getting more and more interesting. Let’s read on. It turned out that the company asked the prosecution to look for the hackers. So, we had the human error made by an outside company, and on the other hand we had hackers and yet the IT manager resigned in the process. What really happened?

Many months later it was publicly revealed that paradoxically LOT’s systems were not the hackers’ target, but their tool. Of course, the IT security sector had suspected it for some time already. What did it mean? It meant that it wasn’t the Internet that launched a DDoS attack on LOT, it was the opposite. How was that even possible? It turned out that reconfiguring the company’s firewall, or replacing it without transferring whole configuration, caused the company’s DNS server to become publicly accessible. It was quickly identified by some Chinese botnet as a so-called open resolver, i.e. a server that enables one of DDoS attacks, and it turned out that this server was used for a DDoS attack on some foreign entities. Indeed, LOT’s links were clogged, but they were congested with outbound traffic, not inbound traffic. The problem was sort of similar, the path was the same, just the direction was different. The announcement that LOT was the victim of an attack still circulates in the media, but it was also a kind of co-conspirator and there is official information from the prosecutor’s office that LOT’s system was used for actions against other entities’ servers. What is the result of the fact that the sense of the communication was quite the opposite at first? The company owes a special tribute to its PR department, they were the ones who took care of that. To this day this alleged attack on LOT is referred to in international presentations on critical infrastructure security as one of the most important attacks in history.

Few of these attacks are documented, so each one comes in handy when it comes to presentations and on one of them we could read that LOT was hacked, and on another that there was some kind of attack on the internal network. The correction of this information was never released to the public, so there was no information that LOT wasn’t hacked, that it wasn’t DDoS – it was just a configuration problem, some firewall rules were not transferred to the new instance. Why? Probably they had it done on the cheap and it worked, but not really.

If you were to ask me about the DDoS attack I found the most threatening, it was the November 30, 2015 attack on the network of DNS servers handling 13 major DNS servers that actually keep our Internet running. Each of them has dozens of nodes. They operate in such an architecture that it used to be 13 servers, and now it’s 13 times several dozen servers to ensure continuity of traffic. And of course, these root servers are provided with a monitoring system available to the public. If you consult data from November 30, 2015, a cold shiver will go down your spine, because even though there were dozens of servers behind each of those thirteen machines, someone made an effort and generated so much traffic for at least half of them that all of those dozens of servers stopped responding to queries. Fortunately, this architecture is so distributed that other servers took over this traffic and DNS server access could not be disabled. However, the very fact that someone demonstrated that they were able to turn off half of these Root DNS servers – despite their completely distributed architecture – shows that this could have been some sort of test made by an organisation, a country, or whoever was behind it, who wanted to see if it was even possible to disconnect absolutely everyone, worldwide, from the Internet. So, it might be a good idea to keep a piece of paper with IP addresses of the most important servers in your drawer, in case this experiment is to have any follow-up.

Should you ever want to test your network – of course we don’t encourage you to test someone else’s – you can buy such services online without any problem. The services I show are no longer operative, but there are many others available. These are professionally created websites with descriptions of all the benefits of these products. Many marketers could learn from them. Anonymity, different protocols, diversity of services. There are even some very nice websites with descriptions of how these DDoS attacks can be carried out in either layer 4 or layer 7 – you can choose from a variety of options. These companies or entities – I don’t know whether there are any companies behind this – that offer these solutions pretend that these tools are used to test your own infrastructure, but, of course, they are used by hackers and then law enforcement authorities apprehend the perpetrators – but it takes some time. You can often buy such a free trial service. There are even discounts for Black Friday. It is like a normal business that pretends to be legal, while of course it is used for activities far beyond what is legal. Probably those capacities which they announce in their marketing brochures, like terabits per second, are very fictitious. However, if you had an idea of combining several types of services of this type in order to test your infrastructure more effectively, I would like to say that the business model of this kind of company does not encourage it, because they are not DDoS solution providers themselves – they are resellers and they probably rely on one and the same botnet, which sells these solutions to them, and they resell them with a margin. So, someone has a botnet and doesn’t get involved with marketing, sales and advertising, but resells such services wholesale, at a significant discount, and then various entities that take care of marketing and sales, package these services under different logos and offer them to their customers. So, if you buy services from three different entities, it may turn out that the traffic will be sent by the same botnet. As you can see, the packages vary, and starting from $15 a month you can buy a 5-minute-long DDoS with a capacity of up to 250Gbps, which is pretty decent, although I don’t expect that this capacity will actually be provided to the customer, because being a customer of this type of service you have limited possibilities of measuring effectiveness, unless you are actually attacking your own IP address. Some offer larger capacities, others smaller.

As I mentioned, these are trial services, you can test whether it works or not, but what amuses me the most is when I visit a website of a company offering DDoS solutions, and it turns out that they use the services of an anti-DDoS company to make their website work. Why? Imagine you are in the DdoS solution selling business – who is your competitor? Other companies offering DDoS solutions? I wonder what will they use them for? So, it’s a wild market. These companies prank each other all the time. The DDoS hacker site hidden behind the anti-DDoS solution looks quite entertaining.

Thanks for listening to my part of the presentation and now I’m passing the floor to Teodor, who will tell you about one of the anti-DDoS solutions.

— Teodor Buchner —

Thank you very much for the introduction. I had a great time listening to Adam and looking at his slides, I learnt a lot. For example, I learnt to arrive at trade negotiations with a colleague, one of you wearing a black shirt, while the other – a white one. I assume they were playing good cop, bad cop.

— Adam Haertle —

Yes, in this video one of them was actually speaking and the other not so much.

— Teodor Buchner —

The prospect of DDoS in 2019 is indeed an interesting one, as these attacks have evolved and it’s clear that commercially they still have a raison d’être. I wanted to ask you one thing – from your experience, based on documented DDoS attacks – do these botnets exist under their own IP addresses?

— Adam Haertle —

There are many different strategies and in fact we are dealing both with DDoS attacks from the IP addresses of the real perpetrators, but we are also dealing with spoofing the source address to hide the identity of the perpetrator. If they can, they are most likely to use stateless protocols that allow them to hide the source address.

— Teodor Buchner —

This is an interesting topic. If I have a botnet that has 35,000 hosts and I want to monetise it in as many different ways as possible, is it really the best way to announce all the IP addresses belonging to that botnet? Because if someone collects a DDoS attack like this and regroups it by source IP, then theoretically a couple of admins could have their Christmas ruined. My position is as it is and I’m speaking from the perspective of a telecom operator. I think the telecom operator community of the world should show some remorse, because there are good practices for cleaning up outbound traffic, and every operator who operates some kind of autonomous system has these tools, and they should be checking to see whether the traffic that they’re releasing comes from an address that they themselves advertise, or that passes through them, or that they generate themselves. We know what it looks like in practice and therefore this is the aspect which adds to the effectiveness of these crimes, because they can use spoofed addresses, they do not have to provide their own addresses as source. It is enough that they provide any address from the IPv4 space and the operator is satisfied and lets such traffic pass because, after all, they still earn money out of it and everything works – except for the victim’s service.

We have indeed managed to make some contribution to anti-DDoS protection. Of course, we are not the first on the market and there have already been some of these solutions present outside and in Poland. However, we are convinced that a telecommunications operator is responsible for their link and for their customers, and as part of this responsibility, if they have the potential to develop tools, they should do so. We have really made an effort to enter this competition. And now I’d like to say a few words about the matter Adam was talking about earlier. The solution we recently finished is a software protecting operator network against DDoS attacks. What is interesting, it is a software solution, i.e. we managed to build the solution entirely on software technologies. Anyone who has ever seen VHDL knows that FPGA programming is no fun. C++ is much better in this respect, and it’s much easier to find people who will build a code, test it, look for vulnerabilities, and people who will eventually maintain it. And we managed to form a team of professionals, which together with our Security Operations Centre started to create such a solution.

Here you have the general structure of the system and how it works. So, we have two elements here – we have a traffic sensor that runs on measurement protocols and aggregates data coming from routers. A DDoS attack resembles an attack of a troll with a big club and is therefore hard to miss. Although the traffic data provided by the edge routers is sampled, i.e., we see statistics on every thousandth packet, it turns out that for volumetric attacks this is completely enough. There were of course some pitfalls in the aggregation because information from routers flows in portions, so it was a challenge to put these portions together in a meaningful way into a single parameter that would describe the state of a given connection, but fortunately we managed to do that. So, we have a sensor that can detect that something is wrong, and if we receive a signal from the sensor, we can internally redirect this traffic to a scrubbing unit. We have the operator who can either make these decisions or they can be made automatically, depending on how the system is configured. The system has the advantage that it gets scaled well because it is basically built on standard IT hardware. As I said it does not require a FPGA, so here we have a very low technological barrier to entry. It’s a pretty simple language – if C++ can be called a simple language – but if C++ is not considered simple, I don’t know what language is. We used Python too, as far as simple languages go.

Here are the project details. We generated this project with a partner, a technical university, so we used each other’s expertise. This is publicly available data, so I am not bragging about anything here. The solution architecture is based on x86. Here we come to an interesting issue from the IT perspective, namely handling of 10 gigabit and higher connections. Those who have ever handled it, or tried to analyse such traffic, know that the time for analysing a single packet is extremely short, and therefore it must be done efficiently. If we’re talking about efficiency, then it probably should be multithreaded – so we actually made the solution multithreaded at the level of the scrubbing module and with all the blessings associated with multiple threads, i.e. thread synchronisation, accesses to shared resources, etc. There were many issues to be resolved. I think that if one day we decided to talk about all of this, it would turn into a scientific session. There were indeed many technical problems to be solved along the way.

In fact, this part after the technical one, the service, is quite routine. It’s basically a web-based system for managing configurations, customers or anything that can be managed from the operator console. On the other hand, the truly out-of-the-box solutions are deployed under all this and we’re really happy that we managed to do it. It’s a material on which you can build, because if we already have such a platform, we can actually try to use it in various contexts. I truly hope that in the upcoming months you will hear from us about the direction we’ve decided to go, but our team still has to face plenty of challenges. I think this is a good prognosis for the future.

We talk all the time about volumetric attacks, and that doesn’t exhaust the whole topic. There was a clever Ping of Heath attack, I think it was back in the days of Windows 95. It was a single packet attack, which was able to disable the entire system, and it opened people’s eyes to the reality that in order to make a system unavailable, you don’t need a large number of packets. Those of you who have ever dealt with building and maintaining IT systems and watched e.g. logs from some application server and database under it, know that there are situations that a client has sent a completely legal query, and the database is lacking one index or two or three, and so the query keeps processing, the application lags, the client gets frustrated, and finally the session breaks down, and when it breaks down, the client tries again, and so the server load grows, and we have actually such a DDoS auto-attack, i.e. one wherein the system creator has attacked themselves, that is, he or she has made a system which can get lagged with one well-tailored query. So, indeed, such a DDoS attack has very different faces. An attack can be generated on many different levels.

The simplest volumetric attack is exactly like a troll attack, while much more sophisticated attacks are also possible: like attack in layer 7 on a badly written application, which is easy to put down with a relatively small number of packets. This is the number of packets that will not be alarming to anyone. The operator is likely not to notice it at all. And there were more attacks like this, including on the TCP protocol itself. As for example Slowloris – there may be a packet and there may be an attack on layer 7 and before we manage to close the http session, we add another packet to it right at the end and force the server to have resources involved in handling this session all the time – we don’t let it die in peace. The same manoeuvre can be done at the TCP session level. You can also add packets one at a time and then the server gets clogged as well. Here you can actually see a significant difference between TCP and UDP. TCP servers, due to the fact that they are stateful, the connection is harder to handle, it’s a little bit easier to jam them, so there’’ a trend of switching to stateless connections, i.e. moving from TCP to UDP, and handling retransmissions or other things related to session continuity on layer 7. This is the direction in which the Internet market is going, and we are trying to follow it and slowly prepare for this transition.

So, when we’re talking about the complexity of the TAMA solution, we mean primarily volumetric attacks. However, we plan to introduce a project that extends the scope of such protection – we are really trying to seize the opportunity. So, we have scaling, performance, flexibility, and a lot of different features of a distributed system that give us added business value. And let’s face it – not many Polish solutions were implemented, so we joined an elite club and we are pretty happy about it. We believe it will be solution that will satisfy us and our customers and will allow us to develop this technological platform in different directions.

Did you follow the legend regarding non-volumetric attacks? Do they find a raison d’être somewhere in this underworld?

— Adam Haertle —

It seems to me that in the DDoS world we are actually dealing, as you correctly observed, with trolls. And it’s hard to expect much sophistication from a troll. However, it’s easiest for this troll to just swing this club, and these volumetric attacks are actually the most popular, because there it simply works on a “fire and forget” basis, and it’s so universal that it doesn’t need to be tailored to specific victims. Alternatively, what needs to be adjusted is the volumes, and you can see that if we talk about the most common targets of attackers, like game servers, we don’t expect that these are some duels of cryptography professors – although maybe I’m wrongly insulting cryptography professors or gamers – but if someone really wants to, they are actually able to exhaust resources on other layers, but this actually requires preparation, approach, analysis. We know of many different examples, like banks which, even in Poland last year, one day stopped operating – it was perhaps December 10, the pay day in many Polish companies. Also, it was Monday, so some online shopping got stuck, and it turned out that systems of 4 or 5 Polish banks fell under a DDoS attack generated by their own systems or clients, and in fact those were attacks on other layers. In one bank the mechanism of sending text messages failed, in another the web servers couldn’t handle tasks. Today all it takes is a simple discount on Xiaomi and – as we saw yesterday – it shuts down any server, even a very well prepared one. Although, I have to admit, on Black Friday most of the vendors have stepped up though. Maybe the discounts were not so great. In case of a higher discount, those serves would definitely be down too. As you can see, these attacks on layers other than in the volumetric model are rather spontaneously generated by demand. Very rarely do we have actual attackers who have analysed the victim’s infrastructure. What’s more, analysing what actually works and what doesn’t work without access to monitoring systems is difficult because we attack on layer 7, and we don’t know if we’re already at half success or 90%, or maybe they just added 16 new processors, so they’ll manage somehow. I think that from the attacker’s point of view it’s much simpler to just increase the volume and watch it die, rather than penetrate the higher layers of the protocol. I expect that actually if I heard of some attack where someone was draining resources somewhere in the backend, non-volumetric on the connection, I would rather look for the perpetrators among the employees of the organisation who know where those vulnerabilities are and are able to attack them accordingly.

— Teodor Buchner —

There is indeed a cost effectiveness coefficient in, for example, reflective type of attacks, where you measure the amplification factor, and this attack basically consists in using a service – like in case of LOT – as a reflection point, a springboard to attack someone completely different, and what really matters is the ratio of introduced traffic to removed traffic, because it’s not about the attacker generating just a terabyte per second of traffic, which will then be forwarded somewhere else, but to take a kilobyte and generate a terabyte.

— Adam Haertle —

The best of these reflective attacks generate a factor of up to several tens of times, of the type we send one query to a DNS server that is a dozen bytes long and get several kilobytes in response. Such attacks are searched for by attackers, and I think that if we look at historical graphs of DDoS attacks activity, we have actually graphs of both creation of botnets and of discovering next attack methods, because they are detected, they are used, servers are patched – recently it was Memcached, earlier it was DNS. There are at least dozens of services that have already been used in this type of attack, and it’s just that each time one more is being patched.

Questions:

What will you do if a new type of attack appears?

— Teodor Buchner —

In case of basically software solutions like ours, we will prepare and deploy patches. It is a quick operation. What do we mean by “quick”? The first line of defence is the configurability of the device itself, and there we can already do a lot in terms of which way it can extend the scope of protection. Even the simplest attacks like using a new protocol can be handled through configuration. If, on the other hand, there is an attack that requires more precision in separating the wheat from the chaff, then I think that within 24 hours.

— Adam Haertle —

There are indeed attacks that happen several times in a row, but it seems to me that the issue here is that it rarely happens that someone is attacked with something completely new that has not happened before. Unless we are actually talking about these application-based attacks, targeted at a specific customer, most often we learn about the appearance of a new attack from a report of a large company dealing with anti-DDoS protection and the next day all companies are more or less prepared for this attack and before there are any more victims, security solutions are already in place. Hackers use it anyway, because many companies do not use anti-DDoS solutions, so there is no shortage of victims, but the knowledge of how to protect against attacks spreads quite quickly, so I think we can safely assume that the next day most of these customers are already protected.

— Teodor Buchner —

It’s quite rare that a service in a company that operates on that exact port gets attacked, so as a result, if someone attacks me with a protocol for Xbox, that’s something that I can essentially cut out on the connection using various methods.

Will there be an implementation with SIEM systems?

Basically, we are providing logs in a format that is suitable for the collector, consequently adding them to the SIEM is not a matter of some rocket science. There is a standard for these logs, so integration is possible.

So far EXATEL has not been involved in developing solutions for the security industry. Do you have any expertise in this area?

We have been present in the security industry for a long time and we started to build the Security Operations Centre a few years ago – it was one of the first commercial SOCs in Poland – and this SOC created solutions for our own internal needs. As a result, many different ideas were actually used thanks to various talented people who worked at that SOC and we are continuing a tradition.

Who actually wrote this system: you or the Warsaw University of Technology?

This is an interesting issue: what is the dynamics of security issues development at Polish universities. From the academic perspective, this development has accelerated, so there will certainly be many more competences, whereas for traditional units handling telecommunications, entering server security is often a new issue, so these are competences that we have been building together.

There’s also a question about machine learning. Here, when using a connection with such a capacity, there is little time to make decisions, therefore machine learning solutions are more suited to the philosophy of Deep Packet Inspection, i.e. if we have a packet which we want to examine in more depth, we have time to let it go through machine learning, but we managed to use a perceptron code, and that’s basically all that really works there in relation to machine learning – but of course we see a model of using such techniques at further stages of packet processing.

In what directions do you want to develop this product?

Here the challenge is indeed non-volumetric attacks. Maybe their importance is not as great due to the fact that they are not used as much by the hackers as they don’t have the same knock-down power as a .45 ACP, but nevertheless they can overload systems and because of this it is worth being aware that this type of attack has occurred. Apart from that, we’ve discovered a valuable mine of information, which is a global routing table where many interesting things can be found and that may have some significance in relation to DDoS attacks, e.g. SSDP – a protocol for recognising routers. Its frequent source is a lot of unprotected hosts that can be used for a reflective attack using SSDP in South Africa – and we discovered that it’s a frequent problem there. So, this is the kind of information that can be tracked by analysing the global routing table, and a lot of different interesting things happen there too, so maybe when we gather more knowledge we’ll tell you about it.

And how do you want to compete with solutions that have been invested in for 20 years?

Cisco was founded by two students, working in a garage. Every company, even a really big one, develops its competence from some level. We are developing ours on the basis of what we know as a telecommunications operator which has been on the market for 20 years, and I think we have gathered a certain amount of knowledge about the dynamics of networks and their configuration, so this is the capital on which we can build practically everything.

— Adam Haertle —

We are going to answer that question in 20 years’ time.