Security of remote work: How to protect Local Government Units against DDoS attacks (26.05.2020)

Rafał Broda, Project Manager, EXATEL S.A.

Marek Makowski, Product Manager, EXATEL S.A.

— Marek Makowski —

The importance of DDoS attacks has increased significantly in recent times. Of the attacks requiring any technological involvement and actually occurring in cyberspace, this type of attacks is by far the most popular, disregarding Internet crime related to unlawful threats and hate which are addressed in a slightly different way and do not require the support of technology or expert knowledge.

Today we are going to talk about our experience with cyber security, DDoS attacks and how they affect business continuity. We will also mention how to defend against these attacks using the platform we have developed.

04:15

We have been thriving on the telecommunications market for more than 20 years. We provide professional data transmission services for entities operating in Poland and render global wholesale data transmission services for international players and operators. We are responsible for one of the largest east-west data transmission highways. We are also present at all relevant international traffic exchange points. This is the core of our business and it also generates the lion share of the company revenue. However, development and ensured business continuity in this area means that we have to thrive also in areas strongly related to quality assurance. Apart from transmission, we are vigorously developing cyber security, integrator services and are active in the innovation sector. As the network grows, ensuring cyber security is crucial both for us and for our clients. To provide that, we need to know which way we are heading.

Using the example of the 5G network, we can see that the issue of innovation and building a stable image of a reliable partner for the entities we cooperate with is related to the fact that we want to manufacture our own technologies. In 2014, we joined the RAPID-5G EU project. Back then, the technology was not as popular as it is today. Several individuals asked themselves a question: why should a landline operator participate in a project that is related to mobile technologies? Currently, we are the originator and leader of a venture called Polskie 5G (Polish 5G) and a wholesale 5G operator in the 700 MHz band. We are convinced that participation in the development of proprietary technologies and involvement in innovative projects is the right path for EXATEL as a stable and reliable partner we are and want to continue to be.

07:48

Of course, ensuring the security and stability of this network is important to us, hence we propose new initiatives, such as TAMA – our own platform for protection against DDoS attacks – or SDNbox and SDNcore – our proprietary solutions for software network management. This innovative activity is wide enough to establish cooperation with strong scientific centres from Poland and abroad.

But let us focus on the platform itself and the issue of security building. Our solution is distinguished by several basic features: comprehensiveness, scalability, efficiency, flexibility, stability, integrability, transparency and the fact that the solution is designed with the needs of the Polish market in mind. From the very beginning, when we started to create this solution, we wanted it to protect against volumetric attacks which are really a crucial problem in today’s world and looking at the statistics we can see that it is only becoming graver. We also wanted the solution to meet the requirements for performance, scalability and security. It works in a distributed architecture, so it’s not that we have one unit that protects our network but we try to work already at the edge of our network so that the backbone is not used and we can be flexible in meeting the needs of our customers. We own that code, so we are able to develop that platform using the tools that we and the customers think are right.

We depend on absolutely no one. What’s more, we are able to integrate our solution with our customers’ security platforms and this way we developed those fundamental features that became the basis of our solution.

— Rafał Broda — 11:01

To include the features mentioned here, we made certain design assumptions that necessarily followed current programming standards. We live in a DevOps world now – on the one hand, we want to software everything, and on the other, we want to maintain common sense cost, so the obvious direction was to rely on the x86 architecture, its specialized chips or processors. In addition to the base, which is a market standard, we also gained added value – if we created the code, we can also easily integrate with our customers’ needs. We are talking about APIs, or Application Programming Interfaces. It was a value that was satisfied on occasion. Another assumption we’ve made is that there is a trend that you can’t get away from. We can see more and more traffic volumes generated on the Internet, so the solution itself had to provide some kind of scalability, so that when it reached the upper limit, the service would not stop. Currently, individual customers order multi-gigabit connections from us, although a few years ago this was the domain of operators only. Thus, after a few years of development of this product we can say that our assumption was correct, because in fact the scalability of solutions that we achieved is based on the principle of – I will use a cliché expression here – folding more boxes. We are talking about points of contact between operators as well as looking towards individual customers. Another idea was to bring two worlds together – the one of network engineers on one side and the one of developers on the other. Of course, with any operation of this type, there’s a synergy. Sometimes it is an empty word, but in our case it turned out not to be one. From a couple of years of experience that we have with regard to working with that product, there is already some added value that we can say “trickled down” during the product development process. This mainly concerns response times – which are quite fast – and the learning time of the whole system, because this is how anti-DDoS systems should be looked at – as a holistic system. Another assumption was something that is a standard on the operator market and something that we even require from our subcontractors, i.e. the possibility of multi-tenancy, which means a system prepared for many end users, the idea being to separate the business needs of customers while taking into account detailed characteristics of various customers. Customers are different, and the point is that the system should be able to cover all their needs on the one hand, and on the other that their needs – or computing resources – should not overlap, so that the system works independently for each customer. This would be a tailor-made solution. The last big consideration was that the entire system should be quick to identify.

15:23

System security should be considered bearing in mind the following three aspects: confidentiality, integrity and availability. Confidentiality means that we confirm the identity of the person we are communicating with and we know that the person who is on the other end of the line is the one they say they are. The second aspect is data integrity, i.e. ensuring that the data is not changed or modified – that it is original. The third and most important aspect is availability – imagine a situation in which we have prepared servers and a database system, but we cannot access it at any time! We will now focus solely on the accessibility aspect. To sum up, DDoS attack is an attack aimed at loss of availability. When we were designing TAMA, no one could foresee what would be happening on the web that day, i.e. COVID, where the rank of accessibility was growing. You can look at this from two perspectives: the perspective of the end user and the perspective of IT Security administrators. It’s suddenly turned out that a lot of things – from the perspective of the end user – can be done online. Have a look at public affairs, Electronic Platform of Public Administration Services, online shopping or remote working, to name just a few. In April, a well-known security vendor Kaspersky conducted a survey, the result of which showed that 46% of respondents had never worked from home. If working remotely was a complete novelty for so many people, then an effective DDoS attack could somehow disturb it.

Then there’s the perspective of IT administrators who are responsible for the infrastructure. Imagine that we have an entire company set up in such a way that its associates connect through VPN tunnels. The mundane of life – an attack on a VPN concentrator causes the network to be cut off and as a result the company’s work freezes. According to statistics, there are in general three types of attacks: a volumetric attack, i.e. clogging the bottleneck, which is our access to the Internet, an application attack – a specific attack on a server resource (even a specific application), as well as – more and more popular – a mixed attack, i.e. having several features of a volumetric attack and an application attack.

We should also remember that DDoS attacks are often a cover for other actions, so they can be connected with phishing for example. There are some cases when we work in the office, a DDoS attack occurs and there is an official message from the administrators that within the next hour we won’t be able to work. Employees go for a cup of coffee, then go back to their computers and start working with a sense of lost time. Meanwhile, under the cover of this attack, several e-mails could have been sent, e.g. with phishing or ransomware attacks or causing the possibility of data leakage, in the worst case scenario. Let’s remember then that  from the point of view of cyber criminals DDoS can often be a form of reconnaissance of our network.

— Marek Makowski — 20:18

From the perspective of cybercriminals, DDoS attacks are already a thriving business. If we search appropriate internet forums or TOR network, we can order a DDoS attack as a service the price of which starts from literally few dollars for the shortest attacks. Prices obviously go up, depending on the level of complexity, purpose or scenario. If the attackers use botnets based on IoT devices, which we know are poorly secured, easy to take control over and start sending packets from there, then this attack will be cheaper. If, on the other hand, the attackers have servers under their control as part of their equipment in case of which they had to work a little harder to take them over, then this attack will be longer, and the prices will vary depending on the target to be attacked. It turns out, for example, that online stores are not particularly highly valued by cybercriminals, while government infrastructure, on the other hand, involving some higher risk imposes higher prices. There are even price benchmarks published as different prices may be charged in case of a person ordering an attack from Poland, and yet different ones in case of someone from the United States. There is even the option to purchase a monthly subscription for the service or buy it cheaper on Black Friday. As we can see, it is a very well developed business.

DDoS attacks, as Rafał already mentioned, are often a cover for other proper actions. If you can do the right thing yourself, why not outsource the simpler job (namely DDoS attack)? Of course, it can be done. The reasons for action are really varied. Netscout’s report shows that in order for something to sell well you have to prove that you are effective and good and  it. Cybercriminals like to show that they are good and effective as well. Of course, this includes all sorts of extortion payments (e.g. paying 5 Bitcoin to make the attacks on the network stop). Casual vandalism can also be seen in these statistics, and you can find actions to the detriment of the competition.

— Rafał Broda — 24:39

We have the most recent statistics we could get, which is from April of this year. This data is global but it shows a certain tendency and, to tell you the truth, it refers well to the Polish reality. However, the value and speed of those attacks are much lower.

One more thing worth mentioning is the two-dimensionality of the attacks. As we can see in the bottom left corner of the slide (25:00), gigabits and terabits per second are now a standard on this graph. Early April such an attack was reported and this is one dimension. The second dimension, however, which may disrupt operations and limit availability as well is the speed of attacks calculated in “per second” packets and here the values that come up are even 300 Mbps. These are, of course, world statistics and for our country they are correspondingly lower, nevertheless an increasing tendency is also noticeable. A very interesting aspect that will be referenced often here is the length of the attack. Looking closely at these attacks, if we consider only the attacks that can last more than an hour, we are actually narrowing the spectrum of attacks to something like 8%. Over 90% of attacks last less than an hour. For Poland, this advantage is even greater as attacks that last more than an hour are extremely rare. It’s worth remembering that because that’s the nature of today’s attacks.

The next graph shows how DDoS attacks work with the current COVID-19 pandemic. We can see, among other things, that the number of attacks almost doubled in the first quarter of this year compared to the fourth quarter of 2019. We don’t have the statistics for the current quarter yet because we’re in the middle of it but one can guess that those growth will also be sizable.

— Marek Makowski — 27:43

As I mentioned before, from the attackers’ point of view DDoS attacks are a thriving business. Cybercriminals are well aware that many industries work busiest on Monday. It is no different with attackers – the highest percentage of attacks falls on this day of the week.

And what is the characteristics of the attacks? They are exploited effectively. Vulnerabilities related to the most commonly used transmission protocols and TCP synchronization flags play a large role. By flooding the link, they are able to do their job through blocking resources on the side of the devices that are supposed to run communication. More than 90% of attacks are done this way.

As for who is attacking and who is controlling, we also have information from the world. As far as botnets are concerned, the majority of devices generating this sort of traffic come from Brazil, followed by China. The controller is usually the United States – that’s also where most command & control communications come from. We’ve noticed that in Poland this unwanted traffic usually comes from our neighbours, Russia and Germany. These countries most often carry out such attacks on the territory of Poland. We think the data presented in the Kaspersky report is of crucial importance as it shows that there was a threefold increase in attacks on educational resources and official city websites. These attacks accounted for 19% of incidents in the first quarter of 2020, something you definitely need to consider when securing your network.

— Rafał Broda — 32:06

In terms of tools chosen for defence, there are interesting statistics prepared by Netscout, a security vendor, that show that there is a growing awareness that specialised anti-DDoS (sometimes called multi-layer) systems are important. The tools that customers/end users choose to defend themselves are based on what we have, which is a next-generation firewall or mix of firewalls and inspection systems. To conclude, it turns out that with a well-planned attack (or a large attack, for that matter), it’s actually a good thing that these devices we’re talking about have appropriate software, that we have a multi-layered level. However, in case of a serious attack, without cooperation with the operator, our Internet provider, we are not able to handle the situation. There are more and more noticeable tendencies regarding growing awareness that anti-DDos systems must be taken into account when planning Internet access, while the practice itself also shows that there are still attempts to tailor the devices/systems to our needs. However, we have to consider two aspects of such solutions for sure. The first one is an effective DDoS attack on the total throughput. The device we have at the edge of our network will not have a chance to prove how effective it is because all access will be frozen. The second and most important aspect – these devices need to be operated by people who know what they can be used for. The number of IT and Security staff must be increased, and from what we know from the experience of our customers, it is rather difficult, for example due to the needed budget. How to rationally justify the expenses that go to a person responsible for protection against DDoS attacks which occur once in a while and are unpredictable? It’s difficult and that’s why the sensible answer seems to be “as a service” which you can buy from the operator and assign the whole management and protection process (at least of this part of our infrastructure) to our provider.

— Marek Makowski — 34:56

In addition, we know that the operator has slightly different throughputs within its network. A client can buy a link of one or ten GB, but it is known that operator backbone is built on higher throughputs nx10 GBs, nx100 GBs, depending on the needs. In case of an attack on many tens or hundreds of GBs the operator is able to repel this attack within its infrastructure more effectively than in case this attack reaches the “bottleneck” which constitutes access link of a particular client.

Our operator solution, TAMA, monitors traffic entering our network at all traffic exchange points with other operators using netflow. We track traffic and if it’s directed to a given IP address that meets certain assumptions or signatures or exceeds some thresholds that indicate that it’s an ongoing attack, then we are able to launch mitigation actions on our network and repel that attack effectively within the response scenario agreed upon with the customer.

This is a solution based on distributed architecture so this effectiveness and performance is definitely higher. Besides, even if a unit is busy with something else and is no longer able to handle the problem efficiently, we can easily redirect this traffic to another unit and still repel the attack.

— Rafał Broda — 37:25

We mentioned meeting the needs of specific customers and our response is service package. The first package is a standard, fully automatic protection. The next, advanced package includes services provided by our analysists and protection of more objects. Objects, in our understanding, are a very deep level of granulation, because we can treat an entire class C as well as a single IP address as an object. The premium package, on the other hand, is a very sophisticated and tailored offer and we can count the customers to which we provide services in this model on the fingers of one hand.

However, the offer we particularly wanted to draw your attention to and which you will not find in the official announcement, is Anti-DDoS Basic. This package is fully automatic and involves blocking all traffic directed to the attacked IP address. It’s not an additional system – it’s part of our TAMA solution, meaning the same computing resources, the same analytics and active machine learning. This service is characterised by the fact that it relies on continuous analysis. If an attack is detected and predefined static thresholds (which can be modified later) are exceeded, a given address is blocked. Within the service we give you the possibility to use up to three IP prefixes – three objects. Additionally, at the end of the month, if you subscribe to the Anti-DDoS Basic service, you receive a report with alerts and mitigations.

— Marek Makowski — 41:03

It is worth noting that the development project itself has already been completed. We worked in cooperation with National Centre for Research and Development. We continue to successfully develop this system and our development works are scheduled in detail for the next year and a half. In any case, we are constantly analysing the needs of our customers in our market and we constantly strive to adapt this platform to our and our customers’ specifics in order not to focus on functionalities that are rarely used, but actually on what is most needed, so that the mechanisms are as effective as possible and protect both us and our customers. Our network is constantly growing and we are adding cleaning units to it, so that a high level of security is maintained at all times.

— Questions — 44:40

“Do all packets other than TAMA Basic Blackhole block traffic to a given host to layer 4 of the import protocol?”

They don’t block, they clean, so in the case of blackholing we actually deal with blocking all communication to defend a given network, so that all remaining IP addresses can be used as intended. For the other three, we actually pass the traffic through a cleaning unit called Glados in our case, and that traffic is filtered out. In this case, however, we are actually moving in our detection and mitigation to the layer 4 level.

45:51

“How quickly does the system react from the moment of attack to its mitigation?”

First, an attack must be detected, which usually takes some time. In the official SLA we specify a time period of five minutes from the time the attack is detected until mitigation begins. So, the average time is around two minutes.

46:30

“Does passing of the traffic through the scrubbing centre involve any delay?”

It does involve delays because the traffic is routed differently in our network. Of course, the operator network is not small, so the routing protocols decide that the path between the entrance at the point of contact and the client is as short as possible, so we can redirect this traffic through the scrubbing centre – so the delay can occur. The cleaning unit itself introduces a delay of less than one millisecond and another two milliseconds must be added for any additional hops within the network.