TAMA - proprietary Anti-DDoS

The project is made up of several components:

• aperture - probe in C++17
• gladdos - scrubber in C++17 and dpdk
• chell - system configuration, operator and administrator interface; it consists of a backend developed in python/flask and a frontend developed in angular (we use current versions)
• controller - consisting of queue workers and detector, is responsible for collecting statistics.

We also use Debian Stable distribution and the packets that come with it. We work on NoSQL databases: redis and elasticsearch. All components are run in docker containers and managed by docker-compose - in pre-production environments and docker-swarm - during production.

We control the version using Git with a BitBucket remote server. Tasks are organized in JIRA, and documentation developed in Confluence and directly in the code (sphinx). The app is tested on many levels, with each of the components subjected to unit and functional testing. CI is conducted using Jenkins and, depending on the platform, additional tools, among others:

• linting (pylint, tslint, pycodestyle)
• formatting (clang-format, cmake-format, prettier, robot-tidy)
• statistical analysis (SonarQube)
• unit tests (GTest, pytest, jasmine)
• integration tests (selenium, RobotFramework, scapy)
• fuzzing (LINK: article by Michał)
• performance tests (pktgen).

If a code does not pass the tests, it cannot be embedded in the master version – this means it is not implemented. We use so-called quality gates:

• More than 85% lines of code must be tested
• Linter error number cannot increase.

Correct test results enabled implementing a new version within a test environment. This process is triggered on-demand and is automatic, using a parametrized Jenkins project. Ultimately, our testers manually test the application. We develop test scenarios in JIRA, using the Zephyr plugin.

Both the GlaDDoS filter and the Aperture probe require us to extract information like “what configuration applies to a given IP address?”. It is often additionally necessary to use the ‘Longest Prefix Match’, i.e., a configuration for 1.2.0.0/24 has a lower priority than for 1.2.3.0/28, if we enquire about 1.2.3.1, but the configuration actually applies to a 1.2.4.4 query. Sometimes, however, we want to know all applying configs (that is, for 1.2.3.1, both 1.2.3.0/28 and 1.2.0.0/24). Hardware solutions to this problem often include the quite expensive TCAM memory. However, we have developed a purely software-based solution. An algorithm frequently used to solve this issue is the LCTrie. We use a simple TriTrie algorithm, which, in practice, works faster with smaller data sets than the found reference LCTrie implementation. In addition, it is easily expandable, so it works with:

• IPv4 and IPv6,
• longest-prefix-match and ‘all match’,
• small configuration data (thousands of entries) and GeoIP (covering the entire IPv4 space).

TriTrie is generated statically, and then no longer modified. Where we need to maintain a dynamic address list, the code has to be efficient and locks cannot be used (i.e., no memory allocation), we use bloom filters, in the case of which adding an IP address comes down to setting certain bit values to 1. Bloom filters can cause false positive errors when reading, therefore, they cannot be applied in all cases. On the other hand, they do not enable removing added addresses, hence, they are changed and replaced with new, completely clean ones, every now and then.

1. Commented code (doxygen)

• Explaining the intent behind a method, class or code fragment, if necessary. This enables easier understanding of the author’s intentions and verifying whether it really works that way. In the case of modifications, it allows making sure that the intended behaviour is not deteriorating.

2. System documentation in a repo (sphinx)

• architecture
• operating concept
• division into components
• main interfaces
• technical documentation on filter operation (what kinds of attacks are detected and how)
• GlaDDoS network model - how does a network device behave
• administrator documentation
• documentation of system roles and privileges

3. Changelog - description of changes relative to the previous version:

• What is the business sense of changes?
• Release notes: what changes need to be done manually during deployment (e.g., associated with data model migration).
• What are the technical changes relevant from the team’s perspective?

4. Analytical, operational and system documents (confluence)

• What are the functional or visual changes experienced by the users?
• User documentation

5. Task board in JIRA.

Slack cannot be installed on premises and is not end-to-end encrypted. This is why we use a variety of tools, as per need. For instance, we apply Signal - for fast ‘casual’ communication and notifications. This is a cloud-based solution open source product with end-to-end encryption. We also use Mumble - a form of virtual presence that resembles a radio. Usually, the entire team is available ‘on standby’ in rooms, so that it is possible to talk or ‘call’ a meeting without any trouble, much like in a real-life office. Regarding Rocket chat – covers all written ‘ongoing’ communication. Theme rooms and bots that notify us about important events, e.g., like a broken (or repaired) build or performance test results. All this is within our reach, which greatly facilitates work. As to Zoom - if we want to watch or a screen needs to be displayed – this is what we apply. We have our own on-premise server, which makes you feel slightly better.

We use Jenkins to automate the tests. After it receives a notification from Bitbucket, it triggers a series of tests for each of the system components. The integration tests of the GlaDDoS filter mechanisms we undertake are one of the more complicated. They automatically trigger the filter on virtual local TAP interfaces, hook up to them through Python and use Scapy to generate packets and observe GlaDDoS response to transmitted traffic.

We start our cooperation with the client by suggesting network security options, as well as conducting workshops presenting available TAMA functionalities. Based on our standards, we configure the network according to the client’s predefined parameters and product operation scenarios.

We rely on the expert knowledge of people working in our cybersecurity department, as well as SOC operators directly handling the attacks and the network security of our clients. We follow the latest trends and technologies in the cybersecurity domain on an ongoing basis. Within the research activities, we also cooperate with consultants from technical universities and students.

As part of the development work on the TAMA project, we also collect feedback from our clients and SOC operators. This greatly facilitates the assurance of high quality services. Moreover, we analyse the requirements and implementation feasibility of reported functionalities - which enables continuous product improvement.

Our use of agile methodologies enables us to constantly develop our product, implement new functionalities, and collect comments and suggestions from product end-users. Owing to daily meetings, we can solve problems on an ongoing basis and every team member is able to keep track of work progress within the project. After each completed sprint, the project has new functionalities used by the operators or clients. In accordance with agile methods, every completed sprint is analysed by our team in terms of possible streamlining in the work process. Agile project management worked very well in the course of work on the TAMA project.