How FUDO PAM system protects businesses against cyber threats

Paweł Deyk – Project Manager– Sales Support Team, EXATEL

Rafał Wiosna – Presales Engineer, Fudo Security

Mariusz Wołoszyn – Machine Learning Engineer, Fudo Security

Paweł Deyk:

On today’s webinar, which I will present together with Rafał Wiosna and Mariusz Wołoszyn from Fudo Security, we’ll talk a little bit about the FUDO PAM solution. My colleagues will explain and present to you how this solution works.

EXATEL has been on the market for over 20 years. We have been associated primarily with telecommunications, but for several years now we have been increasingly involved in the security. We are a state-owned company with 100% Polish capital. One of our differentiators is the fact that we have three teams operating around the clock 365 days a year. Those are the Customer Service Centre (CSC) which includes multiple telecom operators, the Net Operations Center (NOC) team which monitors the network (both the backbone and our customers’) and additionally, from the NOC team, we have separated the Security Operations Center (SOC) team which deals with security incidents.

In order for the SOC team to work well, it must be equipped with the right tools. One of such tools, which we call the heart of the SOC team, is the SIEM software (an event and log correlation system). To make it work, we need lots of good quality sources. We can have numerous systems connected, from firewalls to anti-DDoS systems, or Web Application Firewall. It can also be a system for managing sessions and privileged users. Our SOC team also uses these tools to handle incidents our clients deal with – through a remote session. This makes us, EXATEL, safe, because we can document what we have done in particular systems and the customer can also feel safe, because they can see what we do, so that they know what to expect.

Rafał Wiosna:

Hello everyone, my name is Rafał Wiosna and I’m a Presales Engineer at Fudo Security. First, I will introduce the Fudo solution to you and then Mariusz will tell you about a new feature of this system that is available since version 4.0 – the artificial intelligence module. At the end I will show you its demo and demonstrate how Fudo works.

Fudo has been in the market since 2005; Fudo PAM became our partner in 2012. Our corporate headquarters are located in the United States, while all production, support and sales departments for Europe and the rest of the world (outside the Americas) are here in Warsaw. With over 6 years of software development, we are currently the most comprehensive PAM on the market. We have many clients, mainly companies working within the financial industry and telecom operators. Wherever there is a need to watch what external contractors do in the network (in the case of telecom operators – what administrators do), Fudo would be the best tool to help with that. A colleague of ours was at the bank recently, diagnosing something, and it turned out that there were over 300 sessions established by Fudo, recorded and supervised by our solution.

The main purpose of installing the Fudo system is to monitor what our external contractors and administrators do (if there is such a need, e.g. in banks it is particularly important and has been sanctioned by the Polish Financial Supervision Authority recommendation). What is important here, on this slide, is that the information about what is going on in the network, who broke something, who sent something to themselves (for example, some database), does not surface months or years after the whole incident. With Fudo this kind of incident can be discovered – without a module – within a month, days even. Currently, we are in a position to proactively combat these events. Detecting whether someone is using someone else’s account, guessed their password (or it was given to them), thanks to the artificial intelligence module, is child’s play.

One of the main problems is third-party contractors sniffing around our systems, whether over VPN or IPSec, and accessing systems that contain sensitive data. Sensitive data is not only databases, but also our customers’ data. We don’t really know what these outside contractors could be doing with this data. For example, one of our clients encountered a problem when one of their databases leaked to the Internet. This clients had Fudo PAM and it turned out that an external contractor simply copied that database and sent it outside – and they did it by searching through mysqldump sessions. This external contractor, under the guise of analysing orders from the client, tracked this data back thanks to the fact that they were using Fudo PAM. Fudo PAM will record such actions and changes made to the system; most importantly, it is a system with privileged access, i.e. it records RDP and SSH sessions (the most commonly used protocols) and is able to record data processing sessions. It can function as a data access audit device. One of our clients in Germany has used Fudo to ensure that if an auditor visits the company, they have all the information already recorded.

Fudo makes it very easy to check whether anyone is doing something wrong with our systems. It’s also a good way to record evidence that someone performed actions that are considered a crime, such as siphoning data off and exporting it outside the company. Thanks to the fact that we cooperate with companies that time-stamp and their certificates are recognised by the courts, with the help of Fudo we can export a record of the session which will serve as a piece of evidence in court. These are extreme cases because we usually want to check what our contractors or administrators (in the case of SOC) are doing. So we’ll check their authorisations, see what they’re changing there. Fudo works 100% on the protocol layer. It is not an agent and you do not need to install anything on your system. This solution grants the possibility to record everything that is going on and what is being put into the network, and what changes are being made to the servers. Nothing will be hidden from your sight. A client once asked me: “What if a user opens a window and changes the font to black?” Fudo will still register that, since that change has been introduced, even though other programs similar to it may not acknowledge it.

When many people think of a PAM system, they actually have Password Vault in mind. This term is usually equated with PAM, whereas PAM is something different than Password Vault. The role of the Password Vault, which is often installed as a PAM, is to allow access to servers. It functions as our security guard who gives us a ticket to access the server. Besides that we can’t really tell what the given user was doing on the server. Not only does Fudo PAM have a small Password Vault inside (we can manage passwords on servers in Fudo), but it also records whole sessions. It’s the equivalent of a camera that records exactly what’s going on. Without this, we can’t tell what our administrator or external user is doing when using our systems.

Fudo PAM has three important modules. The first is a session management system that records protocol sessions. The second module is Fudo PAM’s functionality core. It is Secret Manager, a sort of Password Vault, but used a little bit differently. Secret Manager allows us to secure accounts on servers. It looks like a user logs in with one pair of credentials (e.g. from Active Directory) and connects to an account in case of which the password is known only to Fudo. Service accounts, accounts on devices to which you do not want to give passwords to anyone – you can do it easily through a simple configuration in Fudo. This solution has built-in modules for Cisco, Linux – through WinRM or LDAP Windows or domain respectively can be managed by Fudo. The third module is a nice tool for people who need to verify what they pay their contractors for. This is a productivity analysis module. Fudo records the entire protocol, which makes it very easy to count how much time individual users have spent on these servers. If the company issues invoice for 17 hours of work, you can check whether they actually worked for that amount of hours. What’s more, we can check whether they didn’t just log in, work for about 15 minutes, and then just watch some movie on Netflix in the background – and then charge us for who knows how many hours they pretended to work. With our tool, you can verify their work. The person who manages Fudo has access to a session overview. Fudo calculates the percentage of activity in a session, so you can go to your contractor and say: “Wait, fellas, you fined me for 17 hours of work, but you actually worked for 3 and the rest was just you goofing around. I have it all reported right here.” So there are three main modules of Fudo PAM. But the coolest thing we’ve introduced in version 4.0 must be the artificial intelligence module tied to the session recording module.

Mariusz Wołoszyn:

Before I talk about the AI module, let me explain how recording works in Fudo. Well, we record the protocol of the entire session exactly as it took place. Therefore, for an SSH session, it is simply a video of the entire contents of the terminal, showing all the actions performed, both the commands typed and the results. So, you have access to all the information that is collected, all the keystrokes, all the information that was placed in the SSH streams, etc.

It is the same for RDP. It also records all the communication, you can see all the things that are happening: on screen, the mouse movements, and keystrokes. All this can be recreated in the form of a film where you can see both the image and the performed actions. It can also be subjected to analysis. Thanks to it you can try out various cool things, e.g. detect whether the person who logged in and works on the system is the person who usually does it – who has built some historical data, profile – or whether it is someone who gained access to the account in one way or another and is now trying to do something on our system, maybe not necessarily as intended. For this purpose in Fudo we have two main types of models – the first one is content models, i.e. models that analyse the content of the session in detail. This is a behavioural model for the RDP – analysis how the mouse is moved, and a semantic model for the SSH protocol – analysis of the commands entered. I will later describe how those models work in detail and how they detect all those things, but in general, they are models that evaluate the session. If we have a list of sessions in Fudo, this list will show analysis, evaluation or performance model. It will be in the form of a score which shows whether a session is suspicious, or whether the system considers it to be just fine. The second type of models in Fudo, are quantitative models. They analyse things like the number of connections per unit of time (specific day of the week, hour) so we can check for consistency with historical behaviour. If an employee usually connects to the system on weekends because that is the way they work, the system will not react. However, if an employee does not connect on weekends, or never works at night or does it occasionally, and suddenly there are more connections during those times, it will be detected as an anomaly and reported in the log as a suspicious action. This is not related to a specific session, but to a wider aspect of behaviour and is also reported so that the SOC can take appropriate action.

Rafał:

This happens in real time, so you can see when a session is suspicious.

Mariusz:

Both content and quantitative models run in real time. Of course, some data has to be collected, so in case of content models it will be after a short period of time, but generally we don’t have to wait all day to analyse it.

Currently, there are two model classes: for SSH and for RDP. We also plan to add more, modify the existing ones, etc. At the moment they are designed for the user, however, in the near future, we will also be able to create models for accounts. Fudo enables several people to log in to the same account in the target system, so for example: we have a Root Administrator account or some service account, we cannot modify it, but we would like to grant access to several individuals, and have full accountability, i.e. to know which of our administrators worked as the root, and what is more, we would like to add the possibility to build models not only to individuals, but also to end customers – it is a possibility to perform analysis as well. For these two profiles we can train a model, but Fudo has more capabilities: it can also analyse MySQL, VNC, and raw TCP protocols. However, the AI model doesn’t work with them. The AI model only works for those supported protocols, namely RDP and SSH. We suppose that in the near future the ICA and VNCprotocols will be supported, but currently, in version 4.0., they are not. Per our Fudo for a given protocol, the model must be trained. The model is trained if there is enough historical data – so if you buy a product and insert it today, AI will not work yet because it does not know our users. It hasn’t learnt anything about them yet, it has to collect information and build models. This process takes some time. Everything depends on how intensively your users work and use Fudo.

What does real-time analysis look like? Well, to do real-time analysis, we need to analyse the recorded sessions piece by piece. Therefore, data has to be reprocessed and “cut into pieces” during recording. There are specific numbers of events in these session parts. Events are actions done by the user, like pressing a key or moving the mouse. Depending on how intensively a person works with the system, how much they move the mouse, how many commands they type, at what speed they read the manual or how intensively they write a script, this translates into time. The number of events, on the other hand, is what Fudo and AI are working on. We can’t tell exactly how many minutes one session is, but we know how many events it consists of. Therefore we have some statistical information about how long does a session last for each protocol, but it depends on the specifics of the work. As Rafał mentioned, what the Efficiency Analyzer does – when someone logs in and for two hours reads the manual, or runs the editor and basically does nothing – won’t translate to much. Events don’t happen and information is not collected, even though the session is quite long. So we cannot say that after 40 hours we already have a model, because in practice it may turn out to be much faster if users work intensively. In other situations, it can take much longer. Sessions are analysed in real time (as they are recorded), while session analysis (depending on how many simultaneous sessions we have) may be ready with smaller or larger delay. It’s all set up so that at the latest when the session ends, all sessions are analysed. I say this because in most extreme situations where we suddenly have a thousand sessions, Fudo will have to analyse them at the same time, so they may only appear in some order. In the meantime we have scoring and it may change as a result of performing analysis in the real time. For example: someone has connected to the network and suddenly performs some strange actions, maybe their mouse got dirty and the model detects that something odd is going on here – this is not normal behaviour, but after a while it comes back to normal because the user has already cleaned the mouse and they continue to work normally. So this score may change, i.e. first we have red and then it’s green and here’s information on how these scores look. They are translated into three threat levels for simpler visualization. This information is presented in the session view along with the name and weight of this model. The weights are selected based on how the model behaved running on the training data and are still there in the form of numerical values associated with the specific algorithm used. All of it is presented in a way that is clear to the user and easily interpreted. Models with a high score can be sent for manual verification. This is also a very important functionality because we can minimise the amount of data subject to review. If there are hundreds of people working in a company and those people are recording their sessions, it’s impossible to watch all this footage. On the other hand, if we are able to flag a smaller percentage of these hundreds or thousands of hours per month, we can already look at it and actually verify whether something abnormal was happening there or not.

But we can also do it faster. It happens that our clients have thousands of hours of recorded sessions, but don’t really know what to do with it. It’s impossible to review, so the module allows us to address it. For each user an individual model is created. All models are also individually calibrated, i.e. the aim is to achieve the best possible model sensitivity for a given user with the lowest possible number of false alarms. This is because, depending on how the users behave, for some users you can create models that are very sensitive and have a low false alarm rate – this is due to the fact that those users tend to work in a similar way: the environment and their work characteristics are the same. Other people every now and then connect from a different computer, do something different, sometimes transfer their computer and account to others and unfortunately for those people, we can get low sensitivity, which is based on the characteristics of that user and how they work. Because the models are individually calibrated, we can keep the best possible balance for everyone. For whom it is possible, we have the best models, and for others – of a slightly lower standard. However, the overall solution works pretty well.

What does the quality of these models depend on? First and foremost is the quality of the historical data. If our historical data contains garbage, i.e., our users are not trained and do not work according to the rules, and a number of people log into one user account and all claim to be Mr. Kowalski – while in fact different people connect to the system and behave in different ways – then the model will learn that Mr. Kowalski is schizophrenic and behaves like many different people at the same time, and in the end it will not react in any way next time Mr. Kowalski account behaves odd, because it will be consistent with what the model has observed so far. Another factor that affects the quality of models is the amount of historical data. Try to train the models as soon as you have the minimum amount of data to do something with it, however, the more of this data we have, the better the model will be, and therefore over time (after a user-specific model is created) these models will be improved when more data is gathered. Also note that there is a parameter there that makes the models not access historical data older than one year. This is due to many studies according to which people change their habits and behaviours, and how someone behaved 2 or 3 years ago may not necessarily be a good indicator for how they behave today. If someone has had a long hiatus and doesn’t have that data, that model just may not be there, or it will be weaker. There are also a few parameters that can be configured from the panel, such as the length of the session parts I mentioned earlier. The longer the part, the more data you have in it and the analysis will be better. But this analysis is also a bit delayed and, for example, the first analysis will not be finished after 30 seconds, but 2 minutes, after more traffic and collected data, because we needed to collect more data per session part. There are still features that we can set for some models, such as the number of features analysed, especially with models involving mouse movement.

How does the SSH model work? The way the SSH model works is that we need at least 65 sessions recorded with minimum 25 unique commands each. These are extreme conditions. You may be able to create the model a little earlier or later, but that’s more or less the amount of time needed, statistically, and you need around 300 sessions for a minimum of 10 people. The way it works is that we analyse keystroke events, we put together “words”, “commands”, and it’s professionally called feature engineering. This way we make that data suitable for machine learning audiences, and that’s where the “magic” happens that makes this data usable. How does this work? Well, it turns out that each person behaves in a characteristic way, everyone has their own habits, people perform certain actions in a different way and if you collect enough historical data about these characteristics and analyse it, you can train a certain algorithm, which is able to determine on the basis of the occurrence or non-occurrence of certain actions, how likely it is that this is a session of a certain user – on the basis whether the particular characteristics occur, or maybe some other, typical for other users. This is of course fed back to the various machine learning devices. You can use both neural networks, support vector machine, and other things, while the crux of the problem is to extract these features and develop an algorithm that can cleverly read them, encode them, and let the AI model do its job. Here we have an example of features for 5 different people. You can see that the first person from the left never uses “less”, because they probably prefer “more”; they rarely use “backspace_key”, because they probably delete using “delete” key; they also don’t use “ctrl_a”, but they probably do some other things, e.g. in vi they move up and down (via k and j keys). Person number 2 uses “backspace_key” and uses “ctrl_a” to jump to the beginning of the line, but does not use “delete”, for example. The first person from the right uses “zz” to exit vi , because as we can see they often use “zz”, its use is characteristic to them, and at the same time they never use “vim” and “wq” because they exit via “zz”. This is of course just an example of data collected for each of those users. For each session, this number of features is extracted. We can see here that the positive features, depending on the person, are between 300 and 700, and the negative features are much more numerous – between 2,500-3,000. Thanks to extracting all these features and building a model, we can detect with a high probability and sensitivity whether this certain user is who they claim to be or maybe someone logged in from China and behaves in an unusual way, despite the fact that e.g. they do completely typical things, because they run the editor and read some files, only that they run not this particular editor and not in the way this user does it or as they have been doing it so far.

Rafał:

Fudo is able to detect in real time that someone has hacked into an account and it’s clear from the session.

Mariusz:

It happened not so long ago that a company realized that someone hacked their system when the drives overflowed and the servers stopped because the hackers had exaggerated the amount of dumped data. Most incidents are actually detected after several months, whereas using our solution this time can be cut to minutes or even seconds – certainly not longer than the duration of a session.

For the RDP (mouse) models, things are a little different because the operating characteristics are also different and our approach is focused on the mouse. Well, it turns out that people who work using a mouse do so in a distinctive way that can be compared somewhat to the handwriting. Everyone has their own unique handwriting. A graphologist is able to recognize whether given samples of handwriting belong to the same person. Even random people can usually tell whether the given text was written by a certain person the handwriting of whom they previously saw. For a computer, these features need to be coded appropriately. In such a case we require about 5 hours of recorded sessions of a minimum of 10 people (optimally about 30 hours) and from these recorded mouse movements we extract the corresponding numerical features that are related to curvature, angles, and if enough of these features are collected, such a cumulative distribution can be constructed. It is possible to build such an algorithm that will be able to distinguish whether the distributions that appear to us for a given user match our historical data or not, and most likely something suspicious is going on here. These models have quite high accuracy and are individually calibrated (as with the SSH model), so for some users we have high accuracy, for others however, a little less. We also have some interesting anecdotes. For example, one of our clients said that they had a problem with their wrist and at work they use a mouse with their right hand and at home with their left. This can also have a negative impact on the model, because if it recognises two people under that account, it will be a little less responsive. If we are aware of such a situation, we should set up two Fudo accounts, one for the right and left hand and get two much more accurate models.

We also have quantitative models that are able to detect anomalies in the timing and number of connections. The algorithm looks complicated, but it’s actually really simple. We predict how much we would normally expect to see of a given user’s sessions and historical data at a given point in time, and then we compare it with the actual state at that point in time and how the model performed historically, i.e. how much wrong we were. If this error is significant, then we are dealing with some kind of anomaly and thus we are able to report it as some kind of event requiring additional action. At the bottom of this slide you can see a chart with the daily distribution of the number of connections for different people and you can see that usually this person started work between 7 and 9 am, but there were times when they worked till 11 pm. We are able to respond to that.

Rafał:

I should mention that Fudo will detect such unusual “shots” like, for example, twenty connections on a Saturday at 1 am. Fudo will immediately detect this as an anomaly.

Mariusz:

These models are about the number but also about the timing of connections. If a given user usually connects for longer periods of time, and suddenly those connections are shorter, then of course there’s full tolerance there – anyone can break a session because they have a poor connection or are trying to connect from the middle of nowhere. Or the given user doesn’t know at all what’s going on in this system, they logged in for the first time and are looking around…but usually this user would log in, do the work and disconnect.

Rafał:

I’ll show you more or less what this AI looks like in action. Of course, it’s difficult to collect models if you don’t work on Fudo all the time. I always install the latest version, so I couldn’t really train any model, but what I did was that I recorded one session and played it probably 100 times and trained this Fudo what my session with this cloud server looks like. By the way Fudo is both appliance and virtual, but now we have a cloud version as well. This version works in AWS, but we can also offer Fudo designed to work on a cloud (all major cloud applications).

Now, what is going on here, on this screen? First, you can see previous sessions (when I conducted the demo) and they were all considered suspicious. And here’s what you were talking about: mouse model (Mouse_Biometric), weight 0.91, 100% sure it was a threat, it wasn’t proper session of this user. Well, it sure wasn’t, because it was me, not a tester who had used the session before and trained the model. Now pay attention: at the very top we have a session that hasn’t finished yet, and already Fudo is displeased, because according to it this session looks different than the ones we artificially played. And indeed, I’m the one connected here. Please note, look here at the top: this is the most ordinary Microsoft Desktop client for Mac. I connected to a server that also has nothing installed on it and this session is visible. I’ll show you how it works in general, because it’s a good eye-opener regarding how Fudo works and how recording a session looks like in day-to-day operation. At the moment Fudo is recording everything I do here, in this window.

Mariusz:

Probably the first question here is: how overloading is this AI. Everyone has heard that you have to run GPU’s and god know now much data is needed. During the creation of this solution we discovered that for online analysis we don’t need huge resources. In practice, this can be done efficiently, effectively and quite quickly, without noticeable increase in CPU usage versus recording sessions. The only time when more resources are needed is for model training, where we have to analyse all this historical data, collect it, process it and create a proper model. If we have a lot of recorded data and we’re doing it for the first time, it can consume some resources, that’s why the default scheduler is set to 4:25 am and you can always change it, but there are also these features and all the operations that are possible, like calculations, that are cached so that you’re able to build new models every day and at the same time put as little load on your computer as possible. Probably the larger Fudo models are recommended here, i.e. Fudo 3000, not the smallest one, but if we have a small number of users, the smallest one should work just fine.

Rafał:

On the left is my live protocol session rendered, and on the right is my client, who doesn’t know anyone is watching his system. Please note how it lags. It’s not because the product is weak or because I’m facing some problems here, it’s because it’s on the cloud and that protocol goes to the server through that cloud server, it’s recorded and it’s transmitted in real time, but unfortunately it goes and comes back so slowly that before I move my mouse I have time to finish my coffee. In fact, if you would like to try out our product, we can send you fill version with a time-limited licence. It runs really fast. And what I wanted to show is how exactly Fudo works. On the right you can see a session in which something is happening – and we can look into those sessions, watch them, rewind them, we can see in real time what this user of ours has done, and now, because of the conditions that we’ve made, I’m going to do what administrators like best, which is to go back to real time, go forward and disconnect this user.

Mariusz:

Let’s also add that an administrator can join this session and, for example, enter the password for some ancient application. It can’t be done otherwise.

Rafał:

You can also join this session. Here we see sessions that are archived. Please note the Activity column. Fudo enumerates user activity (here we have a user session that lasted 1 hour and 24 minutes). So that means this work performance analysis module works too. We don’t have much time left, I could talk about Fudo forever. Just to mention that this product has been with us for over 6 years, we already have version 4.0 in which the AI module has been added, we have web recording, and I think that in the next webinar we will show how this web recording works. With Fudo we are able to record a session, for example for the VMware platform, including such functions like visibility of how the user moves the mouse, checking how they got to the site and what they did on the site. We have developed such browser-in-browser technology and we record the entire browser activity. If you have any questions, please contact our colleagues at EXATEL, they already have experience with Fudo implementations and can provide you with more information. We can provide a VM with a license at virtually any time. It’s easy. Our users say that once you understand what it is all about, the setup is quite simple. I know this from experience. I was interested in Fudo as I was still working for a company that needed a solution like that. My SAP manager wanted to know what the people working on our servers were doing and we tried out a few products. That was 5 years ago and I think Fudo hasn’t been known yet. Then this one company just showed us a Polish product. The support was great. The engineer installed it and even told us how it worked. All of this because people responsible for SAP wondered what our external contractors were doing on our servers. This is actually the case in most companies that implemented such a solution – a company management decide they want to know what those people using VPNs are doing on their servers.