Big and enormous incidents, or: the administrator is only human

— Adam Haertle —

Today I will be talking about slip-ups – one of my favourite topics. There are all sorts of interesting things going on at different companies, and today I will try to answer one of the most fundamental questions: “Who messed it up?”. Because there are occasions when we know that something went wrong, but we don’t quite know why or how. Fortunately, sometimes someone writes a report about the incident and based on that report we can tell what really happened. Not just what was shown in the media.

You may remember June 21, 2015. Due to a cyber attack, 11 planes failed to leave the Okęcie Airport. This was a very serious incident that is still cited in the world press as an example of attack on critical infrastructure. This was an airport in a major country and planes could not leave due to a cyber attack, so all eyes are on it to figure out what really happened. What was the first, hot message? The hot message stated the systems were under attack. We were dealing with some kind of attack on the ground systems. It didn’t look so good. Then, a spokesperson informed us that we were the subject of a massive attack.
The massive attack was associated with DDoS, so it was further suggested that this was not the only airline that used similar solutions, so there was a risk that this attack could affect others as well. The situation became even more serious because we were dealing with a global threat. So, we had a potential scenario here – it was a DDoS attack, but we kept looking at these press releases, and at some point it was announced that it wasn’t a hacker attack, but a human error, which is often the cause of various security incidents. But what does a DDoS attack have to do with human error? Things got interesting, especially since the culprit came up in August as the IT director resigned and the company was in search for hackers. So, we had an attack, hackers and human error. Where does the truth lie? We had to wait a while for the truth to be revealed. In September, information that LOT systems were not the target of the hacker’s attack, but rather their tool, leaked to the press and the situation changed. So, it wasn’t LOT that was DDoSed but LOT performed the DDoS attack? How was this possible?

In December 2015, the prosecutor’s office discontinued the proceedings and it turned out that indeed, this situation really took place – LOT performed a DDoS attack on a certain Chinese casino. How could it be true? How come? So far, international reports and presentations at serious conferences have stated that LOT systems were hacked or that LOT was the victim of a DDoS attack from an internal network, which raises the profile of the incident even more. However, the answer was quite simple. Who messed it up? LOT had scheduled a firewall replacement and to do so it hired a company that won a contract in a tender in which the price was one of the main criteria. This company, however, didn’t quite rewrite the rules the way they were supposed to – it was hardware replacement that required a configuration rewrite. If anyone here had ever been responsible for such a project, they know that it is difficult, inconvenient and laborious, and to reduce labour intensity, you simply have to rewrite fewer lines of the rules. And it just happened that the rule that cut off the DNS server from requests from the world didn’t make it into the new configuration, and that DNS server was a so-called open-resolver and could be used in a DDoS attack.
It took hackers who scan the entire Internet for such servers about 24 hours to identify that this particular server was now available to the entire world and they used it for a DDoS attack. So, the fact that the planes didn’t take off that day was because LOT DNS server DDoSed this Chinese casino. Possible? Sure it is. So it wasn’t a hack, it wasn’t a DDoS attack, it was actually a configuration error. Someone did not properly complete the configuration of one device and as a result an attack on the critical infrastructure of a major country took place. So, remember to configure your firewalls correctly – especially with regard to the access to DNS servers.

The second story, a recent one – from this year. It involves Norsk Hydro. This company was painfully attacked by ransomware, and if you’re interested in history, especially World War II, you may associate Norsk Hydro with something completely different. This is because this company was one of the world’s largest producers of heavy water and when Nazi Germany occupied Norway, they decided to continue its production. So they produced a lot of it, and the Allies tried to damage that factory at all costs. The Nazis covered it with a thick layer of concrete, the Allies were bombing it but could not break through. In the end, special forces blew the whole thing up. This was a rather spectacular story. So, now in 2019 Norsk Hydro is making a comeback. On March 18, 2019, Norsk Hydro announced they had a new CEO and it was quite an opening, because on the very next day they announced that they also had ransomware on their network that disabled all their systems. I don’t wish anyone that kind of first day on the job. I believe that the only person who had a worse first day at work was the one in charge of air traffic control in the United States who on September 11 had to ground all the planes that were in American airspace at the moment. CEO of Norsk Hydro comes second in the ranking.

So, on March 19, it turned out that all computers throughout the company didn’t work, and the company relied on industrial processes controlled by computers, and those processes didn’t work either. There are warnings posted on the doors: “Please do not connect your computers to the Internet or the corporate network because we are under a ransomware attack.” What’s interesting, in connection to this attack there was a memo that apart from ProtonMail, the hackers used Polish e-mail service o2. If any of you knows why in addition to ProtonMail they used o2, and why o2 is such an attractive mailbox for international hackers, please let me know. I have no idea, but this is our Polish touch in this matter. What was the effect of such ransomware attack? Of course, the files were encrypted and unreadable. Norsk Hydro – the second largest aluminium producer in the world, a company operating in dozens of countries and employing thousands of people. How is it possible that it was attacked by ransomware that encrypted everything? Didn’t they have some Next-Generation Anti-Ransomware solutions? Of course they had, and the criminals were well aware of it, because the first thing they ran on the infected computers was a certain script.

This script was about 30 pages long and each page was responsible for disabling all possible security solutions. They didn’t even check what was installed on the victim’s computer, they simply ran a 30-kilobyte script, which contained commands to disable all services that may protect these computers from possible infection. As a result, after releasing the script with domain administrator authorisations, one could easily encrypt, infect, and do whatever they wanted with the station.

Once it was encrypted, it was necessary to ensure that no one could get to it too quickly, so administrator passwords were changed – of course this was done by the ransomware – and then log off all sessions and disable all network interfaces. That’s all. No one can get to such a station either remotely or locally, you actually have to enter a password to unlock it. Hackers change the password to a string of characters – at least in this variant – so no one is able to guess it and all they have to do is wait until ransom is paid. Norsk Hydro decided not to pay the ransom, which was a brave decision and I recommend it to all companies that have backups. How did Norsk Hydro deal with this attack? The company points out that one of the things that helped them somehow start responding to this attack in the first place was that they boldly moved their e-mail along with Microsoft authentication to the cloud. It’s not a easy decision, whereas the fact that they had e-mail along with authentication and with AD in the cloud meant that despite such an attack, they could announce the following: “Okay, don’t turn on your computers, but you have your phones, you have your tablets, and you have communication within the organisation,” and communication in the face of such an attack is essential. You don’t have to check and hand out private mailboxes afterwards. Recently, a company in a certain US city was completely encrypted and told its employees, “Set up mailboxes on Gmail”. When Gmail saw that 500 accounts from one IP showed up at once, it blocked them all because they thought it was some kind of botnet activity. So, it’s good to have mail that works in the face of a ransomware attack.

Other companies were also attacked, so this company was not the only victim. However, what was peculiar is that Norsk Hydro announced its failure really nicely. If you’re preparing any response plans in the event of some massive malware attack, or just freezing your company activity, I’d recommend looking through Norsk Hydro’s social media profiles, as they even have videos of their employees talking about how they’re rebuilding the company’s infrastructure. It’s quite an unusual and a surprisingly mature approach that they were actually sharing their processes with the market, saying from day one what had happened, how it had happened, how they were fighting it. And it worked because the market actually appreciated it. There was a clear communication and regular updates. The market appreciated it because the timing of the attack was that dashed line on the chart that showed that this attack didn’t really affect the valuation of the stock. Stock prices depend mainly on international aluminium prices, which is Norsk Hydro’s main product, but the attack itself did not cause the stock price to collapse because the company took a mature approach and showed that it could quickly rebuild its business.
But, back to the main question: who messed it up? So, it turned out that someone got into Norsk Hydro’s servers and got domain admin authorisations, and with those they sent this ransomware to the station. In fact, this was a scenario that can be defended against on a general basis – we are trying to defend our infrastructure and identify serious attacks – but if the hacker already is at this stage, we have basically lost and can only wait for the moment when they run encryption software. While no one has definitively attributed the attack so far, FireEye suggested that the group that had previously targeted payment cards and payment terminals of large retail chains was probably using the same tool that was used at Norsk Hydro. So, this was such a small suggestion that maybe – just maybe – someone was diversifying their business and instead of just stealing payment card data, they were also taking to ransomware extortion.

Next story – Office of Personnel Management. In the United States, it was decided to create a single company to recruit personnel for government agencies and verify access to classified information. In Poland, this system operates in a completely different way. In the US, it is the Office of Personnel Management that issues the equivalents of our security clearances. In addition, this institution handles all government recruitment processes. And one day this institution announces: “Someone broke into our system.” They broke in and stole a total of 4 million CVs. Was it a serious incident? After all, those 4 million CVs (or even more) already were available on LinkedIn, so nothing else was likely to be found there. Just 4 million CVs. After a few days the company said that the number of stolen CVs amounted to 18 million. But I guess it was still nothing to be worried about? It was just 18 million CVs after all. There were probably a billion of them on LinkedIn! A few days went by again, and it became clear that the hackers stole SF-86 forms as well. What is SF-86? If any of you has ever filled out a questionnaire to have access to classified information, this is it. It’s a document in which we answer sometimes simple questions like what’s our name, where do we live, where and when were we born, and sometimes, we answer a little more sophisticated questions, such as: how often do we use drugs, what problems have we had with alcohol, are we divorced, what credits have we taken out, and what are our other possible sensitive data. So it turned out that the content of not only the questions, but also the answers of those several million people who have ever applied for jobs for the United States government were stolen. The question is: could it be any worse? If we work in IT we know that things can always be worse. Always, because a few days later it turned out that the candidates’ fingerprints were also leaked. While addictions and lovers can be changed, fingerprints are a bit more difficult to change, although apparently this can also be done, but it is much more work. So, someone had stolen really sensitive data of US government employees. So we are talking about a very serious case here.
Where did it all start? It turned out that a year before, in November 2013, this company had reported a somewhat unusual incident, namely someone had broken into the infrastructure and stole network documentation. Just that, and they never hacked in again. This is rather surprising – who wanted this documentation alone? What does anyone get out of reading network documentation? Here my professional advice – they won’t steal your documentation if you don’t have it. If you ever want to justify certain gaps in your company records, you can refer to this example. Instead, it became evident that the hackers studied this network documentation, and concluded that it didn’t make sense to hack this company directly and hacked into its two vendors. Possible? Sure it is. The stolen documentation showed that the company was well secured so the hackers realised that if they tried to get in through other routes they would be immediately detected. So they took advantage of points of access that were already there – they simply hacked into subcontractors’ systems and from their infrastructure got into the company’s infrastructure.
On March 20, 2014, a bomb went off as OPM discovered these intruders in its network and decided to approach it in a mature way. There are few institutions that, when an intruder is detected, do not cut them off from the network, but observe them. It requires a lot of faith in our own abilities and willingness to accept risk in order to keep an eye on an intruder in your network, but it also gives you great benefits, because if we have such skills and we believe in stopping the intruder if they reach too far, then we can see what they are looking for, which often also allows us to see who they are and who they represent. So, they had observed this intruder in their network for two months and then decided to remove them. For those two months, the intruder had done nothing that would force the company to kick them out immediately, so this removal was planned out. You may remember this game called “Spy vs. Spy”, it was really cool, I loved it. In this game, we have two spies on two different levels of the house, and that’s exactly what happened with the Office of Personnel Management, because they only saw this spy at the top, and they didn’t see that at the same time they had a second spy group on their network that was doing exactly the same thing as the first one, but they were watching the first one, and they were absolutely convinced that the situation was under control, because they could see these hackers and their every move, and then kick them out of the network. The former group was removed indeed, while the latter continued their work undisturbed, as the company had already rested on its laurels. The effect?

From July 1, 2014 to March 30, 2015, the hackers downloaded document after document, database after database without anyone bothering them. Why were they discovered on March 30? It’s also a funny story, because it happens that we have some licenses, some salesman comes to us and offers a license, talks about the product and says: “Here you have a demo license, install it whenever you want and test it”. And, of course, such companies were also coming to the Office of Personnel Management; one of them was Cylance, which gave trial licenses to some of the company’s employees. So, they kept those licenses in their drawers (because they didn’t really need them and had other things to do), until one day one of those employees said: “I have some free time today, I was cleaning out my drawers, I have this license here, so I’ll test it.” He installed it and saw that a McAfee library was connecting to an opmsecurity.org server. Okay, why not, nothing wrong with that… but wait! We don’t use McAfee! He decided to check who the domain opmsecurity.org was registered to. And, it was registered to Steve Rogers, aka Captain America. It also linked to a second domain: opmlearning.org. Who was this domain registered to? To Tony Stark, also known as Iron Man. So, since they were not using McAfee, and someone had registered domains under such funny data, it could actually be worth looking into. And they started looking into it. Once they looked into it and fired up the full license, one employee said the console lit up like his grandmother’s Christmas tree because they found two thousand malware samples on their network within two months. So sometimes it is worth actually running these trial licenses.

There’s also a rather sad story to it, because the company had found out that they had this infection on their network five days before their scheduled meeting with another vendor of similar software that was also supposed to help with incident response, and they didn’t tell that vendor that they already knew about the infection. The vendor arrived, thought the company knew nothing about any infections and was asked to run a demo. And this demo of course detected the infection, because it wasn’t a bad program either, and the vendor was really excited to have detected an infection in such an important government organisation (such a great entry in the portfolio!). They immediately stepped forward that they would provide everything they could, licenses, employees, servers, and summed up the total cost of handling this incident (not knowing that the company was aware of it) to $800k. But when they issued the invoice, unfortunately it was not paid because there was no purchase order. The lesson here is that before you sell something to someone, make sure the order has been placed and that the person in question actually has intentions of paying that invoice. Because after such a magnificent incident this company was on the verge of bankruptcy. And how much did it cost the Office of Personnel Management? About half a billion dollars so far, because they had to provide some services to the victims – at least monitoring of their account balances and whether someone was taking out loans using their personal data. That was quite an expensive incident… and all you had to do was to turn on that trial license a few months earlier.

Since this was a serious incident, the subject was of interest to the Senate Committee, which in the United States has certain powers and technical capabilities to conduct i.a. analyses of such incidents. The Commission’s report could be summed up more or less like the situation of most Polish companies:

1. No qualified employees and no clearly defined accountability rules (i.e. HR issues),
2. No basic system configuration rules in place,

Even when someone audited a station or server, they didn’t know if that configuration was good or not, because it wasn’t established which configuration would be good.

3. No evidence of vulnerability scanning.

That’s such a nice way of putting things, i.e. employees claimed to perform such scanning, but there was no evidence of it – so most probably they didn’t do it at all.

4. No inventory.

If you’re managing IT, invite your antivirus manager, your Active Directory manager, your Helpdesk manager, and the manager of a key system that was used throughout the company, and have each of them write on a piece of paper how many workstations they have under their control and put that piece of paper in an envelope, and then compare those numbers and find out the spread between the lowest and highest numbers. This is a really cool exercise.

There was update management, but ineffective. A lot of servers were not updated. There was also a SIEM implemented, but 20% of the systems were not connected, and those that were connected generated mostly false positives. Also, it is not enough to connect, you still need to configure it properly. There was two-factor authentication implemented, but only for workstations. There were smart cards for workstations, but it was not required to have a card to log into the application, because everyone assumed that the card to the station was enough. However, if the stations were taken over by hackers, they already logged into the systems without any additional measures. Two-factor authentication for remote access was also implemented, but it was done when the hackers had already been in the network, so several months too late. In addition, after implementing this two-factor authentication, not all network connections were restarted. So, while 2FA was implemented, the hacker sessions continued without 2FA. A really nice case. And finally, the question: who was behind it? This data has never showed up again, it has never been used, it has never been sold, there is no evidence that anyone anywhere has tried to use it.

And there were several such incidents in which data has never come up again. The one we have just talked about (OPM), the Marriott hotel chain (which took over the Starwood hotel chain and stored the data of 500 million of their clients, sometimes including passport scans), the biggest US airline – United Airlines – and Anthem, the equivalent of Polish PZU, one of the biggest insurance companies in the US. Imagine what volumes of data must have been stored by these four institutions – and within two years they had their systems hacked and all their customer databases stolen. Now, what are the chances of a CIA agent landing in China not being recognised before they even leave the airport terminal? These are interesting incidents. This data has never come to the market – someone has it and is using it.

And to top it off, I’ll tell you about an incident of Equifax, one of the largest credit information bureaus in the United States. Equifax, which collects information on all clients in the US, one day announced they had an incident regarding their clients’ information, and they are actually trying to hide the amount of client data that was leaked. As it turned out, their database contained about 150 million records, which means that data of one in every two Americans, not just adults, was in this database and was stolen. How did this incident happen? This story begins on February 14, 2017, when someone discovered a vulnerability in an Apache Struts module. This module was used by many applications and it had rather trivial bugs. There was a problem with its update because it was embedded in the application, so it wasn’t enough to just release the update, you still needed to do quite extensive testing. Whoever had discovered this bug, reported it to Apache quietly, without announcing it on the Internet.

Apache fixed this bug in the next update, but apparently someone studied the updates and noticed that there had been a bug that was very interesting and now it was fixed, because on March 6 an exploit for this bug appeared on the Internet, and this exploit was quite trivial – you just had to type this payload in the header of the http request, so we had command execution by typing a command in the web request in the appropriate place of the field, and this command was executed with the authorisations of the server on the other side. On March 8 Talos, the threat analysis arm of Cisco, stated that they could already see the attacks. Some of them were simple (the hacker asked whether the attack was working), some were a bit more elaborate (the hacker installed their software on the station where this attack was commenced right away). On March 8, when the attacks were actually live, the Department of Homeland Security issued a message stating that “we have a threat and live attacks – please take care of it” and they sent it to every major institution in the country.

On March 9, the attack was described in the internal communication system of Equifax, which had a vulnerability management department, and this department sent out an e-mail to 400 people in the organisation (which shows the scale of the security teams and administration): “there is a vulnerability, you have 48 hours to patch it as it is actively used by hackers”. Also on March 9, the company launched a vulnerability scanner and that scanner failed to detect vulnerabilities on Equifax servers. It didn’t detect any vulnerabilities even though it was a generally good scanner – however, it turned out that it scanned only the main folder of the web server, it didn’t scan subfolders, and that’s exactly where the vulnerable applications were. So it wasn’t enough to scan the server, it had to be set properly to do its job. So the company concluded that they had no vulnerable stations and there was nothing to worry about. Later on, the investigative analysis showed that on March 10 hackers with better scanners attempted to find these vulnerable apps for the first time and tried whether they worked.

On March 13, webshells were already set up on servers, there was already a successful break-in and hackers got into the company’s infrastructure – by the way, they entered through a system that was created in the 1970s. Here in Poland, it is our advantage that most of the systems we have are much newer. In the States, however, these systems had been implemented before Poland was even computerised, and this system had been updated regularly since the 1970s, though apparently not very effectively. On March 14, the security team wrote Snort rules and implemented them, but on March 13 they already had the servers hacked, so they were one day late. On March 15, they scanned the servers two more times, but again they didn’t detect any vulnerability (one of the scanners was provided by an outside vendor). So if a vulnerability scanner determines there is no vulnerability, it doesn’t mean that a criminal scanner will not detect it after all. The hackers got into a web server and on that server 30 different hacker tools were then discovered, and it turned out that by having access to that server they were able to read application configuration files. The database login and password were stored in the application configuration files. Without a doubt, hackers wondered why there was only one login and one password for all databases, but that was unfortunately the way it was – it was often more convenient to administer systems this way. So they took advantage of that and got access to 48 bases. Investigators who came later to see what was going on said the hackers must have sent nearly 10,000 queries to these databases and then took 265 dumps.

Why such a huge disparity? Because those databases were a terrible mess. It’s also some kind of security technique, because even the administrators didn’t know what data was there, so these poor thieves had to really do thousands of queries to even find the data they wanted to steal. The tables were not described or indexed, there was no documentation, so they had to go through the database manually to see what was there, if it was of interest and dump it. And so they spent two months in that network. How do we detect such hackers? On July 29 at 9:00 p.m., someone pointed out that this company also had network monitoring tools. Of course it did. Most large companies do. And it had, among other things, a tool to monitor SSL traffic. Such a man-in-the-middle attack, i.e. substituted certificates instead of the original ones in order to decrypt this SSL traffic and analyse it, see whether anything bad was happening there. However, it turned out that unfortunately, as of January 31, 2016, this device was not working. Why? Because its SSL certificates expired and no one noticed that for a year and a half. So you might have a box for hundreds of thousands of dollars that does a great job as a security tool, only again you forgot to update the certificates on it, no one paid attention to the expired certificate messages that probably came in every day to some unsupported mailbox, until finally someone read the message and said: “Oh dear, SSL session decryption in our monitoring system has been disabled for about a year and a half. John, please go and change the certificate to a valid one.” John would do that and after a minute he noticed traffic to China that shouldn’t be there. Possible? Sure it is. So, once they noticed this traffic to China, it was a big deal, they realised they had been hacked, so they called the biggest incident response company in the world, Mandiant.
Mandiant arrived and unfortunately had to perform some exhausting work, as they had to identify the data that was leaked. Do you remember what the status of these databases was? So Mandiant had to go through all the databases and see what the hackers might have stolen, because no one at the company could tell what was in those databases. The databases work, but the man who set them up is dead. A common scenario in long-established companies. Eventually Mandiant figured out what had happened, and the incident could be communicated. Indeed, on September 15, a statement about the incident was released. Please note the domain used to post this incident message: equifaxsecurity2017.com. This is an invitation to phishers and other criminals. If you have an incident, make a subdomain of the main domain to give it some credibility, because if you make a website like Equifax (equifaxsecurity2017.com), a few hours later some prankster will show up and swap those two words around and make the domain securityequifax2017.com, add some insulting phrases about your competence in the title of the website, and then your customer service staff responding to customers on Twitter will themselves get confused and give the wrong website address. Such a customer, redirected to the wrong website, who is supposed to check if their data leaked, enters their data there (because you have to enter it to check if it leaked) and gets the message “now it’s definitely leaked”. And this is a company that not only runs a credit information bureau, but provides professional incident management services. Clearly, not quite professional. Responding to incidents is also a difficult skill. If you want to show your board some stock chart where a major incident caused the stock price to drop, this is the best example. You won’t find a better one because that first big valley (on the chart) is the security incident where one third of the company’s market value went down the drain.

Another valley is an example that they haven’t quite recovered all their resources yet and are still having problems. In case your board is insightful, they will ask to view other examples of major incidents and here you have six other examples, the presentation of which is going to cancel your security budget. In those cases, each time there was a compromising incident – indicated by a red line – and I mean a really major incident that made it to the world media, and the stock prices were unaffected, because the market doesn’t really care about cyber security. I don’t know what we are working for in this industry at all. Companies can do perfectly well without us, can’t they? Markets just don’t react to these types of incidents – at least not in the long term – or they have great PR departments.

What other effects did this incident have? If you know about some data leak in some listed company, you really shouldn’t trade in its shares, even through your mother-in-law or her neighbour, because then the Financial Supervision Authority will come and beat your hands with a ruler. All those who actually profited from knowing about the incident have been arrested and have to give back the money they made in the process; and some went to jail. And to answer the question “Who messed it up?” for one last time: well, we have specific people here who have messed this up. David Web, the Chief Information Officer who accidentally resigned the day after the incident came to light, Susan Mauldin, the Chief Security Officer who accidentally resigned on the same day as the CIO, we have the CEO who didn’t resign that day and we have a deputy CIO. And that deputy CIO was targeted because when the CIO was going to testify in the Senate, the same day the deputy CIO was fired. And do you know under what pretence was it made?

Remember when I said that the internal security organisation sent an e-mail to 400 people about the necessity to update a vulnerable module? The deputy CIO was one of those 400 people and he was fired for his failure to forward this e-mail. There must have been some scapegoat, although, it’s worth remembering that if you’re fired due to a big incident, then not all is lost. I checked out his LinkedIn profile and what he’s up to now. And guess what? He advises companies on how to manage major incidents. In his profile he even states that he is experienced, because he had “lived through one of the largest cybersecurity breaches of our time”. However, if I am to give you any career advice at all, I suggest you become a CEO, because when he ingloriously left six months later, he was given a mere 90 million dollars as a golden handshake so that he would not be sad that his career was ended with such an incident. I assume that comforted him. So that’s the official advice – it’s better to be healthy and wealthy than poor and ill. I hope you stay healthy and wealthy, and your systems stay safe.