Equifax and cybersecurity – a deep dive

Equifax, one of the largest credit agencies in the US, was the subject of a hack in 2017 that left over 148 million consumers affected.

It was known at the time that the cause was an unpatched Apache Struts (an open source framework for building Java web apps) server in the Equifax environment, but a recent report from a US House Oversight Committee sheds further light on the situation. The overall findings were that the breach was “entirely preventable”.

What’s new?

There is a lot to unpack in the report, there were certainly many more issues than a single unpatched server. Procedural failures, lack of processes, poor internal communication, incomplete inventory – all sorts!

The attack focused on ACIS (Automated Customer Interview System), a custom built, internet facing, customer portal, and a big part of the problem was due to expired SSL certificates.

What are SSL certificates?

SSL stands for Secure Sockets Layer and is a “cryptographic protocol” used to create secure connections across networks. SSL certificates are used to authenticate the identity of a website (or other service) and that it is secure; when you see the padlock icon in the browser bar, that’s an SSL certificate saying the connection safe.

Right, back to the programme

Within a week of the Apache Struts vulnerability being announced, Equifax performed Intrusion Detection System (IDS) scans which returned no results. This was because a required SSL certificate had expired. In fact, it had expired 19 months previously! Upon its eventual renewal, staff “immediately noticed suspicious web traffic”.

All in all, Equifax allowed “at least 324” security certificates to expire, 79 of which were for “business critical domains”.

After the recent O2 outage (caused by an expired software certificate), I canvassed opinion as to whether ITAM teams are currently involved in the management of SSL and software certificates. The responses were mixed, with disagreement as to whether ITAM should be involved at all. It’s abundantly clear that someone needs to be in control of this process within each organisation – and I would say that ITAM are the candidate best suited to the job.

An Equifax employee said they needed to “define who owns SSL [maintenance] and “create an SSL certificate update process”. I’m not proposing ITAM deal with the everyday minutiae and technical aspects – but that certainly sounds like something an ITAM team would be well equipped to handle.

The House Committee report says:

 “Had Equifax implemented a certificate management process with defined roles and responsibilities, the SSL certificate on the device monitoring the ACIS platform would have been active when the intrusion began on May 13, 2017. The company would have been able to see the suspicious traffic to and from the ACIS platform much earlier – potentially mitigating or preventing the data breach.”

Patching

The patching oversights within Equifax appear more widespread than they first seemed. Equifax had a “Patch Management Policy” that specified defined roles for people to have patching responsibility – however the roles were never filled.

The report highlights that Equifax “relied on its employees to know the source and version of all software running on a certain application in order to manually initiate the patching process”. At the time of the attack, Equifax had no automated patching tools – despite management recommending these be implemented by December 31, 2016.

Graeme Payne, former Equifax SVP and CIO for Global Corporate Platforms, the person with ultimate responsibility for the ACIS environment, testified that:

“at the time that the breach was announced, I wasn’t even aware that we were running Apache Struts in the particular environment”

He only became aware when staff asked him to help shut the system down due to the breach. It’s of note that Payne was fired for failing to forward an email relating to the Apache Struts vulnerability.

The report reveals Equifax staff had identified Apache Struts on the system in question during the remediation of an earlier vulnerability in January 2017…but were “surprised to discover” its presence there in July 2017. Clearly information was not flowing to where it should, nor was it being recorded and made available for use in future scenarios.

Timeline of the breach

The timeline for the breach is this:

• March 7, 2017: Apache Struts patch released
• March 8, 2017: Equifax are alerted by US-CERT (US Computer Emergency Readiness Team) to patch the vulnerability
• March 9, 2017: Equifax Global Threat & Vulnerability Management (GTVM) team notifies 430 people internally and requests systems are patched within 48 hours
• March 10, 2017: First evidence of attackers exploiting the vulnerability on Equifax servers
• March 15, 2017: Equifax run scans to identify systems with the Apache Struts vulnerability. They don’t find it on any externally facing systems.
• March 16, 2017: GTVM remind in an internal meeting that Apache Struts needs patching. 430 people receive the meeting slides.
• May 13, 2017: Attackers enter the network and gain access to databases.

1. Initial timeline on Equifax Apache breach

Equifax CIO, David Webb, testified before the House Committee that, if the system had been patched within 48 hours, this would have been a “preventable incident”.

You’re probably thinking “Wow 48 hours? That’s a very short timeline for a large organisation”, and perhaps you’re right. However, that’s what Equifax’s security policy stated and perhaps if they’d implemented the changes recommended in 2015, it would have been possible.

Even if 48 hours was a step too far*, it should be remembered that this wasn’t a one-time “smash ‘n’ grab” raid. The attack “proper” – where databases were accessed and personal data stolen – didn’t start until May 13, so staff had over 2 months to get the patch in place.

The subsequent attack lasted for 76 days, during which the attackers sent 9,000 queries to 48 databases and located unencrypted Personally Identifiable Information (PII) 265 times.

Of course, even had they done so – the expired SSL certificates, sub-par inventory, and poor internal communication would still have caused problems.

*If 48 hours was an impossible target, surely that should have been discussed and the security policy raised accordingly?

Legacy systems

ACIS was a custom-built system, first designed in the 1970s and expanded and enhanced multiple times over the next few decades. Payne notes that, when it came to performing inventory, this caused them problems saying:

“it is hard for those tools to even identify all the components of those systems”

and that, in a more modern system, the software would be “more known” and able to be scanned and tagged by inventory tools.

Technical debt is a significant issue for many organisations, and it is often decided that it’s easier/safer/cheaper to leave it alone, and that any attempt to upgrade the components or migrate to a new system would be too risky/expensive. The various consequences suffered by Equifax should hopefully change that conversation in boardrooms across the globe; when performing the risk analysis of a modernisation project, it is now perhaps obvious the risks presented by not modernising systems. As well as the reputational damage, Equifax stated in their 2018 3^rd quarter earnings that they expect total costs relating to the cybersecurity incident to be “in excess of $350 million for the full year 2018”. Organisations should also consider other potential costs such as fines under schemes like the GDPR.

Where ITAM could have helped

It was identified in the House Committee report that both the IT and Security teams separately held “multiple and incomplete software inventory lists” and, as the reports states, accurate inventory lists are required to “operate, patch, and monitor” a company’s IT systems.

In a 2015 audit of Equifax’s patch management, it was stated they “lacked adequate asset management procedures” and that a “comprehensive IT asset inventory…or a global view of IT infrastructure did not exist”. They were tasked with improving these by June 30, 2017.

Susan Mauldin, then CSO (Chief Security Officer), in her statement regarding inventory of the use of Apache Struts software said:

“…there were various inventory lists around, and I know that in Security, we had our own list…I’m not sure what IT had.”

However, she went on to say that the lack of a comprehensive inventory “would not, from a security perspective, keep us from doing our job properly”. A statement many (including the USA’s National Institute of Standards & Technology (NIST)) would disagree with  and, I think it’s fair to say, the situation proves to be a misguided belief.

The NIST ITAM standard (published post-Equifax) states “To effectively manage, use, and secure each of those assets, you need to know their locations and functions”

ITAM and Security

The House Committee report has some great comments that really show the importance of good ITAM across an organisation:

“In order to effectively implement a patching process, an entity must have a comprehensive inventory of IT assets”

and

“If an organization does not know what is on its networks, it will not know where patching is needed”

The House Committee echo what we at ITAM Review, and many others in the industry, are saying:

“Responsibility for the proper management of IT risk must be shared between the IT and Security teams”

 This relates to the need for stakeholder engagement too – different teams running different reports with different tools for different reasons is bound to provide different results. ITAM and Security working together, pooling resources, and sharing knowledge is much more likely to give a complete, holistic asset overview.

We saw that knowledge of the whereabouts of Apache Struts servers was lost internally so one must surmise that record keeping wasn’t high on the list of tasks with Equifax (which seems a little ironic). Even if you do have detailed records – are they available to everyone who may need that information?

“tools also require people and process to operate and to function properly”

Which is a great example of understanding that a tool isn’t a silver bullet solution for ITAM.

Conclusion

This look inside the workings of Equifax gives plenty of teachable moments, including:

• Importance of inventory
• Importance of patching
• Why ITAM & Security must work together
• How important good communication is
• ITAM being key to wider business security and success

Some of these lessons are for ITAM professionals while others are for CIOs, CEOs, CISOs and other C-level executives. There are things here that will strengthen almost any ITAM request, be they for funding, additional staff, extra remit, better lines of internal communication and more.

It is real word proof that processes, such as patching, and relationships, such as that between ITAM and Security, must be present, strong, and valued within organisations – and the potential dangers when they’re not.

Addendum: Moving forwards

Current Equifax CTO, Bryson Koehler, joined in June 2018 and has the task of moving forwards from the breach and making sure it doesn’t happen again. A recent interview he gave to Forbes gives an interesting insight into their plans.

Koehler says “security and compliance work” is one of their 3 core concepts and that they have “dramatically changed” their culture and philosophy. He states “It is critical that we do not view security as a project off to the side”; I wonder if that same focus is being applied to ITAM and/or if ITAM is inter-linked with security as it so clearly must be.

Further Reading

House Oversight Committee report – https://oversight.house.gov/wp-content/uploads/2018/12/Equifax-Report.pdf

Equifax 3rd quarter 2018 earnings report – https://www.prnewswire.com/news-releases/equifax-releases-third-quarter-2018-results-300737406.html

Forbes interview with Equifax CTO – https://www.forbes.com/sites/peterhigh/2019/01/21/equifax-cto-on-the-companys-path-back-from-the-breach

O2 Outage – https://www.computerweekly.com/news/252454067/O2-outage-highlights-importance-of-software-certificate-audits