Do Security Ratings Give a False Sense of Security?

7 minute read

January 2023

Security ratings are a hot, yet controversial topic. They provide a quantifiable assessment of risk, yet some may question their reliability– security ratings may in fact, provide a false sense of security. When all is said and done, are security ratings unreliable data? To what extent can you use them as you make decisions about the vendors you choose to work with?

We dug into these very questions with Norah Beers, CISO for Grayscale Investments, and Chris Nolan, Value Chain Security Leader at Cisco. Our speakers, both experts in their fields, represent two different sides of the risk assessment process. Norah is on the risk management side, and Chris’ company, Cisco, is often one of the vendors companies are evaluating as they try to mitigate risk. CyberGRX’s Peter Finter led the discussion as Norah and Chris shared their perspectives on whether security ratings provide reliable, actionable information. Listen in to the discussion now:

Use Case Examples for Security Ratings

For Norah and the Grayscale team, security ratings provide a good starting point and initial view into a third party’s risk. She notes, “We need to understand the landscape of third-party risk, and get an initial sense of the risk third parties present before even talking to them.” From there, she’s in a better position to have important conversations with vendors and prioritize the actions her team needs to take next. “Security ratings data makes it easier to deal with risk in a more methodical way,” she noted.

Because vendors are also third-party customers, Chris shared how Cisco uses security ratings from a supplier perspective. He commented that they look at security ratings as an exploratory approach, to understand what the public is saying about a particular company and if they need to dig deeper into an area of concern.

How Security Ratings Impact Vendors

On the other side of the table, Chris explains some of the problematic elements of security ratings—particularly from the perspective of the vendors being assessed. He explains that the issue may not necessarily be a false sense of security. Rather, the ratings may raise false alarms that could unnecessarily impact business relationships. “The experience has been mixed,” he begins. “We’ve been engaged with most of the ratings companies because they’re coming through our global customer base. A lot of these companies use a public information approach, using public data and public IP addresses– what’s in the news, what vulnerabilities are out there, and they are using this information to rate or score a company. This doesn’t tell the whole story because it’s an outside-in view.”

For example, Chris explains that ratings companies may use data from an IP address space that’s no longer associated with Cisco. There are also external training virtual machines that Cisco uses temporarily and that end up getting wiped. These may pop up in a ratings system, making Cisco appear less secure than it actually is.

Additionally, Cisco may get hit by the ratings system based on “customer traffic, like cloud, umbrella, or proxy services,” but these handle customer traffic, not Cisco’s. In some cases, Cisco’s own cyber security systems can, ironically, hurt their rating.  “We do a fair amount of threat hunting and have honeypots deployed.” A honeypot is a cyber decoy designed to attract attackers, luring them towards a digital asset they may want to compromise or steal data from. So it comes as no surprise that Cisco’s honeypots produce what amount to “false alarms.” 

We polled our live audience on their perspectives about how their own company is rated. The results were mixed, with the majority of listeners either not confident in their company’s rating or not engaging in the security rating process.

How Vendors Deal with Security Ratings Data

Managing ratings information can be a significant challenge for a vendor, especially if they contain inaccurate information. If the ratings system relies on the wrong kind of information or doesn’t leverage a comprehensive cyber risk data exchange, it can contribute to a string of problematic information.

As Chris puts it, “We’ve been able to mitigate some of the inaccuracies in the past. But they can end up creating an endless cycle of dealing with ratings that are inaccurate.”

Understanding how your own company is rated impacts your confidence level in the ratings assigned to your third parties, and if you can make risk management decisions from ratings.

We polled our audience a second time, asking how confident they are in making decisions based on security ratings alone. The results revealed that only 9% feel very confident in the information security ratings are providing.

How Ratings Impact the Vendor Onboarding Process

Security ratings do provide data and as our speakers both raised, they help in the initial discovery. As for the confidence level of making decisions off them, Norah explains, “it depends on the nature of the vendor and how Grayscale intends to use them.”

At Grayscale, they use an interactive approach as they debate how and if they want to use a vendor. “I ask questions about the nature of the relationship, the data, and the connectivity they’re going to have. Where do I have risks with this vendor?” she added.

The answers to these questions dictate how they move forward. “If the risks are minor– I don’t have a risk scenario that I need to worry about– then I may be able to use ratings alone. But realistically, if there’s any material risk associated with a vendor, I’ll use ratings as a starting point and a prioritization mechanism, but then I’ll use the validation and profiling tools in CyberGRX to make a deeper assessment,” she said.

On the other hand, when companies take a less thoughtful approach, they may preclude what could’ve been an effective business relationship. As Chris says, “We’ll get customers that use a security rating as a sole identifier and Cisco ends up failing their assessments because of the reasons I’ve previously mentioned.”

The Role of Security Ratings in Ongoing Risk Management 

Ratings can also play a role as security experts review vendors with whom they’ve already entered into partnerships. “We use them because we recheck most of our vendors at least annually, more critical ones quarterly,” Norah explains. “They help us focus on what we need to pay attention to. I use those feeds in combination with data I have coming in from other sources.”

Chris shared that when onboarding new vendors, he’s looking at a variety of information, specifically,  industry standard certifications like SOC2, trust material evidence, who their sub processors are, business continuity, disaster recovery, and if they have a CyberGRX assessment, he pulls that in, too. “It’s about getting them onboard with our third party risk process,” he said.

How Predictive Assessment Data Gives Companies an Advantage

With CyberGRX’s Exchange, companies can leverage analysis derived from a machine learning model that can predict how companies are going to answer assessment questionnaires with up to 91% accuracy. This predictive data populates a questionnaire on behalf of that vendor, and many companies use it to triangulate their risk mitigation and accelerate vendor decisions, as they don’t have to wait on assessment data. Both Grayscale and Cisco use this information as they mitigate their third-party risk.

Chris notes that predictive data provides “an important distinction. A lot of these companies have gone through the assessment process with CyberGRX and met with auditors, so I have more confidence in their data than just a blind outside-in perspective.”

Norah uses predictive assessment data as part of a two-fold strategy. “ On the exploratory side, when I’m thinking about a new outside relationship, I’ve gone out and looked at the predictive data. It helps me determine what kinds of questions I should ask. And, two, when a vendor is not giving me the info I need, I use the predictive score, and it shows what they may or may not be good at. Then I ask if they want to respond to this data. That has been particularly effective in getting a vendor to engage with us.”

Both Chris and Norah recommend taking a thoughtful, balanced approach as you examine your third parties. By combining security ratings with multiple data sources, you can gain a more comprehensive view of the risk each vendor poses. 

To learn more about CyberGRX, we invite you to book a demo. We’ll ask you for a list of your third parties, upload them into our platform, and show your blindspots. See your risks and book a demo now.

Related Articles

About Us

ProcessUnity is a leading provider of cloud-based applications for risk and compliance management. The company’s software as a service (SaaS) platform gives organizations the control to assess, measure, and mitigate risk and to ensure the optimal performance of key business processes. ProcessUnity’s flagship solution, ProcessUnity Vendor Risk Management, protects companies and their brands by reducing risks from third-party vendors and suppliers. ProcessUnity helps customers effectively and efficiently assess and monitor both new and existing vendors – from initial due diligence and onboarding through termination. Headquartered outside of Boston, Massachusetts, ProcessUnity is used by the world’s leading financial service firms and commercial enterprises. For more information, visit