Security Awareness Metrics – What to Measure and How

The purpose of this post is to walk you through and enable you to create a strong metrics framework for your security awareness program. After reading this post you will be able to measure impact, demonstrate value to your leadership and align your program with their strategic priorities.

Defining Awareness

First, what is a security awareness program? It is a structured approach to managing an organization’s human risk. You can gauge and measure the maturity of an awareness program by using the Security Awareness Maturity Model. This blog post assumes you have a mature program (at least Stage Three of the maturity model) and are actively partnered with, or are a part of your security team. Mature awareness programs manage human risk by answering three key questions in this order.

1. Human Risks: What are my top human risks? You cannot manage all human risk, as such you must assess, identify, and prioritize your organization’s top human risks. This should be a data-driven process in partnership with key groups within security such as the Incident Response, Security Operations, Cyber Threat Intelligence or Risk Management teams.
2. Behaviors: What are the key behaviors that most effectively manage those risks? Once again, we need to prioritize behaviors, the fewer behaviors we focus on the more likely people will change those behaviors, and at a lower cost to your organization.
3. Change: How do we motivate and enable people to change those behaviors? One of my favorite behavior change models is the BJ Fogg Behavior model.

Over time, technology, threats, and business requirements change. As such, your organization's human risks, in coordination with your security team, should be reviewed and updated at least annually.

What to Measure

Once you look at security awareness and managing human risk through this lens it becomes much easier to identify what metrics you should be focusing on. Measure what you care about. What do you care about? Your top human risks and the behaviors that most effectively manage those risks. To date, I’ve been hesitant to suggest to organizations exactly what risks and behaviors they should focus on, as risks are often unique to each organization. However, in this post I’m going to try and do just that.

I’m doing this for two reasons. One, my concern is that too many organizations simply don’t have the data / resources to identify their top human risks, as such they don’t know where to start. Two, I’m seeing in many cases it doesn’t matter as almost all the data resources I have been researching such as the annual Verizon DBIR Report, CISA Essentials, and this year’s NCSA / CybSafe Report point to the same finding, most organizations share the same top three human risks – Phishing, Passwords and Updating. As such, I’m going to define these risks, the behaviors that manage these risks, and how to measure those behaviors. Consider this a starting point. If you don’t have any data on your top human risks, this is a fantastic place to start. If you do have the data you need, modify this list as you see fit.

One thing you should decide beforehand is if you want to measure and track behavior by individual or by role / department / business unit. If tracking at the individual level be sure you are taking measures to protect the information and privacy of every individual. Depending on the size of your organization and the amount of data you are collecting, you may also need to partner with someone in your organization who specializes in data analytics / business intelligence to help you normalize / analyze findings. Phishing

Phishing for three years now has been the number one driver of breaches at a global level (2021 Verizon DBIR Report – p15). No matter the number of technical controls we throw at this problem, cyber attackers simply adapt and bypass them. As such we need to teach people how to identify and report these attacks. So, what do we measure? After people have been trained, measure their susceptibility to phishing attacks. Of our top human risks this one is the simplest to measure and why it is such a common metric.

1. Click Rates: Measure the overall click rate of your organization. When you first roll out phishing training this number will drop fast, perhaps from a 20% click rate to less than 2% click rate for more basic phishing templates. Once you are at around 2-3% click rate you may need to start using more difficult / targeted phishing templates. Most phishing vendors support a tiered approach enabling you to use different categories of phishing difficulty. Remember, your goal is not a 0% click rate, as once you hit 2% or less click rate with basic, beginner level phishing lures, your first-time clickers are primarily new hires, and this is a training event for them.
2. Repeat Click Rates: For many organizations this is their most valuable phishing metric as this measures your repeat clickers - the people who are not changing behavior and represent a far greater risk to your organization.
3. Reporting Rates: If you are training and enabling your workforce to report suspected phishing emails, this helps develop your Human Sensor network. For this, it’s not so much the number of people that report that is key, but how fast your security team gets the first reports. The sooner people report a suspected incident, the faster the security team can respond and manage potential incidents. People who report represent the most resilient of your workforce, as they are not only identifying attacks, but enabling the security team to respond and secure the entire organization more proactively.

Passwords

For several years now passwords continue to also be a primary driver of breaches. Cyber attackers have changed their TTPs (Tactics, Techniques and Procedures), moving from gaining access or lateral movement by continually hacking into and infecting systems to using legitimate accounts to more easily pivot and traverse through a victim organization while avoiding detection. As such, both strong passwords and the secure use of those passwords have become key.

1. Strong Passwords: Ensure people are adapting and using strong passwords. Length is the new entropy; passphrases are now highly encouraged. This can be tested by running brute force / cracking solutions against password databases.
2. Password Manager Adoption: We in many ways have made passwords difficult, confusing, and even intimidating for people with various rules and policies. As such, organizations are starting to adopt password managers to make passwords simpler for their workforce. If your organization is / has deployed Password Managers, measure the Password Manager adoption, and use rate. What percentage of your workforce is using Password Managers? You should be able to pull this data from which ever department is deploying / managing Password Managers.
3. Multi-Factor Authentication Adoption: Like Password Managers, if you have rolled out MFA attempt to identify how much of your workforce has adopted it. MFA is especially important for critical or sensitive accounts. Once again, this information should be accessible from whomever is responsible for deploying the MFA solution, responsible for the logging of authentication systems, leads Identity and Access Management, or part of Operations or Security.
4. Password Reuse / Password Sharing: Are people reusing the same password across different work accounts (or even worse reusing work and personal accounts)? Or are people sharing their passwords with fellow co-workers? While this behavior sounds difficult to measure you can effectively measure both behaviors with a security behavior / culture survey. The key is using a scientific approach to how you both write and measure the survey results. For example, one way to measure password sharing would be to ask your workforce

On a scale of 1 – 5, how likely would one of your co-workers share their password with a fellow employee.

If you are unable to launch your own survey, partner with Human Resources and see if you can add several security questions to any type of HR led Employee Engagement or Pulse surveys. Another option is to leverage your Security Ambassadors or security portal.

Updating

Of the three human risks we cover, this one may not apply. We want to ensure the computers and devices people are using, and the applications and apps installed on them, are updated and current. For some organizations this is not an issue as people do not have admin rights or control over work issued devices, instead their devices are actively patched by IT. However, for many organizations this is an issue as so many people are now working remotely from home and are often using personal devices or home networks for work access. There are several ways to measure this.

1. For any devices your organization issues, your Operations, IT, or perhaps even Vulnerability Management teams should be able to remotely track the update status of those devices. In some cases, solutions such as MDM (Mobile Device Management) may be installed on personal devices which can also track updating status.
2. Your Learning Management System (LMS) or phishing platform may be able to automatically track the device, operating system and browser version of any device that connects to them.
3. Assess and survey your workforce to determine if they understand the importance of updating and are actively updating their personal devices, to include enabling automatic updating.

Strategic Metrics

Once you start collecting metrics on peoples’ behaviors, you can use this data to better understand and manage your overall human risk. Three key uses include

• Identify what regions, departments, or business units have the fewest secure behaviors and represent the greatest risk to the organization.

• Identify what regions, departments, or business units are most successfully changing behavior . . . and why. Use lessons learned to apply to your less secure departments or regions.

• When an incident does happen, understand whether that individual was trained. Was the department they were in one of the most secure or least secure departments or business units?

You can also demonstrate the strategic value of your program to leadership by aligning behavior with what leadership really cares about.

1. Number of Incidents: As people change behavior, the overall number of incidents should go down, such as number of infected devices due to people falling victim to phishing attacks or account take-overs due to bad passwords.
2. Attacker Dwell Time: The time it takes to detect a successful cyber attacker in your organization should decrease as you develop a Human Sensor network. The less time an attacker is on your network (dwell time) the less damage they can do.
3. Cost of Incidents: By reducing the number of incidents, and the dwell time of successful attackers, we can reduce overall costs.
4. Policy and Audit Violations: As behaviors change we should see a reduction in the number (or severity) of policy and audit violations.

Summary

This list is neither exhaustive nor perfect, but it’s a starting point. There are a huge number of other metrics you can measure, and sources of data for those metrics.

The key however is not to measure everything, instead you are better off measuring your most useful metrics. And to do that, you first need to know what your top human risks are and the behaviors that manage those risks. To learn more about measuring human risk, consider the two-day SANS MGT433 Managing Human Risk course or the advanced five day SANS MGT521 Security Culture course.

Visit SANS Security Awareness for more information on how to build and mature your security awareness program.