Reflections on the US Government’s OIG Report on CISA’s Automated Indicator Sharing Program

Note: We intentionally held off on publishing this blog until we felt adequate time had passed where we would not influence potential US Government cybersecurity policy discourse.

In late September 2024, the U.S. Department of Homeland Security's (DHS) Office of the Inspector General (OIG) released a memorandum to the Cybersecurity and Infrastructure Security Agency (CISA) Director Jen Easterly. The memorandum outlined findings and recommendations for improving perceived deficiencies with one of its information sharing programs surrounding the "Automated Indicator Sharing (AIS) 2.0 capability."

For context, CISA created the AIS program in 2016 to enable real-time exchange of machine-readable cyber threat indicators (CTIs) – referred to as Indicators of Compromise (IOCs) by the cyber security community – and defensive measures. The program aims to help protect the networks belonging to AIS community participants. In March 2022, CISA upgraded the system to AIS 2.0.

The OIG reported a decline in participation in the AIS program, attributing it to CISA's inability to identify specific program costs and audit expenditure. The report suggested that these financial management issues hindered AIS’s effectiveness in facilitating real-time cyber threat sharing. However, the report erroneously represented AIS as CISA’s sole mechanism for real-time cyber threat information sharing with the private sector, ignoring CISA’s broader efforts to enhance this capability against threats from hostile nation-state and ransomware actors. The report did not evaluate the program’s impact on threat prevention or mitigation; according to the appendix, CISA is in the process of providing such information to the OIG for review.

To its credit, the OIG may focused its evaluation solely on machine-to-machine shared IOCs rather than higher-quality indicators containing surrounding threat activity specific context. The focus on "real-time" machine-to-machine data appeared to largely be evaluated on throughput rather than more useful metrics, as noted above. The report acknowledges that this was just one of many information-sharing capabilities encapsulated in the $31M and $35M expenditures in 2021 and 2022, respectively.  It also acknowledges a longer-term vision under the department’s Threat Intelligence Enterprise Services (TIES) strategy. At least one other information sharing capability—CISA’s Enhanced Cybersecurity Services—was identified as an information sharing capability designed to share sensitive and classified cyber threat information with accredited commercial service providers to detect and block malicious cyber activity.

Background Context on the AIS Program

The Cybersecurity Act of 2015 established a voluntary process for public and private-sector entities to share cyber threat information with each other. The Act requires the Director of National Intelligence (DNI), secretaries of DHS and the Department of Defense (DoD), and the Attorney General (AG) to devise a process to facilitate and promote sharing classified and unclassified CTIs, defensive measures (DMs), and best practices to mitigate cyber threats to the Federal Government and private sector.

In 2016, CISA established the AIS as a solution promised by the Cybersecurity Act of 2015, allowing for real-time exchange of machine-readable IOCs and defensive measures. The AIS program was designed as a mechanism to share information at no cost to participants while allowing them to voluntarily share and receive unclassified cyber threat information. In March 2022, CISA improved its submission process and interface in what is referred to as "AIS 2.0." In a lot of ways, this model was designed as a national-level Information Sharing and Analysis Centers (ISAC) lite model as a central entity to push and pull related information on cyber threats. This is the fourth report issued by the OIG since AIS's inception, taking place on a biannual cadence with the findings of the previous three available in OIG-22-59, OIG-20-74, and OIG-18-10.

AIS is composed of three different IOC collections, represented as Structured Threat Information eXpression (STIX) objects shared via Trusted Automated eXchange of Indicator Information (TAXII):

1. The Federal collection is available to all departments and agencies that sign the Multilateral Information Sharing Agreement.
2. The Public collection is available to all participants. All non-Federal users must sign the AIS Terms of Use.
3. The Cyber Information Sharing and Collaboration Program (CISCP) collection is available to non-Federal entities who sign the Cyber Information Sharing and Collaboration Agreement.

The AIS capability shares machine-readable STIX objects from various CISA-published products to include its indicator bulletins, activity alerts, and analysis reports.

The OIG's Findings

The OIG's assertion that the cyber threat information sharing vis-a-vi the AIS sharing program hinges on two data points raised primarily in its executive summary:

1. The number of participants using AIS to share cyber threat information has declined to its lowest level since 2017 from 304 in CY 2020 to 135 in CY 2022. Concurrently, sharing of IOCs through AIS declined by 93 percent from CY 2020 to CY 2022.
2. Despite CISA completing upgrades to its AIS 2.0 capability in March 2022 to address information-sharing limitations, overall participation in AIS declined because CISA did not have an outreach strategy to recruit and retain data producers across the intelligence community or private sector.

Yet, as we dig into the body of the report, it is hard to argue with the orders of magnitude drop off in indicators shared after the program changed hands to an unspecified government entity due to requirements in the National Defense Authorization Act. Specifically, in 2021, the outgoing Federal partner shared approximately 9 million indicators through AIS whereas in 2022, after the incoming Federal partner took ownership of the system, this number fell to 16,697, representing a decrease of more than 99 percent.

Was the OIG Wrong?

Technically, using their identified highly scoped criteria to evaluate the effectiveness of AIS, they sufficiently answered the mail. Where the analysis, presentation, and recommendations fall short are in a few areas:

1. Failure to consider AIS as a complementary program that supports multiple other service lines of effort identified in the information sharing budget item rather than a standalone program and the interplay with the future TIES strategy. This feeds a point I made earlier about choosing to evaluate on non-material metrics vs. provide an evaluation based on program impact. I suspect the OIG auditors who are generalists in nature were looking to CISA for guidance on domain-specific cyber metrics they should use, but probably never explicitly asked them or made this a clear option during preliminary draft review, rather expecting them to weigh in organically. 
2. An opaque recommendation was provided, buried at the end of the report, on whether tax payer funding would have been better served elsewhere. This indirect call out would have been better served as an explicit recommendation to have an independent evaluation performed on whether AIS was serving the intended function as promised by the 2015 Act or whether its mission scope required updating. Reading between the lines of CISA's response, it seems this may play into their independent evaluation. Jen Easterly is incredibly bright, has built numerous bridges with private sector entities, and from what I've seen from the outside has made large strides to establish core structures for CISA’s success in achieving its very large mission mandate. I would be surprised if AIS 2.0 wasn't already being considered in the future TIES strategy.
3. An acknowledgement as to where other information sharing programs stood relative to AIS that would provide additional context on where resources may have been allocated otherwise instead of AIS to include the Known Exploited Vulnerabilities (KEV) initiative, Enhanced Cybersecurity Services program, Shields Up/Shields Ready, supporting the White House established Counter Ransomware Initiative via the Joint Ransomware Task Force, working across multiple other US Government organizations to issue state-sponsored threat actor advisories in concert with other counter-cyber policy efforts, and likely many other non-publicly transparent lines of effort CISA was pursuing.

For a deeper understanding of the cyber threat landscape and information sharing measures enacted by the government, check out the FOR578^TM: Cyber Threat Intelligence^TM course. This course is designed to equip you and your organization with the level of tactical, operational, and strategic cyber threat intelligence required to better understand the ever-evolving threat landscape and how to effectively counter those threats. Ready to take the next step? Register today or request a demo to see this course in action!