On July 28, 2024, the Cybersecurity and Infrastructure Security Agency (CISA) announced that they piloted an Artificial Intelligence (AI)-enabled vulnerability program to help detect and remediate vulnerabilities in the U.S. government’s critical networks, systems, and software, as required by Executive Order (EO) 14110.
From late 2023 to early 2024, CISA performed the pilot program to determine whether the current vulnerability software products that use AI (including large language models (LLMs)) can more effectively detect vulnerabilities than those that do not use AI. The scope of the pilot program was limited to more recent AI products that were available for use on or prior to December 31, 2023. CISA also used two scenarios for testing the AI tools: tests within a controlled environment and security assessments of federal partner networks.
CISA provided a report detailing the findings of this pilot program to the White House on July 26, 2024. CISA’s key findings include:
- The best use of AI for vulnerability detection is to supplement and enhance the currently existing tools;
- The amount of time it took for analysts to learn to use the new AI capabilities is “substantial,” and any incremental improvement may be “negligible”; and
- AI tools can be “unpredictable,” leading to difficulties in troubleshooting.
Given the constant improvements of AI tools, CISA stated that they “will continue to monitor the market and test tools to ensure CISA’s vulnerability detection capabilities remain state-of-the-art.”
The testing of this pilot program marks a significant step forward in advancing CISA’s Roadmap for AI, a guide detailing CISA’s AI-related efforts, to better understand and address the risks and benefits of AI usage in U.S. critical infrastructure.
[View source.]
