AI Fails to Adequately Patch Software Flaws in 74% of Attempts, Study Reveals

Recent findings suggest that integrating artificial intelligence into cybersecurity defenses may require a more cautious approach than currently anticipated. Research indicates that artificial intelligence and large language models (LLMs) are not yet prepared to effectively address and resolve security vulnerabilities in software.

Off-By-1-Labs, a new security research division established by 1Password, recently published a study examining the performance of LLMs when tasked with generating solutions for complex software vulnerabilities. The researchers initially hypothesized that advanced AI models would achieve a relatively high patch success rate, potentially around 67%, particularly given their training on open-source code and publicly disclosed information. However, the study's outcomes were described as significantly lower and more inconsistent than predicted.

The research team selected six recently disclosed vulnerabilities in open-source software for their evaluation. These particular vulnerabilities were chosen because they were unlikely to have been included in the AI models' training datasets, ensuring a fresh test of the LLMs' capabilities. The study, titled "Frontier Models' Vulnerability Patches are Often F.L.A.W.E.D. Fix-Like Artifacts With Embedded Defects: Common failure modes of LLM-generated security patches," aimed to provide a broad overview of contemporary LLM capabilities rather than a direct comparative analysis between different models.

For the study, Claude and an LLM utilizing OpenAI's Codex coding agent were instructed to create patches for each identified vulnerability. Across various environmental settings and with nine distinct prompts per bug, the LLMs collectively produced 6,080 patch attempts, with approximately 3,040 attempts attributed to each model type across different Common Vulnerabilities and Exposures (CVEs).

A primary observation from the research was that regardless of the environmental variables or prompting techniques, the LLMs frequently generated what the study termed "Fix-Like Artifacts with Embedded Defects" (FLAWED) when attempting to create patches. This phenomenon inspired the name of the research project itself.

These AI-generated "FLAWED" patches often appear to resolve the issue on the surface. However, upon closer inspection, they frequently fail to fully address the vulnerability, incorporate 'fragile' security mechanisms, or, in some instances, introduce new bugs into the software. In more severe cases, these inadequate patches could even alter the intended behavior of an application.

To facilitate further investigation by the research community, 1Password has made its tooling, also named FLAWED, available on GitHub. The researchers noted that while LLMs demonstrate proficiency in identifying a wide array of vulnerabilities, their current effectiveness in patching them remains limited to a narrow subset. Nevertheless, they indicated that opportunities for additional research exist, which could potentially lead to the development of more consistent and robust AI-generated patches.

Keith Hoodlet, head of Off-by-1 Labs, commented that human cybersecurity professionals and AI tools should prioritize vulnerability discovery and triage. This approach, he suggested, could assist defenders in identifying the most critical bugs within their codebase. Hoodlet emphasized that human oversight is still crucial throughout the patching process, enabling companies to make informed decisions regarding which issues to patch, when, and to understand the associated business risks.

Hoodlet concluded that the newly released FLAWED tooling will help organizations evaluate where AI-generated patches might offer better or worse outcomes. This, he stated, will clarify where the expertise of human professionals is most indispensable.