Doyensec finds AI pentest agents agree on only 7 of 73 flaws

A study from the security firm Doyensec offers some evidence about how well AI pentesting agents actually perform. According to a DEV Community report published on October 3, 2026, the research was carried out in May of this year and found that, across 73 vulnerabilities, two AI pentesting agents reached the same judgment on only 7 of them. That comparison is a measure of consistency between the two agents — whether two separate runs over the same set of vulnerabilities arrive at the same conclusion.

The report frames pentest reports as something companies rarely commission on their own initiative. Sell software to anyone with a security team, and eventually someone will ask for one: a SOC 2 auditor, a procurement questionnaire, an insurance renewal. The usual answer is a consultancy that tests the app once, writes a PDF, and is out of date by the next deploy. AI pentesting agents promise to run that test continuously, and according to the report, 2026 is the year when some evidence about how these tools actually behave is finally starting to appear.

The figure of 7 matching judgments out of 73 vulnerabilities is presented as an indication that reliability gaps remain for this class of tooling. The available material does not name the two agents, identify the models behind them, or describe the testing methodology, and it does not offer further details from the study itself.