1. The debate over AI's capability to replace human penetration testers
Discussion participants are divided on whether AI agents like ARTEMIS are truly ready to replace human penetration testers, with some pointing out significant limitations while others see rapid, inevitable progress. Skeptics highlight the high rate of false positives and the AI's failure to spot obvious vulnerabilities as evidence that it is not yet a complete solution. Proponents, however, argue that AI is already superior for routine tasks and will quickly surpass humans in more complex areas.
- tptacek: "I would expect over the medium term agent platforms to trounce un-augmented human testing teams in basically all the 'routinized' pentesting tasks --- network, web, mobile, source code reviews. There are too many aspects of the work that are just perfect fits for agent loops."
- Sytten: "The app automated pentest scanners find the bottom 10-20% of vulns, no real pentester would consider them great. Agents might get us to 40%-50% range, what they are really good at is finding 'signals' that the human should investigate."
- mens_rea: "Deeply flawed paper for several reasons: ... Exaggerated claims (saying A1 beat 50% of testers, yet only 4/10 testers found LESS vulns than A1, and A1 had a nearly 50% false positive rate)."
2. The potential economic impact on the cybersecurity services market
A major theme is the financial pressure AI pentesting will place on the cybersecurity consulting industry, particularly on human billable hours. Even skeptics of the technology's current readiness acknowledge that executive interest in lower-cost AI solutions poses a real threat to traditional service models. Participants discuss how rates have stagnated for years and how AI could accelerate this trend, especially for lower-end, repetitive work.
- falloutx: "An Exec is gonna read this and start salvating at the idea of replacing security teams."
- tptacek: "human-in-the-loop AI-mediated pentesting will absolutely slaughter billable hours for offensive security talent."
- big_youth: "Late-period NCC doesn't look great. But I've been a buyer of these services for the past 5 years... and rates have not gone down; I was shocked at how much we ended up spending compared to what we would have billed out on comparable projects at Matasano... but the high end of the market definitely has not been slaughtered, and I definitely think that is coming."
3. The evolving role of humans in an AI-augmented workflow
There is broad consensus that the role of the human penetration tester will shift, not disappear. Instead of performing repetitive manual checks, humans will act as orchestrators, validators, and interpreters for AI agents. This new model focuses human expertise on higher-level strategy, investigating AI-generated signals, and handling the complex edge cases where current AI struggles.
- nullcathedral: "The productivity gains from LLMs are real, but not in the 'replace humans' direction. Where they shine is the interpretive grunt work... They're straight up a playing field leveler."
- KurSix: "The key driver here isn't even model intelligence, but horizontal scaling. A human pentester is constrained by time and attention, whereas an agent can spin up 1,000 parallel sub-agents... Even if the success rate of a single agent attempt is lower than a human's, the sheer volume of attempts more than compensates for it."
- tptacek: "A pentesting agent directly tests running systems. It's a (much) smarter version of Burp Scanner... Remember, the competition here is against human penetration testers. Humans are extremely lossy testing agents!"