First comparative benchmark of AI agents for API bug detection shows strong performance on simple checks, but major gaps on cross-field and business-logic failures
India | KushoAI today released the first comparative benchmark study of how leading AI coding and testing agents perform at finding bugs in live APIs. While AI tools generate plausible tests quickly, most struggle to detect bugs emerging from field relationships, operation semantics, and business-logic dependencies.
The report evaluated seven AI systems across three groups: general-purpose LLMs, coding agents, and KushoAI’s API testing agent. Each received only a JSON schema and a sample payload for 20 live API scenarios, each containing 97 known functional bugs across three difficulty tiers.
The central finding is a sharp drop in performance as bugs get more complex. Most systems catch simple schema violations: missing fields, wrong types, and null values. Performance falls when detection requires semantic reasoning or understanding how valid fields combine into an invalid business state. On the hardest tier, the strongest coding-agent workflow detected 53%, the strongest general-purpose LLM detected 34%, and KushoAI detected 76%, ranking first across every complexity tier.
“AI can generate tests. That is no longer the hard question,” said Abhishek Saikia, Co-founder and CEO of KushoAI. “The harder question is whether those tests reach the failure modes that matter. Simple schema-level testing is increasingly table stakes. The real gap appears when API testing requires reasoning across fields, states, and business rules.”
This report follows KushoAI’s earlier launch of APIEval-20, the industry’s first open benchmark for evaluating AI agents on API bug detection from schema and payload alone. This study reveals how general-purpose LLMs, coding agents, and purpose-built API testing agents actually perform.
Better prompting helps but does not close the gap. Prompt chaining improved field-level coverage but did not produce the cross-field tests needed to catch business-logic failures. KushoAI showed the lowest run-to-run variance, critical for teams integrating generated tests into CI pipelines.
The findings build on KushoAI’s analysis of 1.4 million test executions across 2,616 organizations. The report positions APIEval-20 as an emerging standard, similar to the role HumanEval and SWE-bench play in software engineering research.
