Devin (Cognition AI)

Devin is the most aggressively marketed product in the AI coding category and the one with the largest documented gap between demo and real-world performance. The March 2024 launch claimed Devin achieved 13.86% on SWE-bench — a figure that independent researchers subsequently found was measured on a non-standard benchmark subset with human assistance not disclosed in the presentation. The claim was not corrected in subsequent marketing.