Cloudflare outage caused by Bot Management bug

Cloudflare’s CEO Matthew Prince published a blog post explaining the company’s “worst outage since 2019,” which briefly knocked many sites offline — including X, ChatGPT, and Downdetector. The outage wasn’t caused by DNS or a cyberattack; Cloudflare traces the failure to its Bot Management system, the CDN feature that helps identify and control automated crawlers (often used to scrape data for training generative AI).

Prince says a change in the ClickHouse query behavior that generates the Bot Management model’s frequently updated configuration file produced a large number of duplicate “feature” rows. That made the configuration file grow rapidly beyond preset memory limits and crashed the core proxy system that processes traffic for anything depending on the bots module.

Because many customers use Cloudflare’s generated bot scores in their blocking rules, the failure produced false positives that cut off legitimate traffic; customers who didn’t rely on the generated scores stayed online. Cloudflare emphasized the problem was a DB/permissions/query issue, not its recent generative-AI “AI Labyrinth” mitigation tech.

Cloudflare listed four immediate plans to prevent recurrence: harden ingestion of generated configs like user input, add more global kill switches, stop core dumps or error reports from overwhelming resources, and review failure modes across core proxy modules.