Creepy Crawlies: The Infrastructure Strain and Residential Proxy Warfare of AI Scrapers
The open-source software ecosystem is experiencing a severe infrastructure crisis as artificial intelligence firms and independent scrapers aggressively harvest training data. A detailed disclosure by Linux systems administrator Konstantin Ryabitsev reveals that git.kernel.org is under constant, heavy load from AI scrapers, which consume more CPU cycles rendering HTML commits than all legitimate access combined—including git clones. Across five geo-distributed nodes, 14 to 16 CPU cores (roughly 20% of total capacity) are permanently dedicated to rendering HTML for scrapers.
The crisis highlights a profound technical irony: while Git is designed to be easily cloned in full, scrapers choose the least efficient method possible. Instead of running a single git clone and walking the commits locally, they render millions of commits as HTML page-by-page to parse them. Because the Linux kernel commit history is guaranteed to be "pure, pre-AI content," it is highly prized to prevent the "digital prion disease" of model collapse. However, because cgit allows arbitrary diffs, patches, and comparisons across thousands of forks, a single repository generates a combinatorial explosion of over a quadrillion valid, crawlable URLs.
To defend the infrastructure, kernel.org deployed Anubis, a proof-of-work (PoW) challenge that forces clients to perform CPU-intensive math before accessing pages. However, scrapers have escalated the conflict by leveraging millions of residential and mobile IP addresses via "proxy SDK monetization" networks (often embedded silently in consumer smart TV apps). These bots make only 4–5 requests per IP before rotating, rendering traditional firewall bans useless. Furthermore, scrapers have begun dedicating their own compute to solve the Anubis challenges, with 33% of the daily 6 million requests now successfully solving the PoW gates and reaching the backend.
Within the systems community, the debate splits between those who view this as a failure of antiquated cgit architecture and those who recognize the sheer impossibility of caching under a combinatorial URL space. While some suggest shifting the rendering work to client-side JavaScript, others note that the fundamental issue is the tragedy of the commons: AI companies are shifting their massive pretraining costs onto the physical infrastructure of the open web.