- Bot overload via residential proxies
- initramfs: "I've been noticing page views in the past several months with a much wider span of origin on my Blogger stats. Before I would get a few from several countries, but now I am getting views from tiny countries and obscure or outdated browsers and operating systems, which leads me to think scrapers could be using VPN services in various countries along with header anonymizers that mask the device that they are using."
-
Demiurge: "Over the last few years this counter went from 100-200 users online to thousands. … it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button."
-
Proof‑of‑work mitigations (Anubis) and their limits
- semiquaver: "This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices."
-
tptacek: "It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual users."
-
Technical burden of serving git repositories
- teo_zero: "But there are 'cubic bazillions' of possible URLs that are being requested. Even if they boil down to 'only' some millions actual commits, their rendered HTMLs are all different."
-
rcxdude: "The issue is there's far more pages than could reasonably be stored: cgit is an old-school server-side renderer and there's a combinatorial explosion of possible pages it could render. The article calculates there's about a quadrillion different pages that could be 'crawled'"
-
Economic/legal remedies (charging bots, lawsuits)
- Demiurge: "The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large."
- mmooss: "Indeed. The solution isn't technical but legal. It's clearly abusive of - really stealing - other people's resources; there's no question about it."
Creepy Crawlies
📝 Discussion Summary (Click to expand)
🚀 Project Ideas
Generating project ideas…
Gathering the best ideas from the HN discussion…