IdeaWell

Project ideas from Hacker News discussions.

I posted to HN, and then my website was flooded with AI bot traffic

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

1. Combating abusive bot traffic and scraping
Many commenters described relentless automated requests from cloud providers, residential proxies, and AI‑driven scrapers, and shared mitigation tactics such as IP‑level blocking, Cloudflare edge rules, and CAPTCHA/Turnstile defenses.
- “There is so much spam, abuse, and aggressive scraping that I’ve had to block: … AWS, Google Cloud, Azure, Alibaba Cloud, Oracle … I banned Alibaba’s Cloud two years ago and it still hits one of my domains 1 million times per day.” — switz
- “Application‑level banning seems very inefficient … Just drop any incoming packets from these domains/IPs.” — Sharlin
- “They are banned at the edge in Cloudflare (now), they never hit my application.” — switz
- “I did this for alyph.ai, but was recently flooded with so many browser‑using, Turnstile‑defeating bots … I had to completely turn off sign ups.” — rrr_oh_man

2. Using LLMs to revive legacy tools (e.g., an Emacs web browser)
Several participants praised the effort to make old‑school browsers usable on today’s web by leveraging large language models.
- “It’s nice to see someone making oldschool web browsers work properly on the modern web.” — ben_w
- “:) ive been dreaming of a usable emacs browser for so long” — sammy0910

3. Site architecture choices: single‑page Org‑mode export
Discussion turned to the simplicity of a single‑page site generated via Org‑mode, its trade‑offs, and possible improvements.
- “What confused me most about this is that the entire site is, apparently, a single page … containing all posts.” — oneeyedpigeon
- “it's org-html-export-to-html babyyy lmao” — sammy0910
- “You can also link between Org documents and org-publish will export the documents with working web links …” — kqr

🚀 Project Ideas

Generating project ideas…

Emacs LLM Web Enhancer

Summary

  • Provides LLM-powered readability, ad removal, and natural‑language interaction for Emacs’ built‑in web browser (eww) to make it usable on modern sites.
  • Core value: Brings a usable, distraction‑free web experience inside Emacs without leaving the editor.

Details

Key Value
Target Audience Emacs power users, developers, writers who prefer keyboard‑driven workflows
Core Feature LLM‑driven page simplification (summarization, ad stripping) + voice/text commands to fill forms or navigate via natural language
Tech Stack Emacs Lisp, Python LLM API (Ollama/HuggingFace), tree‑sitter for HTML parsing, optional WebKit via eww
Difficulty Medium
Monetization Hobby

Notes

  • Commenters expressed long‑standing desire: “ive been dreaming of a usable emacs browser for so long” (sammy0910) and “nice to see someone making oldschool web browsers work properly on the modern web” (ben_w).
  • Could spark discussion on extending Emacs with AI while preserving its lightweight ethos.

SlowBot Tarpit Proxy

Summary

  • A lightweight reverse proxy that detects likely bots and serves responses extremely slowly, consuming attacker resources and deterring abuse.
  • Core value: Reduces unwanted traffic without blocking legitimate visitors, using adaptive throttling instead of outright bans.

Details

Key Value
Target Audience Indie site owners, SaaS founders, anyone hit by scraping, AI crawlers, or credential‑stuffing bots
Core Feature Heuristic‑based bot detection (UA, request rate, missing JS) → dynamic tarpit mode (slow drip‑feed responses) with whitelist/fallback
Tech Stack Go or Rust for high‑performance networking, Redis for state tracking, Prometheus/Grafana for metrics, optional eBPF for kernel‑level bypass
Difficulty Medium‑High
Monetization Revenue-ready: SaaS pricing – Free tier (<100k requests/mo), $10/mo for up to 1M requests, custom enterprise plans

Notes

  • Demonstrated effectiveness: “I built my own L7 proxy that utterly fucks the bots up… trap about 3000 concurrent crawlers” (robotmay).
  • Addresses frustration with cloud‑provider abuse: “blocking cloud providers … only way” (switz).
  • Enables practical debate on trade‑offs between security and user experience.

Cloud Abuse IP Reputation Service

Summary

  • Aggregates abuse reports from AbuseIPDB, spamhaus, cloud provider logs, and community feeds into a continuously updated IP reputation score.
  • Provides a DNSBL and simple API for firewalls, fail2ban, and cloud‑flare integrations to auto‑block abusive IPs.
  • Core value: Gives self‑hosters an easy, up‑to‑date blocklist without maintaining custom rules.

Details

Key Value
Target Audience Self‑hosted service operators, sysadmins, small businesses frustrated by cloud‑IP bot traffic
Core Feature Real‑time IP reputation scoring, automatic update pipelines, ready‑to‑use blocklist (DNSBL) + API + integration scripts (iptables, nftables, Cloudflare firewall, fail2ban)
Tech Stack Python backend, PostgreSQL, Kafka/Kinesis for streaming logs, Go agents for log pulling, Docker/Kubernetes for deployment
Difficulty High
Monetization Revenue-ready: Subscription – $5/mo basic feed, $20/mo premium with API & SLA, volume‑discount tiers for larger ISPs

Notes

  • Highlights pain: “I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day” (switz); “You can do that with any system made in the past 15, maybe 20 years.” (esseph).
  • Offers a concrete tool for the community to combat the rising tide of cloud‑based scraping and abuse, likely to generate lively HN discussion.

Read Later