Show HN: Training a model to identify AI web content from structure alone

📝 Discussion Summary (Click to expand)

Prevalent themes in the discussion

Reliance on LLMs vs. deterministic approaches – Commenters worry that the tool leans on uncertain, prompt‑based LLM outputs instead of solid, reproducible code.

“I'd rather see this sort of thing done deterministically with actual code vs lossy human english prompts and a dependency on token spend to a single third party …” — asdff
Reproducibility and long‑term viability – The dependence on external models that may change or be withdrawn raises concerns about being unable to replicate results or adapt training data later.

“…who will probably pull the underlying model used in what a few short years probably) to replicate the results or try and use different training data.” — asdff
Project perceived as producing low‑value or polluting output – The effort is likened to AI trying to improve its own waste, suggesting it merely creates more noise or “pollution.”

“Is the goal of this to help AI pick corn kernels out of it's own shit for the purposes of slightly raising the bar on how sloppy the slop is? … This feels like a factory built specifically to manufacture pollution.” — pooploop64

🚀 Project Ideas

Generating project ideas…

Provides a code‑only library that automatically computes statistical thresholds (IQR, Z‑score, percentile‑based) for data cleaning and feature engineering without relying on LLM prompts or third‑party model calls.
Core value: reproducible, zero‑cost preprocessing logic that can be version‑controlled and audited.

Key	Value
Target Audience	ML engineers, data scientists building pipelines
Core Feature	Automated threshold calculation with configurable rules; outputs ready‑to‑use Python functions or config files
Tech Stack	Python, NumPy, Pandas, Click (CLI), Pytest (testing)
Difficulty	Medium
Monetization	Hobby

asdff noted: “I’d rather see this sort of thing done deterministically with actual code vs lossy human english prompts.” DDT directly addresses that desire for deterministic, prompt‑free thresholding.
Enables discussion on HN about reducing LLM dependency in ML workflows and improving pipeline reproducibility.

Scans raw text corpora to estimate the proportion of LLM‑generated content, flagging likely synthetic passages that could cause model collapse or data pollution.
Core value: protects training data quality by giving teams a measurable way to filter out AI‑generated “slop” before it degrades model performance.

Key	Value
Target Audience	Researchers, data curators, companies training LLMs
Core Feature	Ensemble scorer (perplexity, token‑distribution, lightweight classifier) producing AI‑likelihood scores and optional filtering reports
Tech Stack	Python, HuggingFace Transformers, FastAPI (optional API), Docker for deployment
Difficulty	High
Monetization	Revenue-ready: usage‑based pricing per million tokens scanned

pooploop64 warned about “a factory built specifically to manufacture pollution”; AIGD lets teams detect and remove that pollution before it harms models.
Sparks practical utility for data‑quality audits and can fuel HN debates on synthetic data risks and mitigation strategies.

A lightweight domain‑specific language (YAML‑like) for declaring data‑processing steps (filter, impute, encode, split) that compiles to deterministic, testable Python code, eliminating the need for LLM‑guided prompt engineering.
Core value: turn high‑level specs into reproducible pipelines that live in version control and require no external token spend.

Key	Value
Target Audience	Data engineers, ML ops teams
Core Feature	DSL authoring → compile‑time generation of Python functions + auto‑generated unit tests
Tech Stack	Python, Lark (parser), Jinja2 (code templating), GitHub Actions (CI)
Difficulty	Medium
Monetization	Hobby

asdff’s call for “deterministic … actual code” is satisfied by DetPipe’s code‑first approach to pipeline definition.
Offers a concrete alternative to LLM‑driven workflow tooling, likely to generate discussion on HN about trustworthy, auditable ML infrastructure.