IdeaWell

Project ideas from Hacker News discussions.

GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Here are the 5 most prevalent themes from the Hacker News discussion:

1. Incentive Structures Drive Academic Misconduct

The consensus is that the "publish or perish" culture and flawed metrics (like the h-index and citation counts) prioritize quantity over quality. This environment incentivizes shortcuts, making the use of LLMs for fraudulent work a predictable outcome rather than an isolated incident.

"When the stakes are so high and output is so valued, and when reproducibility isn't required, it disincentivizes thorough work. The system is set up in a way that is making it fail." — freedomben

"Fundamentally... you have to measure productivity somehow... That turns out to be very hard to do." — StableAlkyne

2. Reproducibility is a Systemic Challenge

Participants argue that the reproducibility crisis stems from a lack of incentives for replication studies. Reproducing work does not advance careers or secure grants as effectively as publishing novel findings, creating a structural barrier to verifying scientific claims.

"The challenge is there really isn't a good way to incentivize that work." — StableAlkyne

"The prestige and livelihood is directly tied to discovery rather than reproducibility." — StableAlkyne

3. Peer Review is Overburdened and Lacks Verification Tools

Many commenters defended the peer reviewers, arguing that checking every citation is unrealistic given the volume of submissions to top conferences. The lack of automated tools to verify references was identified as a major gap in the review process.

"As one who reviews 20+ papers per year, we don't have time to verify each reference." — emil-lp

"Academic venues don't have enough reviewers. This problem isn't new, and as publication volumes increase, it's getting sharply worse." — gcr

4. The Severity of Hallucinated Citations is Debated

While some view fabricated references as an indication of total fraud, others argue that citation errors do not necessarily invalidate the scientific content of a paper. The discussion highlighted the difficulty in distinguishing between malicious fraud and negligence, especially when authors rely on AI for formatting.

"Even if 1.1% of the papers have one or more incorrect references... the content of the papers themselves are not necessarily invalidated." — NeurIPS board (via gcr)

"Fabrications carry intent to decieve. I don't think hallucinations necessarily do. If anything, they're a matter of negligence, not deception." — gcr

5. Severe Penalties and Systemic Reform are Needed

A significant portion of the discussion called for harsher consequences for academic fraud, ranging from career bans to criminal charges for misusing public funds. However, others countered that the focus should be on fixing the incentive structures rather than relying solely on punishment.

"If I steal hundreds of thousands of dollars... and produce fake output... it's no different than stealing a car." — Proziam

"The harsher the punishment, the more due process required... we can all see what's going on... but it is possible to make an honest mistake with your BibTeX." — currymj

🚀 Project Ideas

Reproducibility Verification Tracker

Summary

  • Solves the core problem of unreproducible research by creating a public ledger for every paper's replication attempts, successes, and failures.
  • Core value proposition: Shifts academic incentives from "novelty at all costs" to "verified reliability," enabling researchers to earn credit for replication work and allowing readers to instantly see a paper's validation status.

Details

Key Value
Target Audience Academic researchers, PhD students, research labs, journal editors, and grant funding organizations.
Core Feature A web platform that links to existing paper databases (like Semantic Scholar) and allows users to register replication attempts. Citations are bi-directional: original papers list their replication statuses, and replication papers are explicitly linked to the original.
Tech Stack Python (Django/FastAPI), PostgreSQL, React/Next.js, integration with Semantic Scholar API, ORCID for author verification.
Difficulty Medium
Monetization Revenue-ready: Institutional subscriptions for universities and research labs; premium API access for data aggregators and publishers.

Notes

  • HN commenters would love it: As cogman10 stated, "I'd love to see future reporting that instead of saying 'Research finds amazing chemical x which does y' you see 'Researcher reproduces amazing results for chemical x which does y. First discovered by z'." This tool makes that cultural shift practical.
  • Potential for discussion or practical utility: It directly addresses the "replication crisis" mentioned by multiple users (StableAlkyne, reliabilityguy) and aligns with the call for new incentives (soiltype, godelski).

Citation Sanity Check

Summary

  • Prevents hallucinated citations (like those discussed in the NeurIPS example) by verifying references before submission to journals or conferences.
  • Core value proposition: An automated pre-screening tool for authors and reviewers that validates every bibliography entry against global publication databases, flagging missing, misattributed, or AI-generated fake references immediately.

Details

Key Value
Target Audience Individual researchers, PhD students, conference organizers, and journal editorial boards.
Core Feature API-driven service where users upload a PDF or BibTeX file. The system cross-references every entry with DOI, ISBN, and title match databases, providing a confidence score and specific flags for "Hallucinated," "Minor Error," or "Verified."
Tech Stack Python (for PDF parsing and string matching), Integration with Crossref, PubMed, and Google Scholar APIs, FastAPI backend.
Difficulty Low
Monetization Revenue-ready: Freemium model (limited checks for free) with paid plans for heavy users or institutional integrations (e.g., embedded in submission portals like OpenReview).

Notes

  • HN commenters would love it: It addresses j2kun's observation that even top-tier papers contain citation errors (missing authors, wrong venues). It solves the "needle in a haystack" problem reviewers face.
  • Potential for discussion or practical utility: It offers a concrete technical fix to the "sloppy bibliography" issue mentioned by davidguetta and the general negligence in peer review highlighted by alansaber.

ScholarCheck

Summary

  • A browser extension and journal plugin that automatically highlights citations in academic papers that have been validated by independent reproduction or flagged for invalidation.
  • Core value proposition: It turns the static PDF reading experience into an interactive reliability audit, allowing readers to instantly see the "citation web" and whether a cited paper stands on solid ground or has been debunked.

Details

Key Value
Target Audience Reviewers, researchers reading literature, and science journalists.
Core Feature Overlay on PDF readers (like the Google Scholar PDF reader mentioned by gcr) that pulls data from the "Reproducibility Verification Tracker" (Project #1) to color-code citations (e.g., Green = Reproduced, Red = Invalidated, Grey = Unvalidated).
Tech Stack Browser Extension (JavaScript/TypeScript), Chrome/Firefox APIs, REST API integration with Project #1.
Difficulty Low
Monetization Hobby: Open source project, maintained by a non-profit or academic consortium.

Notes

  • HN commenters would love it: gcr mentioned Semantic Scholar doing great work on citation visibility; this tool extends that by adding a "trust layer." It directly implements the desire of godzillabrennus to push citations to include validations.
  • Potential for discussion or practical utility: It combats the issue rtkwe raised about citing invalidated research and makes the "failure to reproduce" visible to anyone reading the paper, not just those actively searching for it.

LabArchive AI

Summary

  • An intelligent digital lab notebook that enforces open science practices by requiring structured data logging and automated reproducibility reports for ML/AI research.
  • Core value proposition: Solves the "IKEA furniture" problem (StableAlkyne) where papers lack weights or algorithms by generating a standardized, executable "reproducibility packet" alongside every experiment run, ensuring that negative results and code are preserved.

Details

Key Value
Target Audience ML/AI Researchers, PhD students in computer science, and academic labs.
Core Feature A tool that tracks code, hyperparameters, datasets, and random seeds during experimentation. It auto-generates a "Reproducibility Report" (a standardized document) required for paper submission to venues like NeurIPS.
Tech Stack Python (PyTorch/TensorFlow integrations), Docker (for environment snapshots), Git, Web Dashboard (React).
Difficulty Medium
Monetization Revenue-ready: SaaS subscription for research groups; "Reproducibility Certified" badge for venues paying for integration.

Notes

  • HN commenters would love it: It addresses the complaint by f311a and StableAlkyne that many papers fail to provide reproducible code or weights. It shifts the burden from the reviewer to the author's workflow.
  • Potential for discussion or practical utility: It institutionalizes the "PoC or GTFO" ethos mentioned by f311a by making the proof of concept (the reproducible packet) a mandatory output of the research process.

ReproCoin

Summary

  • A decentralized incentive platform where researchers earn reputation points (or micro-grants) for successfully reproducing or validating existing research papers.
  • Core value proposition: Addresses the economic and career disincentive to perform replication studies. By gamifying and funding replication, it creates a marketplace for verification rather than just novelty.

Details

Key Value
Target Audience Post-docs, independent researchers, and labs with spare compute/cycles.
Core Feature A bounty system where original authors or institutions can put up bounties for reproducibility. Researchers claim bounties by submitting verification code and results. Results are peer-validated and recorded on a public ledger.
Tech Stack Smart Contracts (Ethereum/Solidity or similar) for bounties, IPFS for storing data/code, Web3 wallet integration for researchers.
Difficulty High
Monetization Revenue-ready: Transaction fees on bounty payouts; premium services for institutions setting up large-scale verification programs.

Notes

  • HN commenters would love it: This directly tackles the "incentive" problem discussed extensively by StableAlkyne, geokon, and sroussey. It monetizes the act of reproducing results, making it a viable career move rather than a dead end.
  • Potential for discussion or practical utility: It creates a direct economic counter-force to the "publish or perish" culture that prioritizes novelty over reliability, as lamented by freedomben.

CiteGuard

Summary

  • A submission management system for conferences and journals that requires strict citation verification via DOI/ISBN lookup before a paper can be submitted to the review queue.
  • Core value proposition: Prevents "hallucinated citations" at the source by technically enforcing a "zero-tolerance" policy for unverifiable references during the submission process, removing the burden from reviewers.

Details

Key Value
Target Audience Conference organizers (e.g., NeurIPS, CVPR), journal editors, and university submission systems.
Core Feature A plugin for existing submission systems (like OpenReview) that blocks the "Submit" button until all citations pass an automated verification API check, providing detailed error logs for authors to fix.
Tech Stack Python/Flask, Integration with Crossref API, Google Scholar API, and existing submission platform APIs.
Difficulty Low/Medium
Monetization Revenue-ready: Licensing the plugin to conference organizers; consulting services for implementing "integrity-first" submission workflows.

Notes

  • HN commenters would love it: It addresses the shock expressed by dtartarotti and emil-lp that hallucinations passed through peer review. It automates the "basic verification" that colechristensen notes was missing.
  • Potential for discussion or practical utility: It provides a concrete technical barrier against the "slop" mentioned by godelski, ensuring that only verifiable references enter the academic record.

Read Later