IdeaWell

Project ideas from Hacker News discussions.

New information extracted from Snowden PDFs through metadata version analysis

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

1. PDF Incremental Updates Retain Revision History

PDFs append new object generations without deleting old ones, enabling recovery of prior versions.
"a PDF is mostly just a bunch of objects... update these objects to overwrite them by appending a new 'generation' of an object" (aidos).
"These PDFs apparently used the 'incremental update' feature of PDF, where edits... are merely appended" (layer8).

2. Tools for PDF Inspection and Recovery

Users recommend tools like pdfresurrect, mutool, and qpdf to extract revisions and analyze structure.
"The PDF format allows for previous changes to be retained... This tool extracts all previous revisions" (flotzam, quoting pdfresurrect).
"I recommend mutool for decompressing the PDF... mutool clean -d in.pdf out.pdf" (aidos).
"There needs to be better tooling... using qpdf to export QDF" (alhirzel).

3. Redaction Failures and Secure Sharing Methods

Journalists' poor redactions leaked data; suggestions include rasterizing or print/scan to strip metadata, avoiding printer tracking dots.
"the journalists did the redactions. The metadata timestamps... show that the versions were created three weeks before" (libroot).
"The 'print and scan physical papers back to a PDF of images' technique... is looking better" (password4321).
"Note that all (edit: color-/ink-) printers have 'invisible... yellow dotcodes'" (cookiengineer).

🚀 Project Ideas

PDF Ghosthunter

Summary

  • A specialized GUI-based forensic tool for inspecting the internal object structure and revision history of PDF files.
  • It solves the "hidden data" problem by visualizing orphaned objects, previous generations of edited blocks, and incremental updates (those "%%EOF" markers).
  • Core value proposition: Providing non-technical users (journalists, legal teams) a "What You See Is NOT What You Get" view to prevent embarrassing metadata leaks.

Details

Key Value
Target Audience Journalists, OSINT researchers, and legal professionals.
Core Feature Tree-view of PDF objects with side-by-side comparison of document revisions.
Tech Stack Python, PyQt/Electron, qpdf or mutool backend.
Difficulty Medium
Monetization Revenue-ready: SaaS for teams or a "Pro" desktop license.

Notes

  • Directly addresses the request for a GUI: "It is just begging for a GUI to wrap around it... [qpdf] is primarily about editing pdfs rather than inspecting."
  • Prevents the "Snowden/Epstein" style leaks mentioned where redactions were merely appended rather than scrubbed.

ScrubConnect

Summary

  • A security-focused "PDF Sanitizer" that goes beyond metadata removal by using a virtual "Print-Raster-OCR" pipeline.
  • It solves the problem of subtle tracking (like modulated character spacing or invisible yellow dots) by flattening the document into a high-resolution image and then reconstructing a clean, text-searchable PDF using OCR.
  • Core value proposition: Guaranteed "analog hole" protection without needing physical paper or a printer.

Details

Key Value
Target Audience Whistleblowers, privacy advocates, and government contractors.
Core Feature Automated Raster-to-OCR pipeline with noise/fuzzing filters to defeat steganography.
Tech Stack Tesseract OCR, Ghostscript, Python/Docker.
Difficulty Medium
Monetization Hobby (Open Source) or "Pay-per-scan" API for secure document handling.

Notes

  • Addresses the debate between "Print to PDF" vs "Rasterizing": "I'd worry print to pdf might be ineffective. I think rasterizing is the way to go."
  • Solves the Section 508 compliance pain point mentioned: "Then use OCR to convert it back from raster for Section 508 compliance."

YellowDot Auditor

Summary

  • A specialized image analysis tool and library to detect and decode Printer Machine Identification Codes (MIC) or "yellow dots" in scanned documents.
  • It solves the uncertainty of whether a printer is "snitching" on the user's IP address or serial number by highlighting these microscopic marks using blue-channel isolation.
  • Core value proposition: Transparency for the "2D printing world" which currently lacks the open-source auditing tools available in 3D printing.

Details

Key Value
Target Audience Security researchers and privacy-conscious office administrators.
Core Feature Blue-light filter simulation and pattern matching to decode MIC serial numbers.
Tech Stack OpenCV, Python, WebGL for browser-based filtering.
Difficulty High
Monetization Hobby

Notes

  • Responds to the discussion about tracking: "Note that all color printers have 'invisible to the human eye' yellow dotcodes... It's mindboggling how much... this is completely lacking in the 2d printing world."
  • Provides a software alternative to the "UV flashlight" method mentioned by users.

Read Later