IdeaWell

Project ideas from Hacker News discussions.

I accidentally turned LLM memory into program analysis

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Theme 1 – Blending classic symbolic AI with LLMs
Leveraging heuristic search, Datalog, and other logic‑programming techniques to steer LLM‑driven agents.

linguae: “leveraging techniques from “old‑school” AI such as heuristic search to guide agents when it comes to planning. The use of Datalog in this article resonates with me, since logic programming was a major part of old‑fashioned symbolic AI.”

Theme 2 – Structured, queryable knowledge graphs as LLM grounding
Extracting facts into Datalog/CodeQL‑style or entity‑relationship graphs so models can pose and answer precise questions.

tptacek: “It's an agent system that basically embeds the core idea of CodeQL (Datalog extraction from codebases) and then allows a model to pose questions and answer them.”
keeda: “…used an LLM to decompose articles into a set of statements which were used to construct an entity‑relationship graph of facts and events. It then queried that using conventional graph query methods, much like DataLog / Lemmalog is doing here.”

Theme 3 – Practical utility for debugging, investigation, and reducing hallucinations
Applying the hybrid system to real‑world tasks like tracing hardware faults, refining hypotheses, and persisting verified facts to avoid LLM forgetfulness.

trinsic2: “I ran into the very same problem of the LLM forgetting that we ruled out a conclusion … I had to keep reminding we ruled out that conclusion prior … eventually i got the sources to a point where they were good enough hypothesis …”

🚀 Project Ideas

LogiAgent: LLM Agent with Datalog-backed Memory and Hypothesis Tracking

Summary

  • Solves the problem of LLMs forgetting previously ruled‑out conclusions during debugging conversations by persisting hypotheses in a Datalog knowledge base.
  • Provides a grounded reasoning layer that lets agents query symbolic facts and maintain a trace of excluded possibilities.

Details

Key Value
Target Audience Developers and SREs using local LLMs for root‑cause analysis, debugging, or agentic coding
Core Feature Integrates a local LLM with a Datalog KB (e.g., Soufflé) to store facts, hypotheses, and exclusion flags; offers API to mark a hypothesis as ruled out and query it later
Tech Stack Python, FastAPI or LlamaIndex for LLM wrapper, Soufflé Datalog engine, SQLite for persistence, optional Docker
Difficulty Medium
Monetization Revenue-ready: Subscription tier for hosted API + enterprise support

Notes

  • HN users expressed frustration: "I had to keep reminding we ruled out that conclusion prior..." (trinsic2) and "the LLM forgetting that we ruled out a conclusion".
  • Combining LLMs with old‑school symbolic AI (Datalog) could spark discussion on hybrid agents and improve reliability in iterative debugging.

TraceGraph: Log-to-Graph Extraction for Incident Investigation

Summary

  • Turns raw system logs, bug reports, and forum threads into an entity‑relationship/timeline graph that can be queried with natural language, addressing the need to investigate obscure hardware failures that appear only under specific conditions.
  • Enables analysts to ask “what changed before the failure?” and get precise, source‑backed answers.

Details

Key Value
Target Audience SREs, hardware engineers, and support teams dealing with intermittent failures and log‑based root cause analysis
Core Feature Pipeline that parses logs/text, extracts timestamped events, entities (peripherals, modules, error codes), and relationships; stores them in a Datalog/Neo4j graph; provides a chat‑like interface that translates user questions into graph queries
Tech Stack Python (spaCy/Stanza for NLP), Apache Arrow or Pandas for processing, Neo4j or Datalog (Soufflé) for storage, React frontend, FastAPI backend
Difficulty High
Monetization Revenue-ready: Per‑seat SaaS plan with free tier for small teams

Notes

  • Commenter trinsic2 described needing to "investigate obscure hardware failures in the logs that I couldn't confirm because the problem was not being observed while the device was in my shop." and keeda mentioned timeline‑based queries where LLMs sucked.
  • Provides a concrete way to ground LLM answers in authoritative log‑derived facts, reducing hallucinations and enabling reproducible investigations.

HeuristicGuide: Search‑Augmented Planner for LLM Coding Agents

Summary

  • Adds heuristic search (A*, BFS) and constraint checking from a Datalog codebase to LLM‑driven coding agents, helping them avoid dead‑ends and reducing the need for manual prompting.
  • Improves the reliability of agentic code generation by combining old‑school AI planning with modern LLMs.

Details

Key Value
Target Audience Developers building agentic coding tools (e.g., Copilot‑like assistants) or researchers experimenting with LLM agents for software synthesis
Core Feature Planner module that takes a high‑level goal, expands candidate edits using an LLM, scores them with heuristic functions (e.g., test pass likelihood, syntax validity), and consults a Datalog‑encoded code‑base model (call graphs, type info) to prune infeasible paths
Tech Stack Python, LLM integration via litellm or Ollama, heuristic library (networkx), Datalog engine (Soufflé or LogicBlox), optional WASM for sandbox
Difficulty Medium
Monetization Hobby (open‑source prototype; could later offer premium plugins or consulting)

Notes

  • linguee mentioned leveraging heuristic search to guide agents when planning, and tptacek compared the idea to embedding CodeQL's Datalog extraction.
  • HN community loves discussions about merging symbolic AI with LLMs; this project offers a tangible experiment that could spark further debate and practical adoption.

Read Later