Project ideas from Hacker News discussions.

Building durable workflows on Postgres

📝 Discussion Summary (Click to expand)

```markdown

1. PostgreSQL as the backbone for queues & durable workflows

"Continuously amazed by what you can do with few tools, as long as Postgres is a part of your toolkit." – sgt
"I used SKIP LOCKED so that workers can compete safely." – sgt
"Advisory locks are preferred for this anyways because holding a lot of SELECT FOR UPDATE doesn’t scale too well." – bootsmann These snippets show a strong preference for leveraging PostgreSQL primitives (SKIP LOCKED, advisory locks, pgmq/pgflow) to implement reliable queues without adding separate services.

2. Temporal’s scalability and cost concerns

"If you're doing anything non‑trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck." – temporal_thr123
"Temporal is, in my opinion having run it in prod for over a year – poorly designed, slow and ridiculously heavy infra wise." – temporal_thr123
"We run many hundreds of thousands of these a day... Temporal was a bad fit for us, and we regret it deeply." – temporal_thr321

The community repeatedly flags Temporal’s heavy infrastructure requirements, steep price at scale, and operational pain.

3. Alternatives & trade‑offs (DBOS, pgflow, pgmq, absurd, etc.)

"PgFlow is pretty awesome for DAG workflows – it's built on pgmq (which does the heavy lifting, making it backend agnostic)." – cpursley
"If you don’t need a ton of throughput I think absurd (and our Rust derivative durable) are very nice options that keep the client side extremely simple." – vrm
"All you need is Postgres until you scale into TBs of data. The main benefit is centralizing all the data in one place so we don’t need to worry about copying data in between multiple systems." – hmaxdml

These points capture the ongoing debate: stay within PostgreSQL for many use‑cases, but consider dedicated alternatives like DBOS, pgflow, or absurd when specific limitations arise.


🚀 Project Ideas

AtomicDB – Durable Workflow Engine for Postgres

Summary

  • Builds durable, stateful workflows directly on Postgres with atomic enqueue‑within‑transaction semantics.
  • Eliminates race conditions and the need for external orchestrators while keeping the API simple.

Details| Key | Value |

|-----|-------| | Target Audience | Elixir, Node.js, Python developers building job queues, notification pipelines, and data‑processing workflows | | Core Feature | Atomic workflow definition with built‑in retry, idempotent steps, and state stored in Postgres tables; conflict‑free execution using SKIP LOCKED and optional advisory locks | | Tech Stack | PostgreSQL 13+, PL/pgSQL/SQL, SDKs in Elixir, Python, Node.js, Docker for local dev | | Difficulty | Medium | | Monetization | Revenue-ready: $0.01 per 1,000 workflow executions (tiered, free up to 10k) |

Notes

  • Directly addresses HN frustrations: “atomic messaging tied to a PostgreSQL transaction” and “avoid Temporal’s cost and complexity” (see DBOS slide discussion). - Provides a lightweight alternative that can be self‑hosted or offered as a hosted SaaS, making it attractive for teams scaling beyond simple queues.

QueuePulse – Autonomous Postgres Queue Scaler

Summary

  • Auto‑optimizes high‑throughput Postgres job queues, preventing vacuum bloat and dead‑tuple slowdown that cripple SKIP LOCKED at scale.
  • Supplies an admin UI and API for real‑time queue health monitoring and partitioning.

Details

Key Value
Target Audience Ops and backend engineers running heavy job queues (e.g., email, payment processing) on a single PostgreSQL instance
Core Feature Time‑bucket table sharding, background vacuum/reorg worker, and index‑aware job fetch that keeps throughput >10k ops/sec without manual tuning
Tech Stack PostgreSQL extensions (PL/pgSQL), Node.js backend, React admin UI, Docker for deployment
Difficulty High
Monetization Revenue-ready: $0.005 per 1,000 jobs processed (hosted SaaS tier)

Notes

  • Solves the exact pain point highlighted by nulltrace: “vacuum can’t keep up → dead tuples pile up, planner stops using index.”
  • Appeals to teams that want to stay within PostgreSQL rather than migrate to external queues, aligning with HN discussions about “queues built on Postgres” and “need for scalable LISTEN/NOTIFY”.

ServerlessDurable – Pay‑as‑You‑Go Durable Function Runner on Postgres

Summary- Lets developers write durable, retry‑aware functions that are persisted in Postgres, handling large payloads via signed URLs.

  • Provides an out‑of‑the‑box scheduler, observability dashboard, and idempotent execution guarantees.

Details

Key Value
Target Audience Product teams and solo hackers seeking durable execution without Temporal or DBOS overhead; any stack that already uses Postgres
Core Feature Function definitions stored as JSON, runtime scheduling via triggers, automatic retries, external object storage linkage for >1 MB payloads, real‑time status dashboard
Tech Stack PostgreSQL, FastAPI/Node.js serverless layer, S3/MinIO for large payloads, Docker for local dev
Difficulty Medium
Monetization Revenue-ready: $0.001 per function execution (pay‑as‑you‑go, 100k free executions/month)

Notes

  • Directly responds to HN concerns about “large payload handling” and “cost of Temporal Cloud” (e.g., switchbak’s external storage comment).
  • Offers a cheaper, simpler alternative to Temporal’s pricing model, making it attractive for high‑volume workflows discussed in the thread.

Read Later