How We Pushed CDC into Postgres

📝 Discussion Summary (Click to expand)

Snowflake’s appeal but underlying complexity

“Snowflake is a really amazing product. It's been a delight using it the last few years.” — whateveracct
Replication systems are fragile; schema‑change handling is a major pain point

“This was basically Vertica's party trick ... fragile to schema updates in some direction.” — gopalv
ClickHouse (with peerdb) offers a robust, batteries‑included solution, albeit with its own limits

“Clickhouse really nailed this with the acquisition of peerdb. I used it with many terabyte databases and I essentially never thought about it.” — bastawhiz

🚀 Project Ideas

Provides a fully managed, CDC‑aware pipeline that mirrors PostgreSQL changes straight to Snowflake, Redshift, or BigQuery with automatic schema evolution and fail‑over.
Core value: eliminates fragile hand‑rolled GoldenGate‑style scripts and Aurora zero‑ETL pitfalls, delivering reliable, near‑real‑time data for analytics.

Key	Value
Target Audience	Data engineers & analytics teams running PostgreSQL workloads that need low‑latency warehouse sync
Core Feature	Serverless CDC connector with automatic schema migration, backpressure handling, and self‑healing retries
Tech Stack	AWS Lambda + Step Functions, PostgreSQL logical decoding, Kafka‑style log compacted topics, Snowflake bulk ingest API
Difficulty	Medium
Monetization	Revenue-ready: pay‑as‑you‑go $0.01/GB transferred + $0.02/1k connector‑instances

HN commenters lament Aurora zero‑ETL instability and fragile CDC tools—this directly addresses those pain points.
Potential for community contributions around open‑source connectors and schema‑drift detection.

A visual, low‑code workflow builder that deploys and monitors replication jobs across multiple clouds, handling schema changes automatically.
Core value: lets teams replace brittle GoldenGate‑style hacks with a single pane of glass that self‑heals pipeline failures.

Key	Value
Target Audience	DevOps & data platform engineers managing heterogeneous replication pipelines
Core Feature	Drag‑and‑drop DAG designer, built‑in alerting, versioned schema migrations, and auto‑retry with exponential backoff
Tech Stack	React front‑end, Node.js worker services, PostgreSQL for state, Redis for queue, Kubernetes for scaling
Difficulty	High
Monetization	Revenue-ready: tiered SaaS $49/mo per workspace + $0.005 per pipeline‑run

Users repeatedly cite Aurora zero‑ETL failures and ClickHouse peerdb reliability—ZetaFlow offers the “batteries‑included” experience they request.
Could spark discussion on UI vs. code‑first pipeline management in the HN community.

A lightweight PostgreSQL extension that streams WAL changes to immutable S3/Blob storage with built‑in compaction and versioning, enabling reliable downstream analytics.
Core value: provides a stable, low‑maintenance replication primitive that avoids the fragility of manual GoldenGate or Aurora zero‑ETL setups.

Key	Value
Target Audience	Open‑source contributors and data engineers who need CDC without managed SaaS costs
Core Feature	Transaction‑log tailing, automatic S3 key partitioning, periodic compaction, schema‑aware record conversion
Tech Stack	C extension for PostgreSQL, AWS S3 API, Python glue for compaction jobs, Docker for deployment
Difficulty	High
Monetization	Hobby

Directly answers “pg_lake is missing CDC capabilities” and the frustration with schema updates breaking pipelines.
Likely to generate technical discussion around performance tuning and integration with Trino/ClickHouse.