Data engineering: LLM extraction pipelines over a unified entity graph
PodcastPal
A podcast discovery app built on one unified graph linking live events, host-and-guest relationships, and the merch those people sell.
- Role
- Sole engineer, designer, and operator
- Year
- 2026
- Status
- In Development
- Site
- podcastpal.io
- React 19 / Vite / TypeScript
- Python / FastAPI
- Gemini 2.5 Flash (extraction)
- PostgreSQL + PostGIS (Neon)
- Lambda / EventBridge / SQS ingestion
- D3.js / Leaflet
- AWS CDK / Caddy on EC2
5,500+
Podcast episodes ingested and graphed
2026-04 · RSS ingestion + LLM extraction pipeline backfill
Challenge
Podcast events, host-and-guest relationships, and the merch those people endorse are usually separate features. PodcastPal makes them three views of one entity graph that enrich each other. The hard part: building it cheaply and correctly from messy third-party RSS, without a graph database or runaway per-request LLM cost.
Process
Ingestion is off the request path. A scheduled Lambda scrapes RSS into an S3 bronze tier; SQS workers run Gemini 2.5 Flash extraction into entities (podcasts, people, venues, events, merch); pages serve from Postgres with no per-request model cost. The graph lives in PostgreSQL with PostGIS by design: recursive queries and junction tables handle the relationship web at expected scale for roughly nothing, versus a graph database’s idle cost. A per-call cost tracker and budget ceiling cap extraction spend. D3 renders the lineage graph; Leaflet maps events.
Before beta, a dual-model adversarial audit had two independent AI reviewers work the same brief in isolation. I merged their findings into a fix queue and closed it out. Authorization, JWT validation, and SQL parameterization came back clean.
Result
Deployed to production at podcastpal.io with over 5,500 episodes ingested and graphed. Pre-launch, not open to customers. An entity-resolution pass for graph quality is designed and partly underway. The unified graph is the product spine.