← Back to Work

Latency engineering for an LLM-and-third-party-API recommendation pipeline

MovieNightPal

A family movie-night assistant that tracks each member's success and failure sentiment on every pick, so 'what should we watch' stops being a fight.

Role
Sole engineer, designer, and operator
Year
2026
Status
In Development
  • React 19 / Vite / TypeScript
  • Python / FastAPI
  • Gemini API (Flash) (sentiment + recommendations)
  • OMDb + Watchmode APIs (metadata + availability)
  • PostgreSQL (Neon)
  • AWS Cognito
  • AWS CDK / EC2 + ALB

34.7s cold to 0.24s cached

Recommendation endpoint latency (Gemini + OMDb), cold vs. cached

2026-07-03 · Staging verification of the household-scoped cache, docs/session_contexts (2026-07-04 session doc)

Challenge

Movie-night paralysis is a group-decision problem disguised as a search problem: every household member has their own taste and their own history of what actually worked on past nights, and picking something everyone will enjoy means weighing all of that at once, not just matching genres. MovieNightPal tracks success and failure sentiment per family member against every past pick, then routes that history through an LLM to recommend the next one. The catch: a recommendation that reasons over a household’s whole viewing history and cross-references third-party movie metadata is inherently slow, and slow is exactly what kills a feature meant to end an argument in the living room.

Process

The recommendation engine scores candidates against the household’s actual history: genre affinity learned from past outcomes, age fit against the youngest member, repeat avoidance, and success-rate weighting, rather than a generic trending list. I made one rule a code-level invariant, not a UI convention: an under-13 member’s movie marks can only be written by a parent acting as proxy, enforced by a guard on the write path that rejects anything else, and covered by tests.

Recommendations run Gemini Flash reasoning against OMDb metadata, and an uncached call on staging measured at roughly 35 seconds with intermittent empty responses, an unacceptable wait for a household standing in front of the TV. I fixed it with an in-process, household-scoped TTL cache with single-flight request collapsing and stale-while-error fallback, plus an explicit refresh escape hatch for when a household genuinely wants a new answer rather than the cached one. The measured result: 34.7 seconds cold down to 0.24 seconds cached on the exact path users hit most.

The other distinctive piece is editorial, not code: the classics shelf is backed by a YAML canon in the repo, but the actual source of truth is a Google Sheet. A sync script pulls the Sheet, validates every row against the schema, and only writes the YAML if validation passes; a failed sync leaves the bad output quarantined for inspection instead of landing broken data. That pipeline took the classics catalog from 19 to 178 films in a June 2026 production sync-validate-seed cycle without ever hand-editing the file that ships.

Result

Live at movienightpal.com, and in active development on a staging branch that has passed a full Tonight-to-Discover-to-vote-to-Score-the-Night user journey end to end (604 backend and 445 frontend tests green, verified 2026-07-04). It is pre-launch, with no customers, and I intentionally hold the production site a step behind the staging build while that surface finishes soaking. The reusable idea is the caching pattern: treat a slow LLM-plus-third-party pipeline as a cache-invalidation problem instead of trying to make the underlying call faster.