← Back to Work

Latency engineering for an LLM-and-third-party-API recommendation pipeline

MovieNightPal

A family movie-night assistant that tracks each member's success and failure sentiment on every pick, so 'what should we watch' stops being a fight.

Role
Sole engineer, designer, and operator
Year
2026
Status
In Development
  • React 19 / Vite / TypeScript
  • Python / FastAPI
  • Gemini API (Flash) (sentiment + recommendations)
  • OMDb + Watchmode APIs (metadata + availability)
  • PostgreSQL (Neon)
  • AWS Cognito
  • AWS CDK / EC2 + ALB

34.7s cold to 0.24s cached

Recommendation endpoint latency (Gemini + OMDb), cold vs. cached

2026-07-03 · Staging verification of the household-scoped cache, docs/session_contexts (2026-07-04 session doc)

Challenge

Movie-night paralysis is a group decision posing as a search problem. Each household member has their own taste and history of what worked on past nights, and a good pick weighs all of it, beyond genre matching. MovieNightPal tracks each member’s success and failure sentiment on every past pick and feeds that history to an LLM to recommend the next one. The catch: reasoning over a whole viewing history plus third-party movie metadata is slow, and slow kills a feature meant to end a living-room argument.

Process

The engine scores candidates on the household’s history instead of a generic trending list: genre affinity learned from past outcomes, age fit for the youngest member, repeat avoidance, and success-rate weighting. One rule is a code-level invariant: only a parent acting as proxy can write an under-13 member’s movie marks. A write-path guard rejects anything else, and tests cover it.

Recommendations run Gemini Flash against OMDb metadata. Uncached calls on staging took about 35 seconds, with intermittent empty responses. I added an in-process, household-scoped TTL cache with single-flight request collapsing, stale-while-error fallback, and an explicit refresh for households that want a new answer. Measured result: 34.7 seconds cold to 0.24 seconds cached on the path users hit most.

The classics shelf ships as a YAML canon in the repo; its source of truth is a Google Sheet. A sync script pulls the Sheet, validates every row against the schema, and writes the YAML only on a pass. Failed output is quarantined for inspection. A June 2026 production sync-validate-seed cycle grew the catalog from 19 to 178 films with no hand edits to the shipped file.

Result

Deployed at movienightpal.com, with active development on a staging branch that has passed the full Tonight-to-Discover-to-vote-to-Score-the-Night user journey (604 backend and 445 frontend tests green, verified 2026-07-04). Pre-launch, no customers; I keep production a step behind staging while that surface finishes soaking. The reusable idea: treat a slow LLM-plus-third-party pipeline as a cache-invalidation problem instead of trying to make the underlying call faster.