projects / 06
cue
multimodal video search
Upload hours of video or paste a link, then ask in plain language. cue indexes every scene three ways, by what is said, what is shown and what is written on screen, and answers with timestamped clips, each ranked by a reranker that says why it matched. The foundation is in place: the monorepo, the schema on Neon with pgvector, an interface for every model, and upload and link ingest with deduplication. The processing pipeline and search are being built now.
highlights
- Every model sits behind an interface (speech-to-text, embedders, OCR, reranker, answerer), so providers swap without touching the pipeline
- Uploads go straight to storage through presigned URLs and are deduplicated by SHA-256 content hash
- Vector and keyword search live in one Postgres: three pgvector embeddings and a tsvector per scene
how it works
- 01
Ingest
Upload a file through a presigned URL or paste a link, which yt-dlp resolves for video pages. The API hashes the file, skips anything it has already seen and queues the first stage.
- 02
Decode and segment
ffmpeg extracts 16 kHz mono audio and samples frames. Scene detection then cuts the video into segments, merging short scenes and splitting long ones, with a keyframe for each.
- 03
Transcribe
OpenAI speech-to-text returns word-level timestamps. Long audio is split under the API's size limit and the timestamps are stitched back together.
- 04
See
Keyframes get Voyage multimodal embeddings and OCR for text on screen, skipping frames that barely changed.
- 05
Index
Transcript and on-screen text are embedded, with a cache keyed by content hash so nothing is embedded twice, and the video is marked ready.
- 06
Search
A query runs vector search on each modality alongside Postgres keyword search, merges the lists with reciprocal rank fusion and has Claude rerank the top hits with a short reason each. Neighbouring hits join into clips whose edges snap to word boundaries.
design decisions
Hosted models behind interfaces
Each model role is an interface with adapters registered at startup. Swapping a provider is one adapter, and the eval harness can compare them on the same videos.
One queue per stage
Each stage has its own BullMQ queue and a row in stage_runs with an input hash, so a rerun skips unchanged work and a failed video resumes from the first unfinished stage instead of starting over.
pgvector over a separate vector store
Embeddings sit next to the transcript, the keyword index and the timestamps, so hybrid search and its fusion run as queries in one database with nothing to keep in sync.
Evals before polish
Retrieval is measured with recall@1/5/10 on MSR-VTT, timestamps with hit-within-window and IoU on hand-labelled queries, and every modality is ablated to show what it adds.
known limitations
- Early: the foundation and the ingest endpoints work; the processing stages, search, evals and UI are in progress.
- Indexing runs on hosted APIs (OpenAI, Voyage, Anthropic), so it needs a network and costs money per video.
stack
- typescript
- next.js
- express
- bullmq
- redis
- postgres
- pgvector
- drizzle
- s3
- openai
- voyage ai
- claude