YouTube Transcript Retrieval Lab

Overview

YouTube Transcript Retrieval Lab started as a RAG project for studying long English and Japanese YouTube videos, and grew into a lab for testing how well search works over them. Videos start in a catalog, get indexed into a library with their timestamped transcripts, and then I run two retrieval setups side by side on the same question, label what comes back, and see which one actually finds the right moments.

The raw transcript stays the source of truth, so every result and answer links back to a timestamp in the video. Summaries and study tools like flashcards and topic maps use the same evidence.

Core flow:

Catalog: discovered videos, metadata only
  -> Library: transcripts + timestamped chunks
  -> E5 + FAISS (semantic) and BM25 (keyword, Japanese-aware)
  -> Search: Run A vs Run B
       hybrid, dense, lexical, or agentic, optional reranker
  -> relevance labels + Precision@K / nDCG@K
  -> Evaluate: query sets, runs, offline benchmark

Enlarge image ↗

Why I Built It

I started this project because I wanted a tool I would actually use myself. While studying for the JLPT, I realized listening was one of my weaker areas, so I started spending more time with long-form Japanese YouTube videos. I kept wishing I had an easier way to check whether I was really following the content, pull out the main points, and revisit parts I did not fully catch. I was also practicing building retrieval systems, so I thought, why not build something around that too?

Once I began building it, I wanted to add more than just search. Since I was trying to improve retrieval quality, I added features that show where an answer came from, make it easier to inspect what was being retrieved, and help me tell whether the search was actually getting better over time.

That part ended up being the most fun. It started as a personal learning tool and turned into a lab for building and measuring retrieval more carefully.

How It Evolved

Transcript search to RAG (Feb–Jun 2026). It started as a RAG project: ingest YouTube transcripts, chunk them with timestamps, search them with hybrid semantic (E5 + FAISS) and keyword (BM25) retrieval, and answer questions with citations back to the video. Summaries and study tools came along the way, and a first offline benchmark let me check whether search was improving.

Agentic retrieval (Jul–Aug 2026). An agent picks semantic or keyword search, checks whether the evidence holds up, switches tools when it's weak, and reads nearby transcript context. I added cross-encoder reranking, made the agent's decisions deterministic, and added video routing so questions across the whole library search the most relevant videos first.

Evals and A/B testing (Aug–Oct 2026). Codex CLI drafts evaluation cases that I review before they become data, and Search became an A/B workbench: two retrieval setups on the same question, relevance labels, and per-query Precision@K and nDCG@K, with a catalog and library around it. Building it turned up a measurement problem. My labels were also feeding a ranking adjustment, so re-running a query pushed the results I had just marked relevant up the list. Comparisons now switch that off, so my labels can't shape the rankings being compared.

How It Works

• Catalog and index: discovered videos sit in a read-only catalog with metadata and review status, and videos or playlists are indexed into the library with timestamped transcripts
• Search: semantic search with E5 embeddings in FAISS, keyword search with Japanese-aware BM25, and hybrid fusion of both, with an optional cross-encoder reranker
• Agentic retrieval: a deterministic policy picks semantic or keyword search, switches when evidence is weak, and reads nearby transcript context
• Compare and label: Search runs two setups on the same question, shows how the rankings moved, and scores both runs from my relevance labels
• Evaluate: query sets, saved runs, and Precision@K, Recall@K, MRR, and nDCG@K, plus an offline benchmark that runs without network access or API keys
• Grounded answers: answers cite the retrieved transcript chunks and say when the evidence isn't enough, using ChatGPT, Claude, or Sakana AI

Features

• Search workbench: compare Run A and Run B side by side, with rank changes, relevance labels, per-query Precision@K and nDCG@K, and the steps each run took to find its results
• Library: indexed videos with chunk counts, chunking status, and catalog review status, plus one form for adding videos or playlists
• Catalog: browse discovered videos by language, topic, and review status before deciding what to index
• Evaluate: save queries from Search, run query sets, label results, and compare runs. Codex CLI can draft evaluation cases for human approval
• Answers and study tools: citation-backed answers, five-theme summaries, flashcards, and topic maps built from the same transcript evidence
• Local video: OCR on video files I own adds on-screen text as evidence

Enlarge image ↗

Project Status

This project is still a work in progress