Long-form technical video libraries are hard to search because the useful unit is rarely the whole video. A one-hour conference recording might mention dependency injection for three minutes, compare workflow engines later, and answer an unrelated audience question near the end. Title search and description search miss most of that detail, while full transcription alone still leaves users scanning a wall of text.
A caption-based video semantic search pipeline turns available video captions into timestamped, embedding-backed passages. The MariaDB hackathon project used YouTube captions as the raw source, restored punctuation, split text into sentence-aligned chunks, embedded those chunks locally with all-MiniLM-L6-v2, and stored the vectors with video identifiers, timestamps, titles, and direct segment links. At query time, the user’s text query is embedded with the same model and compared against stored chunk vectors to return the most relevant moments in the source videos.
The transferable pattern is not tied to one video platform or one vector database. The same architecture works for training recordings, product demos, webinars, podcasts with captions, or internal screen-share archives. The core design challenge is preserving enough temporal provenance to jump back to the exact moment while still creating coherent text chunks that produce useful semantic embeddings.
The pipeline treats caption text as the document body. Captions are attractive because they are already time-aligned with the video and are much cheaper to process than raw audio. The tradeoff is quality: autogenerated captions may lack punctuation, break sentences at arbitrary time intervals, and mistranscribe technical terms. The ingestion step must therefore clean and restructure captions before embedding.
Search results should return passages with start and end times, not just text. Each chunk needs a stable source video ID, title, start timestamp, end timestamp, original caption text, and a direct playback URL. This makes the vector row both a retrieval unit and a navigation unit: users can read the snippet, then jump straight to the matching section of the recording.
The corpus chunks and search queries must be encoded by the same embedding model, with the same normalization behavior, so their vectors live in the same semantic space. The source project uses the Sentence Transformers all-MiniLM-L6-v2 model, which maps sentences and short paragraphs into 384-dimensional dense vectors for semantic search and clustering.
Segment-level retrieval indexes the moment where an idea appears rather than the entire media object. The user asks for a concept, the system retrieves a caption chunk, and the UI links to the precise video timestamp that produced that chunk.
The population path is batch-oriented. It fetches captions for selected video IDs, restores punctuation, splits the repaired text into sentences, groups nearby sentences into token-bounded chunks, embeds each chunk, and inserts vector rows into the database. The source project reports that preparing one hour of captioned video typically took up to about 30 seconds on a local MacBook Pro, including caption fetching, punctuation restoration, chunking, embedding, and database insertion.
# Python: caption chunk population shape
from dataclasses import dataclass
from typing import Iterable
@dataclass
class CaptionSegment:
text: str
start_s: float
end_s: float
@dataclass
class VideoChunk:
video_id: str
title: str
text: str
start_s: float
end_s: float
embedding: list[float]
def populate_video(video_id: str, title: str, embedder, store) -> None:
raw_segments = fetch_captions(video_id)
repaired_text = restore_punctuation(raw_segments)
sentences = split_into_sentences(repaired_text)
for chunk_text, start_s, end_s in token_bounded_chunks(sentences, max_tokens=220):
vector = embedder.encode(chunk_text)
store.insert_chunk(
VideoChunk(
video_id=video_id,
title=title,
text=chunk_text,
start_s=start_s,
end_s=end_s,
embedding=vector,
)
)The query path is intentionally small. It embeds the user query, asks the vector store for the nearest chunks by distance, and renders the best matches with direct segment links. MariaDB Vector supports vector columns, vector indexes, and distance functions such as Euclidean and cosine distance, so a relational row can hold both the semantic vector and the playback provenance.
-- SQL shape: retrieve nearest caption chunks
SELECT
video_id,
title,
start_s,
end_s,
text,
VEC_DISTANCE_EUCLIDEAN(embedding, VEC_FromText(:query_vector)) AS distance
FROM video_chunks
ORDER BY distance
LIMIT :k;A good result is more than a raw row. It should include a readable excerpt, a relevance score or distance, and a URL such as https://www.youtube.com/watch?v=VIDEO_ID&t=STARTs. The system should also show enough neighboring context to make the result understandable without forcing the user to open every candidate.
Technical communities often publish many hours of talks. Caption-based semantic search lets a developer search for an implementation concept, library name, or architecture pattern and jump to the relevant moment, even when the video title is broad or the speaker uses different wording than the query.
Organizations record onboarding sessions, product walkthroughs, design reviews, and customer demos. Turning captions into timestamped semantic chunks makes those recordings discoverable through the same retrieval interface used for documents. This reduces duplicate training sessions and helps engineers locate prior explanations quickly.
The retrieved chunks can feed a RAG system, but the answer should preserve the video timestamp as evidence. A response that says “the speaker compares Temporal and Celery at 18:42” is more useful than a generic answer because it lets the user inspect the source moment directly.
The source repository separates caption retrieval, caption transformation, chunking, embedding, storage, and query services. That boundary matters because each stage changes at a different rate. You may replace a caption provider, switch punctuation models, tune chunk sizes, or move from raw SQL to an ORM without rewriting the entire pipeline.
# Python: service boundaries for replaceable pipeline stages
class CaptionFetcher:
def fetch(self, video_id: str) -> list[CaptionSegment]: ...
class CaptionTransformer:
def repair(self, segments: list[CaptionSegment]) -> list[CaptionSegment]: ...
class Chunker:
def split(self, segments: list[CaptionSegment]) -> list[tuple[str, float, float]]: ...
class Embedder:
def encode(self, text: str) -> list[float]: ...
class ChunkStore:
def insert(self, chunk: VideoChunk) -> None: ...
def nearest(self, vector: list[float], k: int) -> list[VideoChunk]: ...Raw caption intervals are usually bad retrieval units. They may cut off clauses mid-thought or contain too little context for a useful embedding. A better pattern is to restore punctuation, split into sentences, then combine adjacent sentences until a token budget is reached. This keeps chunks coherent while preventing overlong inputs from being truncated by the embedding model.
Do not store embeddings in a detached table without source context. The retrieval row should include the video ID, title, timestamps, text, embedding model identifier, and ingestion version. Those fields make search results debuggable and allow re-embedding when the model or chunking policy changes.