Skip to main content

skills · Retrieval quality engineering

Advanced Retrieval: Hybrid Search and Reranking

Hybrid search and reranking, measured on real labels.

By the end, you'll be able to:

  • A labeled query set from your own users
  • A scorecard with nDCG, recall, latency and p-values per change
  • Tuned BM25 and a chosen embedding model
  • A fusion weight tuned on train and checked on held-out queries
  • A reranker and depth chosen under a latency budget
  • LLM labels checked against yours
  • A regression gate and a one-page retrieval spec

From $9/month8 hours5 chapters20 lessons

“Our reranker made results worse. The per-query numbers showed why.”

About this course

Who it's for

  • You have search or RAG retrieval working and need to make it measurably better
  • You are deciding between BM25, embeddings, hybrid search and a reranker, and want evidence for your data
  • You need a regression check before retrieval changes reach users

What you'll need

  • You write Python and use a virtual environment
  • You have built or run a retriever (course 055 covers the basics)

Skills you'll practice

  • Retrieval evaluation
  • Significance testing
  • BM25 tuning
  • Embedding models
  • HNSW
  • Hybrid search
  • Cross-encoder reranking
  • HyDE

How long it takes

About 8 hours in total

  1. Chapter 1 · Measure before you tune1.5 hours
  2. Chapter 2 · Tune each retriever1.5 hours
  3. Chapter 3 · Fusion, measured1.5 hours
  4. Chapter 4 · Reranking1.5 hours
  5. Chapter 5 · Queries, labels, and keeping it good2 hours

What's included

  • 20 lessons
  • 87 readings
  • 20 worked examples
  • 40 exercises
  • 40 quick checks
  • 20 common myths
  • 20 quizzes
  • 5 end-of-chapter checks

Curriculum

5 chapters, 20 lessons

The full course: every chapter and lesson. Tap a chapter to expand it. Lessons open when you start.

    • BM25 That Reads Your Text
    • Choosing an Embedding Model
    • Approximate Search: Buying Speed With Recall
    • Which Queries Each Retriever Loses
    • Scores in Different Currencies
    • Reciprocal Rank Fusion and Its k
    • The Recall Ceiling
    • Hybrid in the Engine You Run
    • A Cross-Encoder Reads Both
    • Depth and the Latency Budget
    • Choosing a Reranker
    • An LLM as the Reranker
    • Rewriting the Query
    • Labels at Scale With an LLM
    • A Regression Check and Production Signals
    • The Retrieval Spec

Career

Where this course can take you

The roles this course maps to, and the skills you can list on your resume and LinkedIn once you finish it.

Skills you will build

Ready to paste into your resume or LinkedIn profile once you have finished the course.

  • Building a labeled retrieval test setBuild a relevance-labeled evaluation set from real user queries
  • Choosing retrieval metrics by pipeline stageEvaluate retrieval with nDCG, recall and MRR
  • Significance testing for retrieval changesCompare retrieval systems per query with a paired significance test
  • Tuning BM25 on a held-out splitTune BM25 analyzers and parameters for a search corpus
  • Selecting an embedding model by measurementEvaluate embedding models on domain-specific labeled queries
  • Evaluating approximate nearest neighbor searchMeasure HNSW recall against exact search and its effect on ranking quality
  • Per-query loss analysis for retrieversAnalyze retrieval failures per query and categorize them by type
  • Tuning weighted score fusion on a held-out splitCombine keyword and vector search scores with normalization and a tuned weight
  • Evaluating reciprocal rank fusionImplement reciprocal rank fusion for hybrid search and test its rank constant
  • Setting candidate depth from first-stage recallMeasure first-stage recall at several depths to size a reranking stage
  • Configuring hybrid search in a search engineConfigure hybrid search in Elasticsearch, OpenSearch or Postgres with pgvector
  • Evaluating a cross-encoder reranker against its first stageAdd and evaluate a cross-encoder reranking stage in a retrieval pipeline
  • Choosing rerank depth under a latency budgetMeasure p50 and p95 latency of a reranking stage at several candidate depths
  • Comparing rerankers on domain labelsCompare open and hosted reranking models on a labeled domain query set
  • Testing an LLM as a listwise rerankerEvaluate LLM-based listwise reranking with cost and latency per query
  • Evaluating LLM query rewriting for retrievalEvaluate query rewriting and HyDE for search and RAG
  • Validating LLM relevance labels against human judgmentsUse LLM judges for search relevance labeling and measure their agreement with human labels
  • Regression testing and monitoring for search qualityBuild offline regression checks for retrieval quality with significance testing
  • Documenting a retrieval system for handoverDocument search and retrieval architecture decisions with their measurements and trade-offs

Questions

Before you commit

You already have search or RAG retrieval working and need to know, with numbers, whether a change makes it better: a new embedding model, hybrid search, a reranker or query rewriting.

No. In the course's own run on SciFact, a weighted hybrid of BM25 and an embedding model scored higher on average than either alone, but 300 test queries could not confirm the gain at the usual significance level. You learn to run that test on your data.

No GPU. Chapters 1 to 4.3 run on a laptop CPU with open models. Lesson 4.4 and chapter 5 call a hosted LLM; the sample runs cost a few cents, and every step can print its prompt instead of calling a model.

No, if you have built a retriever before. Course 055 RAG Fundamentals teaches the basics this course builds on: keyword search, embeddings and a first RAG pipeline.

The lab is plain Python with open libraries, so every step is visible. Lesson 3.4 maps each setting to Elasticsearch, OpenSearch and Postgres with pgvector, from their documentation as of November 2026.

Start today