skills · Retrieval quality engineering
Advanced Retrieval: Hybrid Search and Reranking
Hybrid search and reranking, measured on real labels.
By the end, you'll be able to:
- A labeled query set from your own users
- A scorecard with nDCG, recall, latency and p-values per change
- Tuned BM25 and a chosen embedding model
- A fusion weight tuned on train and checked on held-out queries
- A reranker and depth chosen under a latency budget
- LLM labels checked against yours
- A regression gate and a one-page retrieval spec
From $9/month8 hours5 chapters20 lessons
“Our reranker made results worse. The per-query numbers showed why.”
About this course
Who it's for
- You have search or RAG retrieval working and need to make it measurably better
- You are deciding between BM25, embeddings, hybrid search and a reranker, and want evidence for your data
- You need a regression check before retrieval changes reach users
What you'll need
- You write Python and use a virtual environment
- You have built or run a retriever (course 055 covers the basics)
Skills you'll practice
- Retrieval evaluation
- Significance testing
- BM25 tuning
- Embedding models
- HNSW
- Hybrid search
- Cross-encoder reranking
- HyDE
How long it takes
About 8 hours in total
- Chapter 1 · Measure before you tune1.5 hours
- Chapter 2 · Tune each retriever1.5 hours
- Chapter 3 · Fusion, measured1.5 hours
- Chapter 4 · Reranking1.5 hours
- Chapter 5 · Queries, labels, and keeping it good2 hours
What's included
- 20 lessons
- 87 readings
- 20 worked examples
- 40 exercises
- 40 quick checks
- 20 common myths
- 20 quizzes
- 5 end-of-chapter checks
Curriculum
5 chapters, 20 lessons
The full course: every chapter and lesson. Tap a chapter to expand it. Lessons open when you start.
Career
Where this course can take you
The roles this course maps to, and the skills you can list on your resume and LinkedIn once you finish it.
Skills you will build
Ready to paste into your resume or LinkedIn profile once you have finished the course.
- Building a labeled retrieval test setBuild a relevance-labeled evaluation set from real user queries
- Choosing retrieval metrics by pipeline stageEvaluate retrieval with nDCG, recall and MRR
- Significance testing for retrieval changesCompare retrieval systems per query with a paired significance test
- Tuning BM25 on a held-out splitTune BM25 analyzers and parameters for a search corpus
- Selecting an embedding model by measurementEvaluate embedding models on domain-specific labeled queries
- Evaluating approximate nearest neighbor searchMeasure HNSW recall against exact search and its effect on ranking quality
- Per-query loss analysis for retrieversAnalyze retrieval failures per query and categorize them by type
- Tuning weighted score fusion on a held-out splitCombine keyword and vector search scores with normalization and a tuned weight
- Evaluating reciprocal rank fusionImplement reciprocal rank fusion for hybrid search and test its rank constant
- Setting candidate depth from first-stage recallMeasure first-stage recall at several depths to size a reranking stage
- Configuring hybrid search in a search engineConfigure hybrid search in Elasticsearch, OpenSearch or Postgres with pgvector
- Evaluating a cross-encoder reranker against its first stageAdd and evaluate a cross-encoder reranking stage in a retrieval pipeline
- Choosing rerank depth under a latency budgetMeasure p50 and p95 latency of a reranking stage at several candidate depths
- Comparing rerankers on domain labelsCompare open and hosted reranking models on a labeled domain query set
- Testing an LLM as a listwise rerankerEvaluate LLM-based listwise reranking with cost and latency per query
- Evaluating LLM query rewriting for retrievalEvaluate query rewriting and HyDE for search and RAG
- Validating LLM relevance labels against human judgmentsUse LLM judges for search relevance labeling and measure their agreement with human labels
- Regression testing and monitoring for search qualityBuild offline regression checks for retrieval quality with significance testing
- Documenting a retrieval system for handoverDocument search and retrieval architecture decisions with their measurements and trade-offs
Questions
Before you commit
Start today