RAG AI Search, a AI-powered retrieval-augmented generation (RAG) search app
A retrieval-augmented Q&A app over indexed course lecture material — answers come only from what's retrieved, with citations, and an honest 'nothing found' when nothing is relevant.
Overview
RAG AI Search indexes CS382/SEIR course lecture PDFs and notes into a local FAISS vector store, then answers questions by retrieving the most relevant chunks and generating from them — never from general knowledge. Off-topic questions get an explicit "nothing relevant found" instead of a guessed answer.
Two answer modes ship side by side: an extractive mode that highlights the most query-relevant sentences straight from retrieved chunks (no API key needed), and an optional Gemini-grounded mode that writes a cited answer strictly from the retrieved context. Running both against the same query makes it possible to tell whether a bad answer is a retrieval problem or a generation problem.
The app is evaluated, not just demoed. EVALUATION.md documents a 13-query retrieval test (92% top-3 hit rate, 0.92 mean reciprocal rank), six end-to-end generation traces graded for grounding and citation accuracy, and a reranking before/after comparison — including a retrieval miss the corpus deliberately keeps in as a demonstration, discussed below.
Highlights
- Retrieval-augmented Q&A over indexed course lecture PDFs — FAISS vector search plus optional cross-encoder reranking, with citations and similarity scores.
- Extractive mode needs no API key; an optional Gemini-grounded mode answers only from retrieved chunks, with an explicit "nothing relevant found" for off-topic queries.
Design Notes
A similarity floor before generating
Retrieval always returns its top-K nearest chunks, even for off-topic questions — cosine similarity has no built-in "nothing matched" signal. A tuned 0.25 cosine floor makes the app refuse instead of guessing: "best chocolate cake recipe" (0.14) and "who won the 2022 World Cup" (0.19) are correctly rejected without ever calling the LLM, while a genuine question scoring 0.29 clears the floor and gets a correct, cited answer.
Measured: 92% hit-rate, 0.92 MRR over 13 queries
rag/evaluate.py runs 13 hand-written queries against the full 16-document / 167-chunk corpus and checks whether the right source lands in the retrieved top-3 — 12 of 13 land there, most at rank 1 (PageRank 0.740, web crawling 0.719, BM25 0.709), despite noisy pypdf slide-text extraction the embeddings turned out to be robust to.
The one miss is real, and it's explained
"What advanced techniques go beyond basic RAG retrieval?" ranks the correct lecture chunk 4th (cosine 0.479), just outside top-3, because a leftover placeholder doc about RAG scores higher (0.607) for reading like clean, generic textbook prose. Noisy-but-correct slide text loses to fluent-but-generic text — kept in the corpus on purpose as a documented finding, not quietly cleaned up to inflate the score.
Reranking widened that exact gap instead of closing it
Cross-encoder reranking left the overall metrics unchanged (92% / 0.92) and made the one miss worse: the correct chunk's rerank score dropped to 0.163 while the generic placeholder rose to 0.925. ms-marco-MiniLM-L-6-v2 rewards fluent phrasing, and fragment-heavy slide text reads as less fluent even when it's the right answer — a genuine negative result, written up rather than hidden.
Exact search over approximate
At roughly 170 chunks, FAISS's brute-force IndexFlatIP is effectively free. An approximate index (IVF/HNSW) would only start paying off at a much larger corpus size.