HowdoyoubuildalocalRAGoveryourcodebase?

RAG for a codebase is worth building only when the alternative, letting the model search the repo itself, stops scaling. This page walks through both, then lays out a local pipeline where every component is named and runs on your own machine. Nothing below was benchmarked by us; numbers come from the papers, vendor posts and developer reports linked inline.

Do you even need RAG for a codebase?

Often you don’t: the teams behind several coding agents dropped the vector index in favor of letting the model run searches in a loop. Boris Cherny of the Claude Code team wrote on Hacker News that “Claude Code doesn’t use RAG currently. In our testing we found that agentic search out-performed RAG for the kinds of things people use Code for” (HN, Feb 2025). Nik Pash made a similar case in May 2025, citing Cline’s choice of agentic exploration over an index: chunked retrieval fills the context with fragments, while an agent that reads folders, files and imports explores code the way an engineer does (post).

The counter-evidence is just as public. Cursor, which ships both grep and semantic search, reported in November 2025 that adding semantic search raised answer accuracy by 12.5% on average across the models it tested (6.5% to 23.5% depending on the model). In an online A/B test, code retention rose 0.3% overall and 2.6% on codebases with 1,000+ files. Cursor’s conclusion is that “the combination of these two leads to the best outcomes” (Cursor blog). Its benchmark is internal, so treat the figures as vendor-reported.

A rough rule that fits both positions:

• Skip the index if the repo is small enough to grep in milliseconds, your model handles tool calls well, and queries name real identifiers.
• Build the index if the repo is large, people ask conceptual questions (“where do we handle retries?”) that share no words with the code, or your local model is too weak to plan a good multi-step search.
• Do both if you can. Hybrid setups are what most tools that kept indexing now ship.

The weaker your local model, the more an index helps, because agentic search leans on the model to guess good search terms. See local LLM coding agents for which models handle tool loops reliably.

How should you chunk code?

Chunk at syntax boundaries, so each chunk is a whole function, method or class, never a fixed number of lines. Line-based splitting cuts functions in half and glues unrelated code together, so the retriever returns fragments no model can use.

The best evidence is the cAST paper from Carnegie Mellon and Augment Code (EMNLP 2025 Findings). It parses files with tree-sitter, walks the syntax tree top-down, recursively splits any node that is too big, and greedily merges small sibling nodes so chunks stay dense. Size is measured in non-whitespace characters, not lines. Against fixed-size line chunking, it reports +4.3 points Recall@5 on RepoEval retrieval and +2.67 points Pass@1 on SWE-bench generation (arXiv 2506.15655). Concatenating the chunks reproduces the file exactly, so it drops into an existing pipeline.

Tree-sitter is the standard parser for this. It is fast, tolerates syntax errors, and updates its tree incrementally as files change, which matters when you re-index on save. Practical rules:

1. One chunk per function or method. Split oversized ones at inner blocks; merge tiny neighbors such as getters.
2. Prepend context. Add the file path and the enclosing class or module signature to each chunk’s text before embedding, so a method keeps track of where it lives.
3. Respect ignore files. Index what git ls-files returns and skip vendored code, lockfiles and build output.
4. Fall back gracefully. Split Markdown by heading and use line windows only for languages without a grammar. Roo Code does exactly this, with chunks of 100 to 1,000 characters (Roo Code docs).

One optional step: Greptile’s founders said at launch that they parse the AST, generate a docstring for each node with an LLM, and embed the docstring instead of the raw code (Launch HN). It helps conceptual queries, but it costs one LLM call per chunk at index time. On a local GPU, that is the slowest step in the pipeline.

Which embedding model and vector store should you use locally?

Use a code-aware embedding model served by Ollama or sentence-transformers, and a small embedded vector store unless you already run a database. Model choice has its own page, our best embedding model for code comparison, so this section only covers the wiring.

• Embeddings: Ollama exposes POST /api/embed on localhost:11434. It accepts an array of inputs and returns normalized vectors, typically 384 to 1,024 dimensions depending on the model (Ollama docs).
• Vector store: sqlite-vec is a dependency-free SQLite extension (MIT/Apache-2.0, still pre-v1), so your index is one file next to the repo. Qdrant in Docker is the common step up, and it is what Roo Code uses.
• Re-index incrementally. Hash each chunk and re-embed only those that changed. Stale indexes are one of the reasons agent teams cite for avoiding RAG.

How do you combine keyword and vector search?

Run both searches over the same chunks and merge the two ranked lists with reciprocal rank fusion (RRF). Code search needs both. Identifiers like retryWithBackoff are exact strings, and BM25 matches them precisely while embeddings blur them into “similar” functions. Natural-language questions share no tokens with the code, which is where vectors win. One HN commenter put the vocabulary-mismatch case plainly: grep returns nothing when the words you search for aren’t the words in the source (HN).

RRF scores each chunk as the sum of 1 / (k + rank) across the lists it appears in. It needs no score normalization and no training. Cormack, Clarke and Büttcher introduced it at SIGIR 2009, found that k = 60 was near-optimal, and showed that it beat both the individual systems and Condorcet fusion (paper). In SQLite you get BM25 from the built-in FTS5 extension and vectors from sqlite-vec, then fuse the two lists in a few lines of code.

After fusion, rerank. A cross-encoder reads the query and each candidate together and scores relevance directly, which is more accurate than comparing vectors but too slow to run over the whole index. Retrieve wide, rerank narrow: Continue’s now-deprecated indexer defaulted to 25 candidates and kept the top 5 after reranking (Continue docs). BAAI/bge-reranker-v2-m3 is a 0.6B, Apache-2.0 cross-encoder that runs on CPU or a small GPU through sentence-transformers. Five reranked chunks also keep you well under the context length a local model can actually use.

What does a minimal fully local pipeline look like?

A minimal local pipeline has six parts, and none of them call a cloud API.

The Python pieces install from PyPI (versions checked 2026-10-28: tree-sitter 0.26.0, sqlite-vec 0.1.9, sentence-transformers 6.2.0):

ollama pull embeddinggemma
pip install tree-sitter tree-sitter-python sqlite-vec sentence-transformers

Swap embeddinggemma for whichever code model you settle on, and add one tree-sitter-<language> grammar package per language in the repo. To let an agent call the index, wrap the query function as an MCP tool. Our Ollama MCP guide covers the server side.

Which existing tools do this for you?

Several open-source tools ship most of this pipeline, so build your own only if you need control over each stage.

Two caveats. claude-context’s 40% token figure is the vendor’s own evaluation (README), and Zilliz sells the managed vector database it recommends. Serena (GitHub) works at the symbol level through the language server protocol rather than through chunk retrieval. It gives the agent lookups like “find references”, which covers many of the structural questions people build RAG for.

What do developers report in practice?

Developers report that both approaches fail in predictable ways, which is why hybrids keep winning.

• Grep is smarter than it looks. One HN commenter who watched Claude Code and Codex work pointed out that the agents build complex grep queries with keywords that never appeared in the user’s question, so the “semantic gap” is partly closed by the model (HN).
• Grep breaks on vocabulary mismatch and scale. The reply in the same debate argued that grep returns nothing when the query and source use different words, and that local-filesystem speed doesn’t carry over to large corpora (HN).
• Indexes save guessing. A developer comparing tree-sitter-plus-embeddings MCP servers with Serena said the index cuts tool calls because the agent stops trying identifier names over and over (HN).
• Token-count chunking is the usual failure. The author of CocoSearch built it because most code RAG tools split on token or character limits and return broken half-functions (Show HN).
• Out of the box, terminal agents don’t embed. Another commenter noted that neither Claude Code nor Codex CLI does semantic search by default, while Cursor and Windsurf do (HN).

Research points the same way. The 2026 “Better Call Grep” paper found that a baseline where the LLM writes ripgrep commands performed comparably to graph-based retrieval for repository-level code completion. Its improved version adds identifier-weighted reranking and reports a 7–16% relative gain in exact match over the best baseline on CrossCodeEval (arXiv 2601.23254).

FAQ