Local RAG With Ollama and a Vector DB: A Self-Contained Pipeline

RAG sends pieces of your data to a model. If those pieces are sensitive, the cloud route raises questions. A fully local RAG stack — local embedding model, local vector DB, local generation model — keeps the entire pipeline on the same box. The architecture is exactly the same as cloud RAG; the addresses just point to localhost.

The four components

1. 1A loader: turns your documents into chunks of text
2. 2An embedding model: turns each chunk into a vector (Ollama can serve nomic-embed-text, mxbai-embed-large, or similar)
3. 3A vector database: stores chunks + vectors for nearest-neighbor lookup (Chroma, Qdrant, LanceDB all run locally)
4. 4A generation model: the chat model Ollama already runs, prompted with retrieved context

from langchain_ollama import OllamaEmbeddings, ChatOllama
from langchain_community.vectorstores import Chroma

embeddings = OllamaEmbeddings(model="nomic-embed-text")
store = Chroma.from_texts(chunks, embeddings, persist_directory="./db")

llm = ChatOllama(model="llama3.1:8b")

def ask(question):
    docs = store.similarity_search(question, k=5)
    context = "\n\n".join(d.page_content for d in docs)
    prompt = f"Use ONLY this context:\n{context}\n\nQuestion: {question}"
    return llm.invoke(prompt)

What gets harder when local

• Embedding throughput: a CPU-only embedder is slow on a million-document corpus
• Index size: a vector DB stores roughly 4-12 KB per chunk — millions of chunks add up
• Quality: local embedding models are decent but the gap to top cloud embeddings is real
• Updating: re-embedding the corpus when you change models takes hours

Apply this

• Index a folder of your own documents using Ollama's embedding model and Chroma
• Wire the retriever to a local Ollama chat model and ask three real questions
• Compare the answers to those of a cloud RAG running on the same documents

The big idea: a usable RAG pipeline can live entirely on one machine. Decide which legs need to be local based on data sensitivity, not architecture purity.