RAG With LangChain: Build and Check a Source-Based Workflow

LangChain is a Python (and JavaScript) library that provides the building blocks for retrieval-augmented generation: document loaders, text splitters, embedding wrappers, vector store connectors, and retrieval chains. You assemble these components into a pipeline that loads your documents, indexes them, and retrieves relevant passages at query time.

A RAG pipeline answers questions from your documents. What it does not supply is the explicit project context that sits around those documents: the decisions about which sources to trust, the tasks that are open, and the follow-up work the last query created. fenbs keeps that record alongside the indexed material, so the team knows what the system is built on and what still needs doing.

What you need

• Python 3.9 or later.
• LangChain and its community packages: pip install langchain langchain-community langchain-openai.
• An OpenAI API key for the embedding model and the LLM, or an alternative provider.
• Chroma for local vector storage in this example: pip install chromadb.
• One or more source documents to index.

Load and split your documents

The first step is getting your documents into LangChain's document format and splitting them into chunks. Chunks of 500-1,000 characters with some overlap work well as a starting point:

from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = DirectoryLoader("./docs", glob="**/*.md", loader_cls=TextLoader)
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
)
chunks = splitter.split_documents(documents)

LangChain includes loaders for PDFs, CSVs, web pages, and many other formats in the langchain-community package. Each loaded document carries metadata — filename, source URL, page number — that will be available when a chunk is retrieved. Keep that metadata; it is how you verify sources.

Create embeddings and store them

Embed the chunks and store them in a local Chroma vector store:

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",
)
vectorstore.persist()

The persist directory saves the index to disk so you do not have to re-embed on every run. Use the same embedding model every time you add documents or query the store — mixing models in the same vector space produces poor retrieval results.

Build the retrieval chain

With the index in place, build a retrieval chain that retrieves relevant chunks and passes them to the model:

from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    return_source_documents=True,
)

result = qa_chain.invoke({"query": "What does the refund policy say?"})
print(result["result"])
for doc in result["source_documents"]:
    print(doc.metadata["source"])

Setting return_source_documents=True gives you the actual chunks the answer was based on. Print the source metadata from each to see which documents contributed. The LangChain documentation at python.langchain.com↗ (opens in a new tab) covers the full range of chain types and retriever options.

Verify your sources

Source verification is the step that most implementations skip. Retrieving source documents is not enough — you need to confirm that the retrieved text actually supports the answer, not just that it was deemed relevant by similarity score. For high-stakes uses (legal, medical, financial), a human should review the sources before an answer is shared externally.

Build a habit of checking a sample of answers against their sources. When you find a case where the answer is confidently wrong or where the retrieved chunk does not contain the information needed, that is a signal to adjust chunk size, retrieval depth, or the prompt template. Record these findings as lessons so future changes to the pipeline are informed by what failed.

One useful test is to ask the pipeline a question whose answer you already know, then check that the retrieved chunks actually contain the answer. If the correct chunk is not in the top four results, the retrieval settings need tuning. Adjust chunk size, overlap, or the number of results fetched, then test again. Keep the test questions in a file alongside the pipeline code so you can rerun them after any change.

Common issues and how to fix them

• **Wrong model at query time**: if you indexed with text-embedding-3-small but query with text-embedding-ada-002, results will be poor. The embedding model must match in both directions.
• **Chunks that lose context**: a paragraph that says "it was approved on Tuesday" makes no sense without knowing what "it" refers to. Increase chunk overlap or add a brief summary at the start of each chunk.
• **Outdated index**: if your source documents change frequently, rebuild or update the index on a schedule. A stale index retrieves outdated passages confidently.
• **Top-k too small**: if the relevant chunk is often in position 5-8 but you only fetch 4 results, raise the k parameter and check whether the answer quality improves.

How to do this with fenbs

1. Create a free account and add a project for the RAG application. Write the key decisions: which sources are authoritative, which model to use for embeddings, and any constraints on the answers the pipeline can give.
2. Add the source documents as Links in fenbs — the document name, its source URL or path, and its current status (authoritative, draft, deprecated).
3. Connect your AI app to fenbs over MCP from Integrations so it can read the source list and project decisions at the start of a session.
4. When source verification finds a bad answer, open a task in fenbs to investigate and fix it. Note what the query was, what the answer was, and what the correct answer should have been.
5. Record lessons — wrong chunk sizes, retrieval failures, sources that needed updating — as Lessons in fenbs so they inform the next pipeline iteration.

The what is RAG article explains the full RAG pipeline from first principles. The agent memory vs RAG article covers how retrieval fits into a broader memory architecture.