How to Build a Custom RAG Pipeline: LangChain & Pinecone

Standard Large Language Models (LLMs) hallucinate when asked about internal company documents, recent events, or specialized domain data. Retrieval-Augmented Generation (RAG) fixes this by injecting relevant, dynamic knowledge directly into the LLM prompt at query time, cutting hallucination rates substantially in production enterprise applications.

Building a basic prototype takes twenty lines of code. Building a production-ready custom RAG pipeline that handles complex PDF layouts, maintains sub-500ms latency, and returns precise citations requires deliberate architectural choices.

In this technical guide, you will learn how to design, code, and optimize a production-grade custom RAG pipeline using LangChain as the orchestration framework and Pinecone as the high-performance vector database.

RAG Architecture Overview: How the Data Flows

To understand why specific chunking and vector index decisions matter, you first need a clear breakdown of the end-to-end RAG lifecycle. The pipeline operates in two distinct phases: Ingestion and Retrieval & Generation.

1. The Data Ingestion Phase

Before any user types a prompt, raw unstructured data (PDFs, Markdown, database dumps, Notion pages) must be converted into a searchable vector index:

1. Document Loading: Raw files are loaded into memory as standardized Document objects containing raw text and key-value metadata (such as source file name, page number, and creation date).
2. Text Chunking: Long documents are parsed and split into manageable, overlapping passages. Smaller blocks preserve semantic focus, while overlapping ensures critical context isn't severed at a split boundary.
3. Embedding Generation: Each text chunk passes through an embedding model (such as OpenAI text-embedding-3-small or Hugging Face bge-large-en), transforming raw text into high-dimensional floating-point vectors.
4. Vector Upserting: Vectors, along with their text content and metadata payload, are batch-uploaded to a Pinecone vector index.

2. The Retrieval & Generation Phase

When an end user submits a natural language question:

1. Query Embedding: The incoming user query passes through the identical embedding model used during ingestion.
2. Vector Similarity Search: Pinecone compares the query vector against stored document vectors using similarity metrics like Cosine Similarity or Dot Product, returning the Top-K closest chunks.
3. Prompt Augmentation: A structured prompt template combines the user's original question with the retrieved text chunks serving as explicit ground truth.
4. LLM Generation: The augmented prompt passes to the LLM (for instance, gpt-4o or claude-3-5-sonnet), which synthesizes an accurate answer grounded strictly in the provided context.

Prerequisites and Environment Setup

To follow along with this implementation, ensure you have Python 3.10+ installed along with API keys for OpenAI (or your preferred LLM provider) and Pinecone.

Required Dependencies

Install the necessary libraries via pip:

bash pip install langchain langchain-openai langchain-community pinecone-client python-dotenv pypdf tiktoken

Managing API Keys

Create a .env file in your root project directory to store environment credentials securely:

env OPENAI_API_KEY=sk-proj-your-openai-api-key PINECONE_API_KEY=pcsk_your_pinecone_api_key PINECONE_INDEX_NAME=custom-rag-index

Initialize environment variables at the top of your Python execution script:

python import os from dotenv import load_dotenv

load_dotenv()

assert os.getenv("OPENAI_API_KEY"), "OPENAI_API_KEY missing from environment." assert os.getenv("PINECONE_API_KEY"), "PINECONE_API_KEY missing from environment."

Setting Up Your Pinecone Vector Database Index

Pinecone provides serverless vector indexes that scale seamlessly without requiring manual cluster provisioning. Choosing the right distance metric and dimension count is critical.

Choosing Distance Metrics

• Cosine Similarity: Best when vector lengths vary widely (for example, raw text passages of varying lengths). Measures the angle between vectors.
• Dot Product: High speed; optimal if your embedding model normalizes vectors to unit length (OpenAI embeddings are normalized by default).
• Euclidean Distance (L2): Measures straight-line distance; common in image recognition or fixed-dimensional spatial embeddings.

For OpenAI text-embedding-3-small, use 1536 dimensions with cosine similarity.

Provisioning the Index Programmatically

Using the modern pinecone-client (v3+), initialize and create the index programmatically:

python import os from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key=os.getenv("PINECONE_API_KEY")) index_name = os.getenv("PINECONE_INDEX_NAME", "custom-rag-index")

Check if index already exists to prevent duplicate creation

existing_indexes = [index.name for index in pc.list_indexes()]

if index_name not in existing_indexes: pc.create_index( name=index_name, dimension=1536, # OpenAI text-embedding-3-small dimension metric="cosine", spec=ServerlessSpec( cloud="aws", region="us-east-1" ) ) print(f"Created index: {index_name}") else: print(f"Index {index_name} already exists.")

Step-by-Step Implementation: Building the RAG Pipeline

Now, let's build a clean, modular Python pipeline using LangChain core primitives.

Step 1: Document Loading

LangChain offers document loaders for over 100 data sources. For this pipeline, we will ingest enterprise documentation from PDF files and Markdown directories.

python from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader

def load_documents(directory_path: str): """Loads all PDFs from a given directory.""" loader = DirectoryLoader( directory_path, glob="**/*.pdf", loader_cls=PyPDFLoader ) documents = loader.load() print(f"Loaded {len(documents)} document pages.") return documents

Step 2: Advanced Text Chunking

Naive chunking (splitting strictly every 500 characters, for example) cuts sentences in half, causing loss of contextual meaning. The RecursiveCharacterTextSplitter intelligently splits on paragraphs, headings, and punctuation boundaries before falling back to character limits.

python from langchain_text_splitters import RecursiveCharacterTextSplitter

def chunk_documents(documents): """Splits documents into optimized token-aware chunks.""" text_splitter = RecursiveCharacterTextSplitter( chunk_size=800, # Targets ~150-200 words per chunk chunk_overlap=150, # Ensures sentence continuity across boundaries length_function=len, separators=["\n\n", "\n", " ", ""] # Tries natural paragraph breaks first ) chunks = text_splitter.split_documents(documents) print(f"Generated {len(chunks)} text chunks.") return chunks

Step 3: Embeddings & Batch Upserting to Pinecone

Connecting our tokenized text chunks to OpenAI's embedding API and storing them in Pinecone:

python from langchain_openai import OpenAIEmbeddings from langchain_pinecone import PineconeVectorStore

def populate_vector_store(chunks, index_name: str): """Embeds and upserts document chunks into Pinecone.""" embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

Upsert chunks directly using LangChain's vectorstore integration

vectorstore = PineconeVectorStore.from_documents( documents=chunks, embedding=embeddings, index_name=index_name ) print("Successfully indexed chunks into Pinecone.") return vectorstore

Step 4: Constructing the Retrieval Chain

With the vector store populated, construct an LCEL (LangChain Expression Language) retrieval chain that takes user queries, fetches relevant chunks, and feeds them into an LLM.

python from langchain_openai import ChatOpenAI from langchain_core.prompts import ChatPromptTemplate from langchain_core.runnables import RunnablePassthrough from langchain_core.output_parsers import StrOutputParser

def build_rag_chain(vectorstore): """Builds a production RAG chain using LCEL."""

Create retriever with search kwargs

retriever = vectorstore.as_retriever( search_type="similarity", search_kwargs={"k": 4} # Retrieve top 4 most relevant chunks )

Define prompt template with explicit context grounding instructions

system_prompt = ( "You are an expert technical assistant for enterprise documentation.\n" "Use the following pieces of retrieved context to answer the question.\n" "If you do not know the answer based strictly on the context, state that you do not know.\n" "Do not make up facts or draw from external knowledge not present in the context.\n\n" "Context:\n{context}" )

prompt = ChatPromptTemplate.from_messages([ ("system", system_prompt), ("human", "{question}") ])

LLM definition

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.0)

Helper to format retrieved document passages into a single text block

def format_docs(docs): return "\n\n".join([f"[Source: {d.metadata.get('source', 'Unknown')}]\n{d.page_content}" for d in docs])

LCEL Pipeline

rag_chain = ( {"context": retriever | format_docs, "question": RunnablePassthrough()} | prompt | llm | StrOutputParser() )

return rag_chain

Step 5: Executing the Pipeline

Here is how you execute the full pipeline in practice:

python

Load and process data

raw_docs = load_documents("./docs_folder") doc_chunks = chunk_documents(raw_docs)

Initialize vector store connection

embeddings = OpenAIEmbeddings(model="text-embedding-3-small") vectorstore = PineconeVectorStore(index_name=index_name, embedding=embeddings)

Upsert data (run only on initial ingest or document update)

populate_vector_store(doc_chunks, index_name)

Query the chain

rag_chain = build_rag_chain(vectorstore) query = "What are the mandatory compliance protocols for data encryption at rest?" response = rag_chain.invoke(query)

print("\n--- RAG Response ---") print(response)

Comparing Chunking Strategies

Choosing the right chunking strategy directly dictates whether your model receives clean signal or confusing noise. Here is how standard chunking strategies compare for common enterprise document types:

Advanced RAG Techniques for Production

Basic RAG works reasonably well for simple Q&A, but real-world production data introduces noise, complex queries, and ambiguous phrasing. Implementing these advanced patterns upgrades a basic script into an enterprise system.

1. Metadata Filtering

Pinecone supports robust metadata filtering alongside vector similarity searches. By tagging vectors with attributes like department, security_level, or document_type, you eliminate irrelevant search spaces instantly.

python

Querying Pinecone with specific metadata filters

filtered_retriever = vectorstore.as_retriever( search_kwargs={ "k": 5, "filter": { "department": {"$eq": "finance"}, "year": {"$gte": 2024} } } )

2. Hybrid Search (Dense + Sparse Retrieval)

Dense embeddings (like OpenAI) excel at capturing broad semantic concepts but struggle with exact keyword matching, such as specialized part numbers, error codes, or personal names. Hybrid search pairs dense vectors with sparse keyword representations (like BM25 or Pinecone's sparse vectors) to achieve stronger retrieval performance across both semantic and exact keyword queries.

3. Reranking (Cohere Rerank)

Vector distance alone does not guarantee that the top 10 retrieved chunks are the most useful. Passing the top 20 retrieved candidates through a dedicated cross-encoder reranker (such as CohereRerank) re-orders the results based on true relevance before sending them to the LLM context window.

python from langchain.retrievers import ContextualCompressionRetriever from langchain_cohere import CohereRerank

Set up Cohere Cross-Encoder Reranker

compressor = CohereRerank(model="rerank-english-v3.0", top_n=4) compression_retriever = ContextualCompressionRetriever( base_compressor=compressor, base_retriever=vectorstore.as_retriever(search_kwargs={"k": 20}) )

4. Contextual Compression

Instead of dumping full 800-token chunks into the prompt, contextual compression extracts only the specific sentences within each chunk that directly address the user's query, drastically reducing token costs and preventing context distraction in long prompts.

Avoid These Common RAG Pitfalls

1. Over-chunking or Under-chunking: Setting chunk sizes below 200 tokens isolates words from their necessary context. Setting chunk sizes above 2,000 tokens dilutes the vector embedding, making specific fact lookup virtually impossible.
2. Ignoring Metadata: Storing raw text without attaching source URLs, document titles, or page numbers makes it impossible to show accurate citations back to end users.
3. Relying Solely on Semantic Distance: Assuming vector cosine distance equals logical answer relevance is a mistake. Always combine vector search with reranking or metadata pre-filtering for mission-critical applications.
4. Unconstrained LLM Prompts: Failing to instruct the model to explicitly say "I don't know" when retrieved context lacks sufficient detail guarantees hallucinated outputs.

Evaluating Your Pipeline with RAGAS

You cannot improve what you do not systematically measure. Evaluating RAG applications relying solely on manual inspection fails at scale. RAGAS (Retrieval Augmented Generation Assessment) is a standard framework for component-level evaluation.

RAGAS measures four core metrics:

• Faithfulness: Measures whether the generated answer relies only on the retrieved context (detects hallucinations).
• Answer Relevance: Measures whether the output directly answers the user's query without off-topic details.
• Context Precision: Evaluates whether the top-ranked retrieved chunks contain relevant information without noise.
• Context Recall: Evaluates whether the retriever fetched all necessary pieces of ground-truth information required to answer the query.

Automating RAGAS tests in your CI/CD pipeline ensures that prompt tweaks or embedding updates never introduce silent regressions into your production application.

Estimated Cost Breakdown for Production Deployment

Understanding operational expenses prevents unexpected monthly cloud API billing surprises. Below is an estimated cost model for an enterprise pipeline processing 100,000 queries per month across a repository of 10,000 documents (~50,000 text chunks):

Final Recommendations and Next Steps

Building a custom RAG pipeline using LangChain and Pinecone gives your application precise, secure access to your organization's internal data. Start by implementing a recursive chunking strategy, index your documents in Pinecone with explicit metadata, and use structured LCEL prompt templates to prevent model hallucinations.

As your query volume scales, layer in hybrid search and Cohere reranking to maintain high precision even across millions of vectors.

Evaluating modern software components and AI tools can be overwhelming. At Saasbonus, we provide independent, hands-on software reviews and technical breakdowns to help you choose the right developer tooling, AI platforms, and cloud infrastructure the first time.