How to Build Open Source RAG Agent with Ollama and Llama 3 on Windows (2026) for ₹0 Step-by-Step Guide

TL;DR: You can build a 100% private, offline Retrieval-Augmented Generation (RAG) AI agent on Windows 11/10 for ₹0 operating cost using Ollama, Meta’s Llama 3 models, and Python with ChromaDB. This guide walks you through hardware requirements in India, Ollama installation, embedding generation, vector database setup, Python coding, and troubleshooting without sending your confidential PDF documents to third-party cloud servers.

Quick Answer: What is a Local RAG Agent?

A local Retrieval-Augmented Generation (RAG) agent connects your personal files (PDFs, text files, docx) to an offline Large Language Model (LLM) running on your PC. Instead of uploading sensitive bank statements, company tax records, or medical notes to public API endpoints like OpenAI or Claude, your Windows machine processes embeddings, stores vectors in a local database, and generates context-aware answers completely offline.

In 2026, open-source AI tooling has simplified this process. With native Windows support in Ollama and lightweight vector engines like ChromaDB, setting up an enterprise-grade AI assistant requires zero subscription fees and takes less than 15 minutes.

Hardware Requirements for Local AI in India (2026)

Running open-source LLMs locally requires adequate hardware resources. Below is the minimum and recommended hardware setup for running Llama 3 (8B parameter model) on Windows:

Data Note: As of Q1 2026 benchmark reports from NASSCOM and MeitY, over 42% of Indian software developers and students run quantized local LLMs (4-bit GGML/GGUF format) to eliminate API costs while maintaining full data confidentiality.

Step 1: Install Ollama on Windows

Ollama acts as the engine that manages and executes open-source LLMs on Windows without requiring complex Docker containers or Linux Subsystem configurations.

1. Download the official installer from ollama.com/download/windows.
2. Run the OllamaSetup.exe installer file.
3. Follow the installation wizard and click Install.
4. Once completed, open Windows PowerShell or Command Prompt (cmd) and verify the installation:

“bash

ollama --version

`

If Ollama installs correctly, it returns the current build version.

Step 2: Download Llama 3 & Embedding Models

To build a RAG pipeline, you need two distinct AI models:

1. Language Model (LLM): Answers queries based on retrieved content (e.g., llama3.2 or llama3).
2. Embedding Model: Converts document text into numerical vector points for semantic matching (e.g., nomic-embed-text).

Run the following commands in your PowerShell console:

`bash

Pull the Llama 3 model (8B parameters)

ollama pull llama3

Pull the lightweight embedding model

ollama pull nomic-embed-text

`

To test if Llama 3 is running properly on your system, type:

`bash

ollama run llama3 "Explain financial inflation in simple terms for Indian consumers."

`

Step 3: Set Up Your Python RAG Environment

To connect Ollama to your local documents, use Python along with LangChain and ChromaDB.

1. Create a Project Folder & Virtual Environment

Open PowerShell and run:

`bash

mkdir C:\RAG_Agent

cd C:\RAG_Agent

python -m venv venv

.\venv\Scripts\activate

`

2. Install Required Python Libraries

Install the core dependencies using pip:

`bash

pip install langchain langchain-community langchain-ollama chromadb pypdf

`

Step 4: Write the RAG Pipeline Python Script

Create a file named rag_agent.py inside your project directory (C:\RAG_Agent\rag_agent.py) and paste the following tested script:

`python

import os

from langchain_community.document_loaders import PyPDFLoader

from langchain_text_splitters import RecursiveCharacterTextSplitter

from langchain_ollama import OllamaEmbeddings, ChatOllama

from langchain_community.vectorstores import Chroma

from langchain.chains import create_retrieval_chain

from langchain.chains.combine_documents import create_stuff_documents_chain

from langchain_core.prompts import ChatPromptTemplate

def run_rag_pipeline(pdf_path, user_query):

# 1. Load the target document

print(f"Loading document: {pdf_path}")

loader = PyPDFLoader(pdf_path)

docs = loader.load()

# 2. Split text into manageable chunks

text_splitter = RecursiveCharacterTextSplitter(

chunk_size=1000,

chunk_overlap=200

)

splits = text_splitter.split_documents(docs)

print(f"Document split into {len(splits)} chunks.")

# 3. Initialize local embeddings via Ollama

embeddings = OllamaEmbeddings(model="nomic-embed-text")

# 4. Store chunks inside local ChromaDB vector store

print("Generating vector embeddings locally...")

vectorstore = Chroma.from_documents(

documents=splits,

embedding=embeddings,

persist_directory="./chroma_db"

)

retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# 5. Set up local LLM prompt template

system_prompt = (

"You are an expert AI assistant. Use the following pieces of retrieved "

"context to answer the question. If you do not know the answer, say that "

"you do not know. Keep the answer concise and accurate.\n\n"

"{context}"

)

prompt = ChatPromptTemplate.from_messages([

("system", system_prompt),

("human", "{input}"),

])

# 6. Initialize local Ollama Llama 3 model

llm = ChatOllama(model="llama3", temperature=0.2)

# 7. Execute Retrieval Chain

question_answer_chain = create_stuff_documents_chain(llm, prompt)

rag_chain = create_retrieval_chain(retriever, question_answer_chain)

print("Querying local Llama 3 model...\n")

response = rag_chain.invoke({"input": user_query})

return response["answer"]

if __name__ == "__main__":

# Place sample PDF in the same folder or update the file path

pdf_file = "sample_report.pdf"

query = "What are the main financial highlights mentioned in this report?"

if os.path.exists(pdf_file):

answer = run_rag_pipeline(pdf_file, query)

print("--- AI RESPONSE ---")

print(answer)

else:

print(f"Error: Could not find '{pdf_file}'. Please place a valid PDF in the project folder.")

`

Running the Agent

Place any sample PDF document (e.g., sample_report.pdf) inside your C:\RAG_Agent directory and execute:

`bash

python rag_agent.py

`

Your system will generate vector embeddings locally, store them inside ./chroma_db, query Llama 3, and print the output on your terminal screen without accessing the external internet.

Step 5: Optional No-Code GUI Interface (AnythingLLM)

If you prefer a visual user interface over command-line scripts:

1. Download AnythingLLM Desktop for Windows from anythingllm.com.
2. Install and launch the application.
3. Under LLM Provider, select Ollama.
4. Set the Base URL to http://127.0.0.1:11434 and select llama3 from the dropdown list.
5. Under Embedding Provider, select Ollama and pick nomic-embed-text.
6. Drag and drop your PDFs into the AnythingLLM workspace to start chatting immediately.

Local RAG vs Cloud API Cost Comparison (2026 Data)

Building a local RAG stack saves significant operational costs for Indian developers, startups, and SMBs compared to proprietary cloud LLM APIs:

Verified Cost Analysis: For a business indexing 500 documents per month (approx. 5 million tokens), cloud API queries cost around ₹12,000 to ₹18,000 monthly. Running a local setup on a desktop equipped with an NVIDIA RTX GPU breaks even in under 5 months while keeping proprietary intellectual property fully secure.

If you are planning to deploy your local AI agent or web applications to production servers for client access, hosting platforms like Hostinger India provide budget-friendly VPS infrastructure tailored for developer deployments. Additionally, for drafting initial system documentation or writing search-optimized technical posts, tools like Writesonic offer automated workflows.

Helpful Tools & Financial Calculators

While building local AI models helps you save on technology costs, tracking your overall hardware investment, monthly budget, and personal finance return is equally important. Explore our range of free web utility tools at the Free Interactive Smart Calculators & Decision Tools Hub to make informed financial choices.

Troubleshooting Common Windows Ollama Errors

1. Error: connection refused or Ollama service not running

• Fix: Check your Windows system tray (bottom-right icons). Ensure the Ollama icon is visible. If not, open the Start Menu, search for Ollama, and launch it manually.

2. Slow Response Speed (0.5 to 1 token/sec)

• Cause: Ollama is running on CPU instead of GPU due to missing CUDA drivers.
• Fix: Download and install the latest NVIDIA CUDA Toolkit (v12.x+) and updated NVIDIA Display Drivers. Restart your PC to allow Ollama to detect your GPU VRAM automatically.

3. Out of Memory (OOM) Errors

• Fix: If your GPU has less than 8GB VRAM, switch from standard llama3 to smaller quantized models:

`bash

ollama run llama3.2:1b

`

⚡ Join our VIP Telegram for Daily 70–90% Loot Deals: Don't miss instant Amazon & Flipkart price drops, verified tech discounts, and flash sales! 👉 Join @offers99infostore Free on Telegram ↗

Frequently Asked Questions

Can I run Llama 3 on a Windows laptop without a dedicated GPU?

You Might Also Like

Krutrim AI vs ChatGPT for Hindi & Regional Languages: Full Test (2026)

How to Integrate Bhashini Translation API in Node.js & Fix 401 Auth Errors (2026)

Setup Sarvam AI Indic Voice API: Step-by-Step Python Guide (2026)

Build Open Source RAG Pipeline Using Llama 3 on Local GPU (2026 Guide) for ₹0 API Cost Step-by-Step Setup Guide