Turn Any Documentation Site Into an AI Agent with LangGraph and Firecrawl

LLMs have training cutoffs. Frameworks don't. When you need current syntax for a library's latest version, an outdated LLM won't help. The solution? Build an agent that accesses live documentation instead, giving you accurate, up-to-date code examples every time.

We're using LangGraph for agent orchestration and Firecrawl for web scraping — one of the frameworks compared in the Best AI Agent Frameworks of 2026 roundup alongside Microsoft Agent Framework, CrewAI, Agno, Haystack, LlamaIndex, and the OpenAI Agent SDK. The result is a Streamlit app that turns any documentation site into an interactive assistant. This approach powers many AI agent projects that need intelligent web content interaction.

You'll build this from scratch, learning LangGraph's ReAct pattern for smart retrieval decisions, real-time token streaming for responsive UX, and Firecrawl integration for scraping clean markdown. The final application understands context and always cites its sources.

Before we start building, you need to understand what makes this system work. We're creating an agent that accesses live documentation, and the architecture underneath determines its intelligence.

Documentation assistants need RAG: retrieving relevant chunks from a vector database and feeding them to an LLM. Basic RAG stops after one retrieve-generate cycle, failing on complex queries that need multiple pieces of information.

Agentic RAG solves this. The agent searches, evaluates results, and searches again with refined queries if needed. It processes chunks across multiple tool calls until it has enough information, rather than stopping after a single retrieval.

Take the question "How do I configure batch scraping with custom headers?" A basic RAG system retrieves chunks about batch scraping and generates an answer from those chunks alone. An agent retrieves chunks about batch scraping, recognizes it needs header-specific information, searches again for "Firecrawl custom headers configuration," then synthesizes both searches into a complete answer. This iterative capability is why we're using agents instead of a basic RAG pipeline.

The agent follows ReAct (Reasoning + Acting): when you ask a question, it reasons about what it needs to do, takes an action like calling a search tool, observes the results, then reasons again about whether it has enough information. This loop continues until the agent can generate a complete answer.

LangGraph implements this through create_react_agent(). You define tools (like our documentation search) and write a system prompt explaining when to use them. LangGraph handles the reasoning loop automatically. The agent decides when to call tools, when to call them again, and when to stop and respond. The LangGraph startup validator tutorial shows how agents make complex decisions across multiple tool calls.

Our system has five main components. Firecrawl handles web scraping and converts documentation pages into clean markdown. The vector store (Chroma) holds embedded document chunks and handles similarity search. The LangGraph agent manages everything, deciding when to search and how to respond. OpenAI provides both the embeddings for semantic search and the LLM for generating answers. Streamlit wraps it all in a web interface.

Data flows through the system in two phases. During setup, Firecrawl's /map endpoint discovers documentation pages, batch scraping pulls content in parallel, the content gets split into chunks and converted to vectors via OpenAI embeddings, then Chroma stores them for fast retrieval. During conversation, user questions go to the LangGraph agent, which decides whether to call the search tool. When it does, the tool queries Chroma for relevant chunks and the agent synthesizes answers from the retrieved information.

Before building the agent, you need to set up your environment. This section walks you through the prerequisites, project structure, and installation steps.

Here's what you need before starting:

• Python 3.10 or higher: The code uses features that won't work on older versions
• Basic LLM knowledge: Understanding what embeddings are and how vector databases work for semantic search will help you follow along
• Environment variables: Familiarity with creating and using .env files to store API keys
• OpenAI API key: Get one from platform.openai.com for embeddings and chat completions
• Firecrawl API key: Get one from firecrawl.dev for web scraping

Both OpenAI and Firecrawl offer free tiers you can use to test the application.

docs-to-agent/
├── app.py                 # Main Streamlit application
├── src/
│   ├── crawler.py        # Firecrawl integration
│   ├── vectorstore.py    # Vector DB management
│   ├── tools.py          # Agent tools
│   └── agent.py          # LangGraph agent setup
├── requirements.txt      # Dependencies
├── .env                  # API keys (not committed)
└── .env.example         # Template

Create a requirements.txt file with these dependencies:

firecrawl-py>=0.0.16
langchain>=0.3.0
langchain-openai>=0.2.0
langgraph>=0.2.0
langchain-chroma>=0.1.0
streamlit>=1.28.0
python-dotenv>=1.0.0
chromadb>=0.4.22
tiktoken>=0.5.0

Install the dependencies:

pip install -r requirements.txt

Next, set up your API keys. Create a .env.example file as a template:

OPENAI_API_KEY=your-openai-key-here
FIRECRAWL_API_KEY=your-firecrawl-key-here

Copy it to .env and add your actual API keys:

cp .env.example .env
# Edit .env and add your real API keys

Note: The sections below break down each script into focused chunks to explain the implementation details. For the complete picture, we recommend opening the GitHub repository for this project in a separate tab while following along.

View the complete script: src/crawler.py

The first step in building our agent is getting the documentation content. Firecrawl handles this with two endpoints: /map for discovering URLs and batch_scrape for extracting content. We'll wrap both in a DocumentationCrawler class that lives in src/crawler.py.

Start by creating the file structure. Inside your project directory, create a src folder and add an __init__.py file to make it a Python package:

mkdir src
touch src/__init__.py

Now create src/crawler.py and add the imports:

"""
Firecrawl integration for discovering and crawling documentation sites.
"""
import os
from typing import List, Dict
from firecrawl import FirecrawlApp
from langchain_core.documents import Document

The imports bring in the Firecrawl client, LangChain's document structure, and Python's type hinting utilities for clear function signatures.

With the imports in place, add the class definition and initialization method:

class DocumentationCrawler:
    """Handles all Firecrawl interactions for documentation crawling."""
 
    def __init__(self):
        """Initialize the Firecrawl client."""
        api_key = os.getenv("FIRECRAWL_API_KEY")
        if not api_key:
            raise ValueError("FIRECRAWL_API_KEY not found in environment variables")
        self.app = FirecrawlApp(api_key=api_key)

The initialization loads the API key from environment variables and raises a clear error if missing, helping catch configuration problems early.

Firecrawl's /map endpoint discovers all URLs on a website without scraping content. It maps site structure in 2-5 seconds for 100 URLs, letting you filter to specific sections (API references, tutorials) before scraping.

Here's how to implement URL discovery. Add the discover_urls() method to your class:

    def discover_urls(self, base_url: str, limit: int = 100) -> List[str]:
        """
        Discover all URLs on a documentation site using the /map endpoint.
 
        Args:
            base_url: The base URL of the documentation site
            limit: Maximum number of URLs to discover
 
        Returns:
            List of discovered URLs
        """
        print(f"🔍 Discovering URLs from {base_url}...")
 
        try:
            response = self.app.map(base_url, limit=limit)
 
            # Extract URLs from response
            if hasattr(response, 'links'):
                urls = []
                for link in response.links:
                    if isinstance(link, str):
                        urls.append(link)
                    elif hasattr(link, 'url'):
                        urls.append(link.url)
                    else:
                        urls.append(link.get('url'))
            else:
                urls = response.get('links', [])
 
            print(f"✅ Found {len(urls)} URLs")
            return urls
 
        except Exception as e:
            print(f"❌ Error discovering URLs: {str(e)}")
            raise

The method handles multiple response formats (strings, objects, dictionaries) and logs errors before re-raising them for debugging.

The mastering the crawl endpoint guide explains how Firecrawl's crawling strategies work and when to use /map versus /crawl.

Firecrawl's batch_scrape() processes multiple URLs in parallel and returns clean markdown instead of raw HTML.

Add the crawl_documentation() method:

    def crawl_documentation(self, urls: List[str], max_pages: int = 50) -> List[Document]:
        """
        Crawl the provided URLs and return LangChain Document objects.
 
        Args:
            urls: List of URLs to crawl
            max_pages: Maximum number of pages to crawl
 
        Returns:
            List of LangChain Document objects
        """
        # Limit the number of URLs to crawl
        urls_to_crawl = urls[:max_pages]
        print(f"📄 Crawling {len(urls_to_crawl)} pages...")
 
        documents = []
 
        try:
            # Use batch scrape for efficiency
            result = self.app.batch_scrape(