🚀 Complete self-hosted AI web scraping stack with ZERO external API costs!

This project combines powerful open-source tools to create a comprehensive web scraping and AI extraction system that runs entirely locally. No OpenAI, no Pinecone, no external APIs - just pure local AI power.

• Web Scraping: Extract content from any website
• AI Extraction: Structure data using local LLMs
• Semantic Search: Find similar content across scraped data
• Embedding Storage: Auto-generate and store vectors
• MCP Integration: Works with Claude Code and AI assistants

# 1. Clone this repository git clone https://github.com/Maheidem/zero-cost-ai-scraper.git cd zero-cost-ai-scraper # 2. Copy environment template cp .env.example .env # 3. Start everything docker-compose up -d # 4. Install Ollama models (takes 15-30 minutes) ./scripts/install-models.sh # 5. Verify setup ./scripts/health-check.sh

That's it! Your zero-cost AI scraper is ready.

• Firecrawl: Web scraping engine (builds from source with patches)
• Ollama: Local LLM processing (qwen3-coder:30b, nomic-embed-text)
• Qdrant: Vector database for embeddings
• SearXNG: Privacy-focused web search
• MCP Server: 6 tools for AI assistant integration

• ✅ Zero External API Costs
• ✅ Privacy-First (everything runs locally)
• ✅ MCP Compatible (works with Claude Code)
• ✅ Auto-Embeddings (builds searchable knowledge base)
• ✅ Semantic Search (find similar content)
• ✅ One-Command Setup

Compare this to cloud solutions that cost $50-200/month!

┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐ │ AI Assistant │───▶│ MCP Server │───▶│ Firecrawl │ │ (Claude Code) │ │ (6 tools) │ │ (Web Scraping) │ └─────────────────┘ └──────────────┘ └─────────────────┘ │ │ │ │ ┌──────────────┐ │ │ │ SearXNG │◀─────────────┘ │ │ (Web Search) │ │ └──────────────┘ ▼ │ ┌─────────────────┐ ┌──────────────┐ │ Ollama │ │ Qdrant │ │ qwen3-coder:30b │ │ (Vector DB) │ │ nomic-embed-text│ │ │ └─────────────────┘ └──────────────┘ 

import requests # Scrape and auto-store embeddings response = requests.post("http://localhost:3002/v1/scrape", json={ "url": "https://python.org/about", "formats": ["markdown"] })

# Extract structured data using local AI response = requests.post("http://localhost:3002/v1/extract", json={ "urls": ["https://news.ycombinator.com"], "prompt": "Extract top stories and their scores", "schema": { "stories": { "type": "array", "items": { "title": "string", "score": "number", "url": "string" } } } }) # Notice: "llmUsage": 0 (local processing!)

# Find similar content across all scraped data response = requests.post("http://localhost:3002/v1/similarity-search", json={ "query": "machine learning tutorials", "limit": 5 })

// Use any of the 6 MCP tools: // - firecrawl_scrape // - firecrawl_extract // - firecrawl_similarity_search // - firecrawl_search // - firecrawl_crawl // - firecrawl_map // Example: Extract data from Reddit mcp__firecrawl-local__firecrawl_extract({ "urls": ["https://reddit.com/r/programming"], "prompt": "Extract trending programming topics", "schema": {"topics": ["string"], "engagement": "number"} })

zero-cost-ai-scraper/ ├── mcp-server/ # MCP server (our code, MIT license) ├── searxng/ # Search engine configuration ├── configs/ # Service configurations ├── scripts/ # Setup and maintenance scripts ├── patches/ # Firecrawl modification patches ├── examples/ # Usage examples ├── docker-compose.yml # Complete stack definition └── docs/ # Comprehensive documentation 

• Original Projects (we build upon):

  • Firecrawl - Web scraping engine
  • SearXNG - Meta search engine
  • Ollama - Local LLM runtime
  • Qdrant - Vector database

• AI Assistants:

  • Claude Code (via MCP)
  • Any MCP-compatible tool
  • Direct API integration

1. Research Automation: Scrape academic papers, documentation, tutorials
2. Content Aggregation: Build knowledge bases from web sources
3. Competitive Intelligence: Monitor competitor websites and news
4. Documentation Extraction: Pull API docs, guides, examples
5. News Monitoring: Track industry news and developments

• No Data Leaves Your Machine: Everything runs locally
• No User Tracking: SearXNG doesn't store search queries
• Open Source: All code is auditable
• Self-Hosted: You control your data

• Docker & Docker Compose: For containerization
• 8GB+ RAM: For running Ollama models
• 50GB+ Disk: For model storage
• Internet: For initial setup and model download

This project builds on amazing open-source work:

• Firecrawl by Sideguide Technologies Inc. (AGPL v3)
• SearXNG by SearXNG team (AGPL v3)
• Our integrations are MIT licensed

See CONTRIBUTING.md for guidelines.

• Our Code: MIT License (MCP server, configurations, scripts)
• Dependencies: Various (see individual project licenses)
• Patches: AGPL v3 compatible modifications

If this helps you save money on AI APIs, please ⭐ star this repository!

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: See docs/ directory

Built with ❤️ for the open-source community

Making enterprise-level AI accessible to everyone, at zero cost.