• Node.js (v18+)
• npm (v9+)

1. Clone the repository:

   git clone https://github.com/jitsmaster/web-crawler-mcp.git cd web-crawler-mcp

2. Install dependencies:

   npm install

3. Build the project:

   npm run build

Create a .env file with the following environment variables:

CRAWL_LINKS=false MAX_DEPTH=3 REQUEST_DELAY=1000 TIMEOUT=5000 MAX_CONCURRENT=5

Start the MCP server:

npm start

Add the following to your MCP settings file:

{ "mcpServers": { "web-crawler": { "command": "node", "args": ["/path/to/web-crawler/build/index.js"], "env": { "CRAWL_LINKS": "false", "MAX_DEPTH": "3", "REQUEST_DELAY": "1000", "TIMEOUT": "5000", "MAX_CONCURRENT": "5" } } } }

The server provides a crawl tool that can be accessed through MCP. Example usage:

{ "url": "https://example.com", "depth": 1 }