The web data platform for AI agents — fetch, search, crawl, extract, monitor, screenshot, and research any URL.
npx webpeel "https://example.com" # Clean markdown npx webpeel search "AI trends 2025" # Web search npx webpeel crawl docs.example.com # Crawl entire siteGet your free API key → · No credit card required · 500 requests/week free
- 65–98% token savings — domain-specific extractors strip boilerplate, ads, and nav before content reaches your agent
- 29 domain extractors — purpose-built parsers for Reddit, Wikipedia, GitHub, Hacker News, YouTube, ArXiv, Amazon, and 22 more
- Zero-config Cloudflare bypass — 4-layer escalation stack handles TLS fingerprinting, edge proxying, and cache fallback automatically
| Feature | Command / API |
|---|---|
| Fetch any URL | webpeel "url" |
| Web search | webpeel search "query" |
| Crawl sites | webpeel crawl "url" --max-pages 50 |
| Screenshots | webpeel screenshot "url" |
| Monitor changes | webpeel monitor "url" --interval 300 |
| Browser actions | --action 'click:.btn,wait:2000' |
| YouTube transcripts | auto-detected |
| PDF extraction | auto-detected |
| MCP server | webpeel mcp |
| Schema extraction | POST /v1/fetch with extract.schema |
| Research agent | POST /v1/agent |
| Smart search | POST /v1/search/smart |
Give Claude, Cursor, or any MCP-compatible agent the ability to browse the web in one config change.
Claude Desktop (~/.claude/claude_desktop_config.json):
{ "mcpServers": { "webpeel": { "command": "npx", "args": ["-y", "webpeel", "mcp"], "env": { "WEBPEEL_API_KEY": "wp_your_key_here" } } } }Cursor / VS Code (.cursor/mcp.json or .vscode/mcp.json):
{ "mcpServers": { "webpeel": { "command": "npx", "args": ["-y", "webpeel", "mcp"], "env": { "WEBPEEL_API_KEY": "wp_your_key_here" } } } }Available MCP tools: webpeel_read, webpeel_find, webpeel_see, webpeel_extract, webpeel_monitor, webpeel_act, webpeel_crawl
# Fetch any page — returns clean markdown + metadata curl "https://api.webpeel.dev/v1/fetch?url=https://stripe.com/pricing" \ -H "Authorization: Bearer $WEBPEEL_API_KEY"{ "url": "https://stripe.com/pricing", "markdown": "# Stripe Pricing\n\n**Integrated per-transaction fees**...", "metadata": { "title": "Pricing & Fees | Stripe", "tokens": 420, "tokensOriginal": 8200, "savingsPct": 94.9 } }WebPeel's 29 domain-specific extractors strip navigation, ads, sidebars, and boilerplate before sending content to your agent.
| Site type | Raw HTML tokens | WebPeel tokens | Savings |
|---|---|---|---|
| News article | 18,000 | 640 | 96% |
| Reddit thread | 24,000 | 890 | 96% |
| Wikipedia page | 31,000 | 2,100 | 93% |
| GitHub README | 5,200 | 1,800 | 65% |
| E-commerce product | 14,000 | 310 | 98% |
Less context used = lower costs + faster inference + longer agent chains.
WebPeel is built with security-first principles:
- Helmet.js headers — HSTS, X-Frame-Options, nosniff, XSS protection on all responses
- Webhook signing — HMAC-SHA256 signatures on all outbound webhooks
- Audit logging — every API call logged with IP, key, and action
- GDPR compliant —
DELETE /v1/accountfor full data erasure - SSH hardened — Fail2Ban, MaxAuthTries, key-only auth on all infrastructure
- 📖 Documentation — Guides, references, and examples
- 💰 Pricing — Plans and limits
- 📝 Blog — Tutorials, comparisons, and use cases
- 📊 Status — Uptime and incidents
- 🔒 Security — Security policy and disclosure
- 📋 SLA — Uptime commitments
Pull requests welcome! Please open an issue first to discuss major changes.
- Fork the repo
- Create your feature branch (
git checkout -b feat/my-feature) - Commit your changes (
git commit -m 'feat: add my feature') - Push to the branch (
git push origin feat/my-feature) - Open a Pull Request
WebPeel SDK License — free for personal and commercial use with attribution. See LICENSE for full terms.