Skip to content

webpeel/webpeel

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1,127 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

WebPeel — Web data API for AI agents

npm version npm downloads GitHub stars License CI

The web data platform for AI agents — fetch, search, crawl, extract, monitor, screenshot, and research any URL.


Quick Start

npx webpeel "https://example.com" # Clean markdown npx webpeel search "AI trends 2025" # Web search npx webpeel crawl docs.example.com # Crawl entire site

Get your free API key → · No credit card required · 500 requests/week free


Why WebPeel

  • 65–98% token savings — domain-specific extractors strip boilerplate, ads, and nav before content reaches your agent
  • 29 domain extractors — purpose-built parsers for Reddit, Wikipedia, GitHub, Hacker News, YouTube, ArXiv, Amazon, and 22 more
  • Zero-config Cloudflare bypass — 4-layer escalation stack handles TLS fingerprinting, edge proxying, and cache fallback automatically

Features

Feature Command / API
Fetch any URL webpeel "url"
Web search webpeel search "query"
Crawl sites webpeel crawl "url" --max-pages 50
Screenshots webpeel screenshot "url"
Monitor changes webpeel monitor "url" --interval 300
Browser actions --action 'click:.btn,wait:2000'
YouTube transcripts auto-detected
PDF extraction auto-detected
MCP server webpeel mcp
Schema extraction POST /v1/fetch with extract.schema
Research agent POST /v1/agent
Smart search POST /v1/search/smart

MCP Integration

Give Claude, Cursor, or any MCP-compatible agent the ability to browse the web in one config change.

Claude Desktop (~/.claude/claude_desktop_config.json):

{ "mcpServers": { "webpeel": { "command": "npx", "args": ["-y", "webpeel", "mcp"], "env": { "WEBPEEL_API_KEY": "wp_your_key_here" } } } }

Cursor / VS Code (.cursor/mcp.json or .vscode/mcp.json):

{ "mcpServers": { "webpeel": { "command": "npx", "args": ["-y", "webpeel", "mcp"], "env": { "WEBPEEL_API_KEY": "wp_your_key_here" } } } }

Available MCP tools: webpeel_read, webpeel_find, webpeel_see, webpeel_extract, webpeel_monitor, webpeel_act, webpeel_crawl

Full MCP setup guide →


API Example

# Fetch any page — returns clean markdown + metadata curl "https://api.webpeel.dev/v1/fetch?url=https://stripe.com/pricing" \ -H "Authorization: Bearer $WEBPEEL_API_KEY"
{ "url": "https://stripe.com/pricing", "markdown": "# Stripe Pricing\n\n**Integrated per-transaction fees**...", "metadata": { "title": "Pricing & Fees | Stripe", "tokens": 420, "tokensOriginal": 8200, "savingsPct": 94.9 } }

Full API reference →


Token Efficiency

WebPeel's 29 domain-specific extractors strip navigation, ads, sidebars, and boilerplate before sending content to your agent.

Site type Raw HTML tokens WebPeel tokens Savings
News article 18,000 640 96%
Reddit thread 24,000 890 96%
Wikipedia page 31,000 2,100 93%
GitHub README 5,200 1,800 65%
E-commerce product 14,000 310 98%

Less context used = lower costs + faster inference + longer agent chains.


Security

WebPeel is built with security-first principles:

  • Helmet.js headers — HSTS, X-Frame-Options, nosniff, XSS protection on all responses
  • Webhook signing — HMAC-SHA256 signatures on all outbound webhooks
  • Audit logging — every API call logged with IP, key, and action
  • GDPR compliantDELETE /v1/account for full data erasure
  • SSH hardened — Fail2Ban, MaxAuthTries, key-only auth on all infrastructure

Security policy →


Links

  • 📖 Documentation — Guides, references, and examples
  • 💰 Pricing — Plans and limits
  • 📝 Blog — Tutorials, comparisons, and use cases
  • 📊 Status — Uptime and incidents
  • 🔒 Security — Security policy and disclosure
  • 📋 SLA — Uptime commitments

Contributing

Pull requests welcome! Please open an issue first to discuss major changes.

  1. Fork the repo
  2. Create your feature branch (git checkout -b feat/my-feature)
  3. Commit your changes (git commit -m 'feat: add my feature')
  4. Push to the branch (git push origin feat/my-feature)
  5. Open a Pull Request

License

WebPeel SDK License — free for personal and commercial use with attribution. See LICENSE for full terms.

Get started free →

About

The web data platform for AI agents. Fetch, search, crawl, extract, monitor, screenshot — one API. 29 domain extractors, 65-98% token savings, MCP server included.

Topics

Resources

License

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Sponsor this project

Packages

 
 
 

Contributors