Scrapy, a fast high-level web crawling & scraping framework for Python.
- Updated
Nov 17, 2025 - Python
#
Crawler
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).
Here are 3,580 public repositories matching this topic...
Python scraper based on AI
markdown crawler web-crawler scraping web-scraper web-scraping data-extraction webscraping web-data-extraction web-search ai-search rag automated-scraper web-data scraping-python web-crawlers llm ai-crawler large-language-model ai-scraping
- Updated
Nov 21, 2025 - Python
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
python api crawler scraper spider async web-scraping douyin tiktok fastapi tiktok-scraper tiktok-api douyin-api pywebio tiktok-signature no-watermark online-parsing douyin-tiktok-api douyin-tiktok-download douyin-scraper
- Updated
Oct 12, 2025 - Python
Incredibly fast crawler designed for OSINT.
- Updated
Mar 31, 2025 - Python
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
python crawler data automation ai mcp scraping crawling web-scraper web-scraping selectors xpath data-extraction stealth webscraping crawling-python playwright web-scraping-python ai-scraping mcp-server
- Updated
Nov 21, 2025 - Python
Use LLMs to track and extract websites, RSS feeds, and social media
- Updated
Nov 5, 2025 - Python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright
- Updated
Nov 21, 2025 - Python
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation
- Updated
Jun 9, 2025 - Python
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
testing crawler automation webdriver headless scraping selenium chromium e2e-tests web-scraping testing-tools fingerprinting cdp browser-automation puppeteer recaptcha-v3 anti-detection playwright turnstile-solver
- Updated
Nov 19, 2025 - Python
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
- Updated
Dec 26, 2022 - Python
实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:
crawler python3 boss scrapy wechat baidu lagou douban-movie baidu-tieba xianyu douban-music ctrip zhilianzhaopin sohu taobao-spider fofa dazhong-spider alitask baotu quanjing
- Updated
May 22, 2024 - Python
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
nlp crawler text-mining html-to-markdown scraping news-aggregator text-extraction web-scraping rss-feed readability tei html2text news-crawler corpus-builder corpus-tools article-extractor text-cleaning rag text-preprocessing llm
- Updated
Sep 12, 2025 - Python
Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀
- Updated
Nov 9, 2025 - Python
- Followers
- 526 followers
- Website
- github.com/topics/crawler
- Wikipedia
- Wikipedia