A scalable, open-source webcrawler that writes website data to file while crawling each new webpage
$ git clone https://github.com/Boomslet/Web_Crawler $ python setup.py install %Run controller.py >>> crawl('https://github.com/')Successfully crawled https://github.com/ Successfully crawled https://github.com/#start-of-content Successfully crawled https://github.com/features Successfully crawled https://github.com/business Successfully crawled https://github.com/pricing Successfully crawled https://github.com/dashboard