Uploaded bySatwik Kansal
Getting started with Web Scraping in Python
The document explains web scraping as a method for extracting large volumes of data from websites into local files, emphasizing its utility for various applications. It details the three main steps of web scraping: getting content, parsing the response, and preserving the data, while outlining tools and libraries available like BeautifulSoup and Scrapy. Additionally, it addresses challenges, ethical considerations, and offers examples of practical applications, stressing the importance of conforming to a site's terms of use.
In this document
Powered by AI
Introduces the presentation topic: web scraping using Python by Satwik Kansal and Pradhvan Bisht.
Defines web scraping as a technique to extract data from websites for various uses.
Highlights reasons to scrape, including API limitations, reducing manual work, and fully accessing desired data.
Lists key skills that might be useful for web scraping: HTML, CSS, XPATH, and Regular Expressions.
Describes web scraping as a three-step process: content retrieval, response parsing, and data optimization.
Discusses methods for getting content using various Python modules and the complexities involved.
Explains two methods for data extraction: regular expressions and parsing libraries, citing popular libraries.
Compares popular parsing libraries: Beautiful Soup, LXML, Scrapy, and HTML5Lib.
Details options for preserving scraped data: writing files, exporting CSVs, or storing in databases.
Gives examples of scraping projects including Twitter data, StackOverflow posts, and university library details.
Advises on tools appropriate for various scraping challenges like dynamic HTML and authentication.
Shares hacks for overcoming common scraping challenges like captchas and performance improvements.
Details challenges faced while automating college library tasks and how Selenium is used as a solution.
Discusses ethical issues in scraping, including usage limits, copyright concerns, and legal protections.
Summarizes web scraping as a legal yet complex method for data acquisition.
Getting started with Web Scraping in Python
- 1. Scrapingtotherescue (Webscrapingusingpython) By : SatwikKansal and Pradhvan Bisht
- 2. Whatiswebscraping ? Web scrapingis a technique to extract large amounts of data from websites whereby the data is extracted and saved to a local file in your computer. The data can be used for several purposes like displaying on your own website and application, performing data analysis or for any other reason.
- 4. whyshouldyouscrape - API maynot provide what you need - No rate limit - Take what you really want! - Reduces manual effort - Swag!
- 5.
- 6. Howit’sdone? Broadly a ThreeStep Process 1. Getting the content (in most cases HTML) 2. Parsing the response. 3. Optimizing/Improving the performance and preserving the data
- 7. GETTINGTHECONTENT ● Using moduleslike urllib, urllib2, requests, mechanize and selenium. ● Involves GET/POST request to the server. ● The response contains the information to be extracted. ● Sometimes not as easy as it may seem.
- 8. ExtractingTheData 1. Using RegularExpression and Basic python Tricky, complex and kind of fragile. 2. Using Parsing Libraries ❏ Two different approaches possible -- Simple Parsing and Search Tree parsing. ❏ Some popular libraries are BeautifulSoup, Lxml, and html5lib. ❏ Each modules has its own techniques and thus its own pros and trade- offs
- 10.
- 11. PreservingTheData 1. Writing toa file. 2. Exporting as csv or excel file. 3. Storing in a database.
- 12. Examples Example 1 :Scraping Tweets from Twitter using BeautifulSoup and python’s Requests module Code Example 2 : Scraping top Stackoverflow posts using Scrapy Code Example 3 : Using Selenium to Log in and fetch library details from a university library site which uses Dynamic HTML.
- 14. WHATTOUSEWHERE 1. Handling dynamicallygenerated html Solutions: Selenium or Spidermonkey 2. Cookie based Authentication Solution : Requests module. 3. Simple scraping Solutions: BeautifulSoup+Requests, Scrapy, Selenium
- 16. Scrapinghacks 1. Overcoming captchas Lookuptables, One time manual entry , Death By Captchas (paid service) 2. Per IP address query limit Using tsocks, ssh_D and socks monkey. 3. Improving performance Multiprocessing , gevent and requests.async() method.
- 17. Example3 Automating My CollegeLibrary Problems : 1. Authentication 2. Dynamically Generated <iframe> tag Solution Selenium with headless Browser like PhantomJS Alternative: Mechanize Code
- 19. EthicsOfScraping Exceeding authorized useof the site Means doing anything that is prohibited in the Terms of Use (See CFAA, breach of contract, unjust enrichment, trespass to chattels, and various state laws similar to CFAA) Copyright Issues If the material you are scraping is not factual, but something that required some amount of creativity to create, you have copyright to worry about. QuickTip -- Conform to the the robots.txt file.
- 21. ● The brute-forceway to get the information required. ● Absolutely Legal ● Not always that easy.