-1
$\begingroup$

I have just recently started creating crawlers to scrape data. As I was creating a script yesterday, a question popped in my head that I have still not been able to find online :

Is there a difference in using different web browsers(Chrome vs Firefox vs Opera, etc.) when scraping data, or automating a task (for instance, automating clicking through a set of links that ultimately lead to downloading a file) ?

Again, I'm fairly new to scraping and automation so I just want to get a sense of the best way to go about doing such things.

Not sure if this matters, but I'm utilizing python.

Any resources/reading material would also be greatly appreciated!

Improve this question
$\endgroup$
2
  • $\begingroup$ I don't think it this a fit for data science. At least you have a useful answer I hope, but now voting to close. $\endgroup$ Commented Mar 2, 2017 at 16:30
  • $\begingroup$ I sure did. I apologize for posting it in not the best fitting "community". I am a bit of novel to the stack community--I'll be more careful with where I post my questions. Thanks! $\endgroup$ Commented Mar 2, 2017 at 16:41

1 Answer 1

Reset to default
3
$\begingroup$

If you are using a scraping tool that manages a web session for you, you should be able to get most things done using a scraper. You should be able to log in to sites and services and download the content that you are interested in.

The biggest difference between scraping directly in Python using BeautifulSoup or something similar is that the basic Python web scrapers will not execute active content on a page. That is, if a page uses Javascript to load some dynamic content and then display it, this content will not be available to you from your scraper.

This may not matter for the sites that you are working with, but more and more sites are relying on Javascript to function, so you may find that you eventually need a scraper that can run Javascript.

If you do need to execute Javascript from your scraper, you'll want use full headless browser to do your scraping. Some examples are:

In terms of actual differences between browsers, there should be minimal differences at this point unless you are dealing with fairly cutting edge features (real time communication, live streaming, etc).

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank you! I had an idea of the dynamic contents that JavaScript introduces when scraping, but this def game me more insight. Much appreciated. $\endgroup$ Commented Mar 2, 2017 at 17:05

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.