Scrape URL provided by CSV

Question

I have a CSV containing some columns with data, the 15th column reports a list of URLs. Now, I need to select each URL from the column, scrape a new price from the target webpage, and then store that in the price column to update the old price.

Without the same column enumeration, here is an approximate CSV:

asin,title,product URL,price KSKFUSH01,Product Title,http://....,56.00

Below is the sample code I wrote, but it merely prints URLs :(

import csv with open('some.csv', 'r') as csv_file: csv_reader = csv.reader(csv_file) for line in csv_reader: print(line[15])

Any help or suggestions about accomplishing this goal?

Thanks

do you have a sample URL? I would recommend looking into BeautifulSoup — usernamenotfound
– usernamenotfound, Commented Dec 5, 2017 at 17:14
Hello tutorial looks great, this is sample URL: amazon.it/dp/B006H9B6XI, but basically i don't know how read each URL from csv file — Stew
– Stew, Commented Dec 5, 2017 at 17:20
If the csv has properly named columns, you can split each line by commas to extract the URLs. — user9048861
– user9048861, Commented Dec 5, 2017 at 17:21
i get an error: AttributeError: 'list' object has no attribute 'split' — Stew
– Stew, Commented Dec 5, 2017 at 17:23
Your writing is difficult to parse. Your sample csv does not match your description. I am updating my answer. — user9048861
– user9048861, Commented Dec 5, 2017 at 17:38

score 1 · Accepted Answer · 2017-12-05 17:56:42Z

It looks like you want to use a csv writer. You can access the URL in each line. Here is how you can write the new price.

import csv import urllib2 from bs4 import BeautifulSoup with open('some.csv', 'r') as csv_file: csv_reader = csv.reader(csv_file) with open('newPricedata.csv', 'w', newline='') as Newcsvfile: Pricewriter = csv.writer(Newcsvfile, delimiter=' ', quotechar='|', quoting=csv.QUOTE_MINIMAL) for line in csv_reader: page = urllib2.urlopen(line[15]) soup = BeautifulSoup(page, ‘html.parser’) price = soup.find(‘td’, attrs={‘class’: ‘a-size-mini a-color-price ebooks-price-savings a-text-normal'}) Pricewriter.writerow(line[0]+','+,line[1]+','....+price.text)

I updated my answer with some basic code that should help you.
It is bad form to do so. Read this post: stackoverflow.com/questions/3146571/…
Ok thank you! I have to test you code now, looks very professional!

usernamenotfound · Accepted Answer · 2017-12-05 17:15:39Z

Here's a good guide on how to scrape websites using BeautifulSoup https://medium.freecodecamp.org/how-to-scrape-websites-with-python-and-beautifulsoup-5946935d93fe

Collectives™ on Stack Overflow

Scrape URL provided by CSV

2 Answers 2

4 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Linked

Related