Use python requests to download CSV

To simplify these answers, and increase performance when downloading a large file, the below may work a bit more efficiently.

import requests from contextlib import closing import csv from codecs import iterdecode url = "http://download-and-process-csv-efficiently/python.csv" with closing(requests.get(url, stream=True)) as r: reader = iterdecode(csv.reader(r.iter_lines(), 'utf-8'), delimiter=',', quotechar='"') for row in reader: print(row) 

By setting stream=True in the GET request, when we pass r.iter_lines() to csv.reader(), we are passing a generator to csv.reader(). By doing so, we enable csv.reader() to lazily iterate over each line in the response with for row in reader.

This avoids loading the entire file into memory before we start processing it, drastically reducing memory overhead for large files.

I like the answers from The Aelfinn and aheld. I can improve them only by shortening a bit more, removing superfluous pieces, using a real data source, making it 2.x & 3.x-compatible, and maintaining the high-level of memory-efficiency seen elsewhere:

import csv import requests CSV_URL = 'http://web.cs.wpi.edu/~cs1004/a16/Resources/SacramentoRealEstateTransactions.csv' with requests.get(CSV_URL, stream=True) as r: lines = (line.decode('utf-8') for line in r.iter_lines()) for row in csv.reader(lines): print(row) 

Too bad 3.x is less flexible CSV-wise because the iterator must emit Unicode strings (while requests does bytes) while the 2.x-only version—for row in csv.reader(r.iter_lines()):—is more Pythonic (shorter and easier-to-read). Anyhow, note the 2.x/3.x solution above won't handle the situation described by the OP where a NEWLINE is found unquoted in the data read.

For the part of the OP's question regarding downloading (vs. processing) the actual CSV file, here's another script that does that, 2.x & 3.x-compatible, minimal, readable, and memory-efficient:

import os import requests CSV_URL = 'http://web.cs.wpi.edu/~cs1004/a16/Resources/SacramentoRealEstateTransactions.csv' with open(os.path.split(CSV_URL)[1], 'wb') as f, \ requests.get(CSV_URL, stream=True) as r: for line in r.iter_lines(): f.write(line+'\n'.encode()) 

You can also use the DictReader to iterate dictionaries of {'columnname': 'value', ...}

import csv import requests response = requests.get('http://example.test/foo.csv') reader = csv.DictReader(response.iter_lines()) for record in reader: print(record) 

I use this code (I use Python 3):

import csv import io import requests url = "http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv" r = requests.get(url) r.encoding = 'utf-8' # useful if encoding is not sent (or not sent properly) by the server csvio = io.StringIO(r.text, newline="") data = [] for row in csv.DictReader(csvio): data.append(row) 

To convert to Pandas DataFrame:

from io import StringIO text=StringIO(download.content.decode('utf-8')) df=pd.read_csv(text) 

From a little search, that I understand the file should be opened in universal newline mode, which you cannot directly do with a response content (I guess).

To finish the task, you can either save the downloaded content to a temporary file, or process it in memory.

Save as file:

import requests import csv import os temp_file_name = 'temp_csv.csv' url = 'http://url.to/file.csv' download = requests.get(url) with open(temp_file_name, 'w') as temp_file: temp_file.writelines(download.content) with open(temp_file_name, 'rU') as temp_file: csv_reader = csv.reader(temp_file, dialect=csv.excel_tab) for line in csv_reader: print line # delete the temp file after process os.remove(temp_file_name) 

In memory:

(To be updated)

You can update the accepted answer with the iter_lines method of requests if the file is very large

import csv import requests CSV_URL = 'http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv' with requests.Session() as s: download = s.get(CSV_URL) line_iterator = (x.decode('utf-8') for x in download.iter_lines(decode_unicode=True)) cr = csv.reader(line_iterator, delimiter=',') my_list = list(cr) for row in my_list: print(row) 

I used below solution (Unfortunately others did not work for me):

import pandas as pd df = pd.read_csv('http://.../file.csv') 

The following approach worked well for me. I also did not need to use csv.reader() or csv.writer() functions, which I feel makes the code cleaner. The code is compatible with Python2 and Python 3.

from six.moves import urllib DOWNLOAD_URL = "https://raw.githubusercontent.com/gjreda/gregreda.com/master/content/notebooks/data/city-of-chicago-salaries.csv" DOWNLOAD_PATH ="datasets\city-of-chicago-salaries.csv" urllib.request.urlretrieve(URL,DOWNLOAD_PATH) 

Note - six is a package that helps in writing code that is compatible with both Python 2 and Python 3. For additional details regarding six see - What does from six.moves import urllib do in Python?

Python3 Supported Code

 with closing(requests.get(PHISHTANK_URL, stream=True})) as r: reader = csv.reader(codecs.iterdecode(r.iter_lines(), 'utf-8'), delimiter=',', quotechar='"') for record in reader: print (record) 

Python 3.x:

For anyone trying the solutions above, if you are getting a \ufeff at every new line, just change the encoding from utf-8 to utf-8-sig.

If you only want the CSV in string format, simply access the .text property of the request, like this:

req = requests.get("https: ...") req.encoding = 'utf-8-sig' csv_data = req.text 

this worked nicely for me:

from csv import DictReader f = requests.get('https://somedomain.com/file').content.decode('utf-8') reader = DictReader(f.split('\n')) csv_dict_list = list(reader) 

There are 2 ways to handle the csv data

To download csv data into a DataFrame

url = 'https://cdn.wsform.com/wp-content/uploads/2020/06/color_srgb.csv' import pandas as pd df = pd.read_csv(url) print(df.head()) 

You should be able to see the output:

 Name HEX RGB 0 White #FFFFFF rgb(100,100,100) 1 Silver #C0C0C0 rgb(75,75,75) 2 Gray #808080 rgb(50,50,50) 3 Black #000000 rgb(0,0,0) 4 Red #FF0000 rgb(100,0,0) 

To download csv data into a csv file

import requests import os resp = requests.get(url) with open('temp.csv', 'w') as f: f.write(resp.content.decode('utf-8')) print('File saved in:', os.getcwd()) 

You should be able to see the saved file in the current working directory (use os.getcwd() to see the current working directory)