127

the following code worked until today when I imported from a Windows machine and got this error:

new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

import csv class CSV: def __init__(self, file=None): self.file = file def read_file(self): data = [] file_read = csv.reader(self.file) for row in file_read: data.append(row) return data def get_row_count(self): return len(self.read_file()) def get_column_count(self): new_data = self.read_file() return len(new_data[0]) def get_data(self, rows=1): data = self.read_file() return data[:rows]

How can I fix this issue?

def upload_configurator(request, id=None): """ A view that allows the user to configurator the uploaded CSV. """ upload = Upload.objects.get(id=id) csvobject = CSV(upload.filepath) upload.num_records = csvobject.get_row_count() upload.num_columns = csvobject.get_column_count() upload.save() form = ConfiguratorForm() row_count = csvobject.get_row_count() colum_count = csvobject.get_column_count() first_row = csvobject.get_data(rows=1) first_two_rows = csvobject.get_data(rows=5)
Improve this question
1

9 Answers 9

Reset to default
185

It'll be good to see the csv file itself, but this might work for you, give it a try, replace:

file_read = csv.reader(self.file)

with:

file_read = csv.reader(self.file, dialect=csv.excel_tab)

Or, open a file with universal newline mode and pass it to csv.reader, like:

reader = csv.reader(open(self.file, 'rU'), dialect=csv.excel_tab)

Or, use splitlines(), like this:

def read_file(self): with open(self.file, 'r') as f: data = [row for row in csv.reader(f.read().splitlines())] return data
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

This now gives the same error, but on line starting upload.num_records = csvobject.get_row_count() now
and when I try the split lines version (which is very cools thanks) I get coercing to Unicode: need string or buffer, S3BotoStorageFile found
What option eventually worked? Btw, you are reading the file twice: in get_row_count() and in get_column_count() - consider reading the file in __init__ and remember data in self.data, then use it in other methods.
+1 for splitlines() which avoids messing around with different formatting options on OSX. Hope it works across other platforms too...
Great answer. Using - "dialect=csv.excel_tab" however, screws up the output when used with csv.DictReader. Just the 'rU' options works magically though
|
55

I realize this is an old post, but I ran into the same problem and don't see the correct answer so I will give it a try

Python Error:

_csv.Error: new-line character seen in unquoted field

Caused by trying to read Macintosh (pre OS X formatted) CSV files. These are text files that use CR for end of line. If using MS Office make sure you select either plain CSV format or CSV (MS-DOS). Do not use CSV (Macintosh) as save-as type.

My preferred EOL version would be LF (Unix/Linux/Apple), but I don't think MS Office provides the option to save in this format.

Improve this answer

4 Comments

MS DOS Comma Separated didn't work for me (same error), but Windows Comma Separated.
If you are on a Mac, this is absolutely the correct answer.
I get the same issue on OS X. I find myself having to make a new CSV file. Simply saving the current as plain CSV format or CSV (MS-DOS) does not fix the issue.
On OS X, Windows Comma Separated csv worked, MS DOS Comma Separated didn't.
33

For Mac OS X, save your CSV file in "Windows Comma Separated (.csv)" format.

Improve this answer

1 Comment

thanks, that was the needed ingredient, as I'm using Mac w/ MS office.
19

If this happens to you on mac (as it did to me):

  1. Save the file as CSV (MS-DOS Comma-Separated)

  2. Run the following script

    with open(csv_filename, 'rU') as csvfile: csvreader = csv.reader(csvfile) for row in csvreader: print ', '.join(row)
Improve this answer

Comments

5

Try to run dos2unix on your windows imported files first

Improve this answer

2 Comments

no really an option I need to allow user to upload csv from both Windows and Macs without any special modification. The import was saved from Excel (Windows) as a CSV so maybe there is something extra that needs to be done in Python to read these?
@GrantU You are referring to Mac OS X 10.0 or later, not Mac OS 9 or earlier, correct? Between 9 and 10, Mac OS switched from \x0d (ProDOS) line endings to \x0a (UNIX) line endings.
2

This is an error that I faced. I had saved .csv file in MAC OSX.

While saving, save it as "Windows Comma Separated Values (.csv)" which resolved the issue.

Improve this answer

Comments

1

This worked for me on OSX.

# allow variable to opened as files from io import StringIO # library to map other strange (accented) characters back into UTF-8 from unidecode import unidecode # cleanse input file with Windows formating to plain UTF-8 string with open(filename, 'rb') as fID: uncleansedBytes = fID.read() # decode the file using the correct encoding scheme # (probably this old windows one) uncleansedText = uncleansedBytes.decode('Windows-1252') # replace carriage-returns with new-lines cleansedText = uncleansedText.replace('\r', '\n') # map any other non UTF-8 characters into UTF-8 asciiText = unidecode(cleansedText) # read each line of the csv file and store as an array of dicts, # use first line as field names for each dict. reader = csv.DictReader(StringIO(cleansedText)) for line_entry in reader: # do something with your read data
Improve this answer

Comments

1

I know this has been answered for quite some time but not solve my problem. I am using DictReader and StringIO for my csv reading due to some other complications. I was able to solve problem more simply by replacing delimiters explicitly:

with urllib.request.urlopen(q) as response: raw_data = response.read() encoding = response.info().get_content_charset('utf8') data = raw_data.decode(encoding) if '\r\n' not in data: # proably a windows delimited thing...try to update it data = data.replace('\r', '\r\n')

Might not be reasonable for enormous CSV files, but worked well for my use case.

Improve this answer

1 Comment

That solved my problem，Thanks！Look here
0

Alternative and fast solution : I faced the same error. I reopened the "wierd" csv file in GNUMERIC on my lubuntu machine and exported the file as csv file. This corrected the issue.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.