Error when writing to a compressed CSV file in Python 3.x [duplicate]

Question

I am trying to write data (contained in a dict) to a compressed (gzip) CSV file. As far as I understand the gzip.GzipFile method should accept a writing operation as a normal file object. Such as:

 import gzip import csv with gzip.GzipFile(filename="test.csv.gz", mode="a") as gziphdlr: writer = csv.DictWriter(gziphdlr, fieldnames=['time','value']) writer.writeheader() writer.writerow({'time': 0.1, "value": 100}) writer.writerow({'time': 0.2, "value": 200})

However, I get the error:

 ... File "/usr/lib/python3.10/csv.py", line 154, in writerow return self.writer.writerow(self._dict_to_list(rowdict)) File "/usr/lib/python3.10/gzip.py", line 285, in write data = memoryview(data) TypeError: memoryview: a bytes-like object is required, not 'str'

Any suggestions where I may be wrong?

Many thanks!

Don't use gzip as your variable name, you're redefining the gzip module. — Barmar
– Barmar, Commented Apr 11, 2023 at 21:11
By default, the inner file in gzip is opened in binary mode. To use text mode (what the csv module outputs) you need mode='at'. — Nick ODell
– Nick ODell, Commented Apr 11, 2023 at 21:13

Corralien · Accepted Answer · 2023-04-11 21:14:27Z

You have to use a buffer text stream like TextIOWrapper:

import gzip import csv import io # Take care using append mode to not write headers multiple times with gzip.GzipFile(filename='test.csv.gz', mode='w') as gzip: buf = io.TextIOWrapper(gzip, write_through=True) writer = csv.DictWriter(buf, fieldnames=['time', 'value']) writer.writeheader() writer.writerow({'time': 0.1, 'value': 100}) writer.writerow({'time': 0.2, 'value': 200})

Try to load data with Pandas:

import pandas as pd df = pd.read_csv('test.csv.gz') print(df) # Output time value 0 0.1 100 1 0.2 200

Collectives™ on Stack Overflow

Error when writing to a compressed CSV file in Python 3.x [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related