How to read giant CSV file by loop

Question

I have a base CSV. It is in the source:311-Service This base has about 11 GB. It is 19 million rows and 41 columns.

I want to take the information only about city: NEW JERSEY form column City. I can use this inquiry only for 500,000 rows. It works!

NYPD = pd.read_csv('c:/1/311_Service_Requests_from_2010_to_Present.csv', nrows=500000, low_memory=False) M = NYPD.loc[NYPD.City=='NEW JERSEY', :] M.to_csv('c:/1/NJ_NYPD.csv')

I need information from all rows of the CSV file, not from only 500 000 rows. I think I need to use a loop and chunksize = 500,000, but I don't know how.

hunksize =500000 i = 0 j = 1 for df in pd.read_csv('c:/1/311_Service_Requests_from_2010_to_Present.csv', chunksize=chunksize, iterator=True, low_memory=False): df.loc[df.City=='NEW JERSEY', :] df.index += j i+=1 df.to_csv('c:/1/NJ_NYPD.csv')

I don't want to translate CSV in to dbase method.

use skiprows argument

Nihal
– Nihal

2019-02-21 05:55:17 +00:00
Commented Feb 21, 2019 at 5:55 — Nihal
– Nihal, Commented Feb 21, 2019 at 5:55

DirtyBit · Accepted Answer · 2019-02-21 06:00:23Z

1

Why not just use the desired column at parse time:

my_filtered_csv = pd.read_csv(filename, usecols=['City'])

answered Feb 21, 2019 at 6:00

DirtyBit

16.8k5 gold badges37 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

penta · Accepted Answer · 2019-02-21 06:21:41Z

How about trying the API call :-

data.cityofnewyork.us/resource/fhrw-4uyv.json?city=NEW%20JERSEY

this loads json only for the specified city, then convert this json to your dataFrame

import requests import json from pandas.io.json import json_normalize data = requests.get('https://data.cityofnewyork.us/resource/fhrw-4uyv.json?city=NEW%20JERSEY') json_normalize(json.loads(data.text))

P.S.:- They have some documentation with python pandas too :-

https://dev.socrata.com/foundry/data.cityofnewyork.us/fhrw-4uyv

P.P.S.:- I did not register for the service, so it shows only 50 records as to my knowledge.

looks promising - I will practice it tonight and I will give an answer if it works. thanks!!

Collectives™ on Stack Overflow

How to read giant CSV file by loop

2 Answers 2

Comments

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Related