8

Having geopandas installed in my Python environment, I can read a shapefile into a geodataframe with

In: import geopandas as gpd myShapefile = gpd.read_file(path_to_my_shapefile) print(myShapefile) Out: myShapefile as a geodataframe

Unfortunately, I have some shapefiles which contain lots of attribute columns which I don't need in the end, slowing down the reading process a lot. Is there any possibility to limit the reading of the shapefile to specific attribute columns?

In regular pandas, I could use the usecols argument to the read_csv and read_table functions to limit the reading to the specified columns, e.g.

import pandas as pd pd.read_csv(path_to_my_csv_file, usecols=['onlyThisColumn', 'andThatColumnAsWell', 'butNoOther'])

However, using usecols in geopandas' read_file method gives an error, probably because geopandas uses Fiona to read shapefiles which does not accept the argument.

 File "C:\Python34-64bit\lib\site-packages\geopandas\io\file.py", line 13, in read_file with fiona.open(filename, **kwargs) as f: TypeError: open() got an unexpected keyword argument 'usecols'

Is there any other argument or way to achieve this with geopandas/Fiona?

Improve this question
1
  • 1
    Thanks for the hint, I marked the answer as the correct one. Commented Jan 13, 2024 at 10:19

4 Answers 4

Reset to default
8

Update for this topic:

geopandas & fiona now support reading specifiec columns from files using the ignore_fields or include_fields parameters in gpd.read_file().

https://geopandas.org/en/stable/docs/user_guide/io.html

Improve this answer
1
  • Great solution! But it doesn't work when reading geodatabases .gdb, unfortunately Commented Jun 6, 2024 at 8:27
4

Building on gene's answer, you can use GeoDataFrame.from_features

The following should do the trick:

import fiona def records(filename, usecols, **kwargs): with fiona.open(filename, **kwargs) as source: for feature in source: f = {k: feature[k] for k in ['id', 'geometry']} f['properties'] = {k: feature['properties'][k] for k in usecols} yield f

And then

gpd.GeoDataFrame.from_features(records(filename), ['prop1', 'prop2'])

I'd be curious to know if this speeds up your code, or if the properties need to be ignored in Fiona rather than the GeoDataFrame construction.

Improve this answer
2
  • Uhm, thanks for your support but I got the impression that this did not speed up anything, reading the files takes more or less as long as before :o I just cannot investigate this "scientifically" (using time measurement) at the moment, maybe later today or tomorrow. Did you have a different impression? Commented Jan 8, 2015 at 13:14
  • 2
    Fiona doesn't support field picking, so there is no speedup: geopandas must read every field and filter them. Could be a good feature, maybe sketch it out at github.com/Toblerity/Fiona/issues? Commented Jan 8, 2015 at 16:34
2

You can "simulate" usecols with a generator:

1) Classical method 

import fiona layer = fiona.open('test.shp') layer.schema {'geometry': 'Point', 'properties': OrderedDict([(u'dip', 'int:2'), (u'dipdir', 'int:3'), (u'Type', 'str:10')])} for feature in layer: print feature {'geometry': {'type': 'Point', 'coordinates': (272070.600041, 155389.38792000001)}, 'type': 'Feature', 'id': '0', 'properties': OrderedDict([(u'dip', 30), (u'dipdir, 130), (u'Type', u'incl')])} {'geometry': {'type': 'Point', 'coordinates': (271066.03214800003, 154475.63137700001)}, 'type': 'Feature', 'id': '1', 'properties': OrderedDict([(u'dip', 55), (u'dipdir', 145), (u'Type', u'incl')])} {'geometry': {'type': 'Point', 'coordinates': (273481.498868, 153923.49298800001)}, 'type': 'Feature', 'id': '2', 'properties': OrderedDict([(u'dip', 40), (u'dipdir', 155), (u'Type', u'incl')])}

2) With a generator (with only geometry and one column here, but it is easy to adapt)

def records(filename, col): # the generator reader = fiona.open(filename) for feat in reader: new = {} new['geometry'] = feat['geometry'] new['properties'] = {} new['properties'][col] = feat['properties'][col] yield new for feat in records("test.shp","dip"): print feat {'geometry': {'type': 'Point', 'coordinates': (272070.600041, 155389.38792000001)}, 'properties': {'dip': 30}} {'geometry': {'type': 'Point', 'coordinates': (271066.03214800003, 154475.63137700001)}, 'properties': {'dip': 55}} {'geometry': {'type': 'Point', 'coordinates': (273481.498868, 153923.49298800001)}, 'properties': {'dip': 40}}
Improve this answer
2
  • As already stated in my comment to jakew's answer, thanks for your approach, but I did not recognize an improvement in reading speed from this. Do you have a different impression? Commented Jan 8, 2015 at 13:17
  • 1
    No, it is only an alternative approach to give a solution to the question of jakew. Commented Jan 8, 2015 at 18:07
1

Well, not a direct solution to the problem, but in some cases the following might be helpful.

It is possible to use the ogr2ogr tool to do the attribute filtering in advance and then read the filtered shapefiles with geopandas / Fiona. My subjective impression is that this is faster than reading the unfiltered shapefiles directly (however, I thought both ogr2ogr and Fiona use the same shapefile reading engine under the hood?!).

To do so (with a properly installed GDAL/OGR on your system):

def filterShapefilesWithOgr2Ogr(inputFolder, outputFolder, attributesToSave): # inputFolder: the folder where the original shapefiles are located # outputFolder: the folder where the filtered shapefiles are to be stored # attributesToSave: string of attributes that shall be copied into the output, e.g. 'ID, location', length' import subprocess # traverse through the input folder os.chdir(inputFolder) for filename in glob.glob('*.shp'): # filter each shapefile from the input folder and save it at output folder print('filtering ' + filename) subprocess.call(["ogr2ogr", "-f", "ESRI Shapefile", "-select", attributesToSave, outputFolder + '/' + filename, filename]) return

If one is only interested into the attributes and not necessarily into the geometry features of the shapefiles, reading only the dbf table of the shapefile into a pandas dataFrame can also speed things up a lot (in my case, 10s vs. 10min).

To do so, I use dbfread (simply install with pip install dbfread).

def readDBFToDataFrame(inputFolder): import glob import dbfread import DBF # traverse through the input folder os.chdir(inputFolder) for filename in glob.glob('*.dbf'): print('reading dbf file ' + filename) # load the dbf file table = DBF(filename, load=False) # iterate through dbf file and store each record in a list listOfRecords = [] for record in table: listOfRecords.append(record) # create a pandas dataframe from the list of records df = pd.DataFrame(listOfRecords) return df
Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.