Skip to content

Working with 189MB JSON document

Saddam H edited this page Jul 1, 2018 · 1 revision

Input File:

Size: 189MB

Link: https://github.com/zemirco/sf-city-lots-json

Array size: 206560

Sample Code:

package main import ( "fmt" "time" "github.com/davecgh/go-spew/spew" "github.com/thedevsaddam/gojsonq" ) func main() { t1 := time.Now() // load file and decode jq := gojsonq.New(). File("./city.json") t2 := time.Now() // run query against the data jq.From("features").WhereEqual("properties.MAPBLKLOT", "6092026") spew.Dump(jq.Get(), jq.Errors()) fmt.Println("Total time: ", time.Since(t1)) fmt.Println("Query time: ", time.Since(t2)) }

Time taken:

Total time: 8.254741002s // file open, read, decoding and query Query time: 216.347892ms // only query 

Output:

go run main.go ([]interface {}) (len=1 cap=1) { (map[string]interface {}) (len=3) { (string) (len=4) "type": (string) (len=7) "Feature", (string) (len=10) "properties": (map[string]interface {}) (len=9) { (string) (len=9) "BLOCK_NUM": (string) (len=4) "6092", (string) (len=7) "FROM_ST": (string) (len=3) "814", (string) (len=6) "STREET": (string) (len=6) "PERSIA", (string) (len=7) "ST_TYPE": (string) (len=3) "AVE", (string) (len=8) "ODD_EVEN": (string) (len=1) "E", (string) (len=9) "MAPBLKLOT": (string) (len=7) "6092026", (string) (len=6) "BLKLOT": (string) (len=7) "6092026", (string) (len=7) "LOT_NUM": (string) (len=3) "026", (string) (len=5) "TO_ST": (string) (len=3) "814" }, (string) (len=8) "geometry": (map[string]interface {}) (len=2) { (string) (len=4) "type": (string) (len=7) "Polygon", (string) (len=11) "coordinates": ([]interface {}) (len=1 cap=1) { ([]interface {}) (len=5 cap=8) { ([]interface {}) (len=3 cap=4) { (float64) -122.42950545086512, (float64) 37.7196273202666, (float64) 0 }, ([]interface {}) (len=3 cap=4) { (float64) -122.42957956336745, (float64) 37.71966264926639, (float64) 0 }, ([]interface {}) (len=3 cap=4) { (float64) -122.42940162924815, (float64) 37.71989812199502, (float64) 0 }, ([]interface {}) (len=3 cap=4) { (float64) -122.42932751546155, (float64) 37.719862792903164, (float64) 0 }, ([]interface {}) (len=3 cap=4) { (float64) -122.42950545086512, (float64) 37.7196273202666, (float64) 0 } } } } } } ([]error) <nil> Total time: 8.254741002s Query time: 216.347892ms

Clone this wiki locally