Download as PDF, PPTX
![… (SpatialKey, [(DateTime, Tile) (DateTime, Tile)]) (SpatialKey, (DateTime, Tile)) (SpatialKey, (DateTime, Tile)) (SpatialKey, (DateTime, Tile)) (SpatialKey, [(DateTime, Tile)]) …](https://image.slidesharecdn.com/slides-170815162027/75/Analyzing-Larger-RasterData-in-a-Jupyter-Notebook-with-GeoPySpark-on-AWS-FOSS4G-2017-Workshop-33-2048.jpg)
![… (SpatialKey, [(DateTime, Tile) (DateTime, Tile)]) (SpatialKey, (DateTime, Tile)) (SpatialKey, (DateTime, Tile)) (SpatialKey, (DateTime, Tile)) (SpatialKey, [(DateTime, Tile)]) (Shuffle) …](https://image.slidesharecdn.com/slides-170815162027/75/Analyzing-Larger-RasterData-in-a-Jupyter-Notebook-with-GeoPySpark-on-AWS-FOSS4G-2017-Workshop-34-2048.jpg)
![(SpatialKey, [(DateTime, Tile) (DateTime, Tile)]) (SpatialKey, [(DateTime, Tile)]) … mosaic (SpatialKey, Tile) (SpatialKey, Tile) … mosaic](https://image.slidesharecdn.com/slides-170815162027/75/Analyzing-Larger-RasterData-in-a-Jupyter-Notebook-with-GeoPySpark-on-AWS-FOSS4G-2017-Workshop-35-2048.jpg)
Uploaded byRob Emanuele
Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS - FOSS4G 2017 Workshop
This document outlines a presentation on analyzing large raster data in a Jupyter notebook with GeoPySpark on AWS. The presentation covers introductory material, exercises on working with land cover and Landsat imagery data, combining data layers to detect crop cycles, and combining different data types to create maps. It discusses where the notebooks are running, data sources, and GeoPySpark capabilities like working with space-time raster data. Attendees are encouraged to tweet maps created during the exercises.
Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS - FOSS4G 2017 Workshop
- 1. Rob Emanuele @lossyrob ANALYZINGLARGE RASTER DATA IN A JUPYTER NOTEBOOK WITH GEOPYSPARK ON AWS
- 2. Connect to theWIFI Network: Harvard University http://getonline.harvard.edu Click “I am a guest” Credentials: U: foss4g2017@gmail.com P: 7RFQU3rm FIRST: Find your Jupyter Notebook URL https://git.io/v77lh (lowercase L) visit the URL next to your name Log in to the Jupyter Hub U: hadoop P: hadoop
- 3. OUTLINE 8:00 - 8:30Intro and Background 8:30 - 9:10 Section 1: Land Cover data 9:10 - 10:00 Section 2: Landsat 8 data 10:00 - 10:10 BREAK 10:10 - 10:30 Deployment and Ingestion 10:30 - 11:10 Section 3: Combining data layers 11:10 - 12:00 Section 4: Making Cool Maps
- 4.
- 8.
- 10. rdd.map(lambda x: x+ 1) Source: http://silverpond.com.au/2016/10/06/balancing-spark.ht
- 15. (1, 1) (2,1)(0, 1) (0, 0) (1, 0) (2, 0) (1, 2) (2, 2)(0, 2)
- 16. (1, 1) (2,1)(0, 1) (0, 0) (1, 0) (2, 0) (1, 2) (2, 2)(0, 2) Node 1 Node 2 Node 3
- 17. (1, 1) (2,1)(0, 1) (0, 0) (1, 0) (2, 0) (1, 2) (2, 2)(0, 2) Node 1 Node 2 Node 3
- 18. (1, 1) (2,1)(0, 1) (0, 0) (1, 0) (2, 0) (1, 2) (2, 2)(0, 2) Node 1 Node 2 Node 3
- 19. (1, 1) (2,1)(0, 1) Node 1 Node 2 Node 3
- 20. (1, 1) (2,1)(0, 1) Node 1 Node 2 Node 3 rdd.bufferTiles(…)
- 21. + + Interactive and BatchProcessing of large raster data Web-Speed Processing of small to medium sized raster data
- 22. GeoTrellis Ecosystem Raster Foundryby Spark SQL and Spark ML support Raster Frames by Spark SQL and Spark ML support GeoPySpark Python bindings Vector Pipes Vector Tiles on Spark PDAL integration Point Clouds on Spark
- 23.
- 24. Started December 2016 FollowsPySpark’s model of communication between the JavaVirtual Machine and Python Access GeoTrellis functionality through Python, and integrates with your favorite python raster tools (numpy + friends). 0.2 is released! GeoPySpark
- 28. EXERCISE 1: ANALYZING LANDCOVER DATA
- 29. EXERCISE 2: WORKING WITHLANDSAT IMAGERY AND NDVITHROUGHTIME
- 31. (SpaceTimeKey, Tile) (SpaceTimeKey, Tile) (SpaceTimeKey,Tile) … SpaceTimeKey ≈ (col, row, instant)
- 32. (SpaceTimeKey, Tile) (SpaceTimeKey, Tile) (SpaceTimeKey,Tile) … lambda lambda lambda (SpatialKey, (DateTime, Tile)) (SpatialKey, (DateTime, Tile)) (SpatialKey, (DateTime, Tile)) …
- 33. … (SpatialKey, [(DateTime, Tile) (DateTime,Tile)]) (SpatialKey, (DateTime, Tile)) (SpatialKey, (DateTime, Tile)) (SpatialKey, (DateTime, Tile)) (SpatialKey, [(DateTime, Tile)]) …
- 34. … (SpatialKey, [(DateTime, Tile) (DateTime,Tile)]) (SpatialKey, (DateTime, Tile)) (SpatialKey, (DateTime, Tile)) (SpatialKey, (DateTime, Tile)) (SpatialKey, [(DateTime, Tile)]) (Shuffle) …
- 35. (SpatialKey, [(DateTime, Tile) (DateTime,Tile)]) (SpatialKey, [(DateTime, Tile)]) … mosaic (SpatialKey, Tile) (SpatialKey, Tile) … mosaic
- 36.
- 37. WHERE AND HOWARETHESE NOTEBOOKS RUNNING?
- 42.
- 43.
- 44. EXERCISE 3: COMBINING LANDCOVER AND NDVITO DETECT CROP CYCLES
- 46.
- 47. (SpaceTimeKey, Tile) (SpaceTimeKey, Tile) (SpaceTimeKey,Tile) … map_to_spatial (SpatialKey, (STK, Tile)) (SpatialKey, (STK, Tile)) (SpatialKey, (STK, Tile)) … map_to_spatial map_to_spatial STK = SpaceTimeKey
- 48. (SpatialKey, (STK, Tile)) (SpatialKey,(STK, Tile)) (SpatialKey, (STK, Tile)) … (SpatialKey, Tile) (SpatialKey, Tile) … ndwi_rdd nlcd_layer.to_numpy_rdd() (SpatialKey, ((STK, Tile), Tile)) (SpatialKey, ((STK, Tile), Tile)) (SpatialKey, ((STK, Tile),Tile)) …
- 49. (SpatialKey, (STK, Tile)) (SpatialKey,(STK, Tile)) (SpatialKey, (STK, Tile)) … (SpatialKey, Tile) (SpatialKey, Tile) … ndwi_rdd nlcd_layer.to_numpy_rdd() (SpatialKey, ((STK, Tile), Tile)) (SpatialKey, ((STK, Tile), Tile)) (SpatialKey, ((STK, Tile),Tile)) … (Shuffle)
- 50. mask_ndwi mask_ndwi mask_ndwi (SpaceTimeKey, Tile) (SpaceTimeKey, Tile) (SpaceTimeKey,Tile) … (SpatialKey, ((STK, Tile), Tile)) (SpatialKey, ((STK, Tile), Tile)) (SpatialKey, ((STK, Tile),Tile)) …
- 51. EXERCISE 4: COMBINING IMAGERY,ELEVATION AND LAND COVER DATA TO MAKE A COOL LOOKING MAP
- 52. EXERCISE 4: COMBINING IMAGERY,ELEVATION AND LAND COVER DATA TO MAKE A COOL LOOKING MAP TWEETYOUR SWEET MAP SCREENSHOTS WITH #GEOPYSPARK #FOSS4G!
- 53.
- 54.