Downloaded 173 times
![Web Processing Service Wikipedia introduces OGC WPS as: [A service] designed to standardize the way that GIS calculations are made available to the Internet. WPS can describe any calculation including all of its inputs and outputs, and trigger its execution The specific processes served up by a WPS implementation are defined by the owner of that implementation. Although WPS was designed to work with spatially referenced data, it can be used with any kind of data. FOSS4G 2016, Bonn 22nd - 26th August 2016](https://image.slidesharecdn.com/gscrunchingfoss4g2016-160829155933/75/Crunching-Data-In-GeoServer-Mastering-Rendering-Transformations-WPS-Processes-And-SQL-Views-FOSS4G-2016-5-2048.jpg)
![Filtering, joining and aggregation Some example control regexps: • Y_INTERV: • Default: 1 • Regex: ^(d)+$ • OP • Default: sum • Regex: ^[avg|sum]$ SELECT (T.TS_VALUE / %Y_INTERV%) AS TS_VALUE, T.CD_TA_OCEANAREA, G.GEOMETRY FROM (SELECT CD_TA_OCEANAREA, OP%(TS_VALUE) AS TS_VALUE FROM FIGIS.TS_FI_TA WHERE FIC_ITEM IN (%FIC_ITEM%) AND CD_GEAR IN (%CD_GEAR%) AND YR_TA IN (%YR_TA%) AND QTR_TA IN (%QTR_TA%) GROUP BY CD_TA_OCEANAREA ) t LEFT OUTER JOIN FIGIS_GIS.GRID_G5 g ON T.CD_TA_OCEANAREA = g.CD_OAREA ORDER BY T.CD_TA_OCEANAREA FOSS4G 2016, Bonn 22nd - 26th August 2016](https://image.slidesharecdn.com/gscrunchingfoss4g2016-160829155933/75/Crunching-Data-In-GeoServer-Mastering-Rendering-Transformations-WPS-Processes-And-SQL-Views-FOSS4G-2016-25-2048.jpg)
Uploaded byGeoSolutions
Crunching Data In GeoServer: Mastering Rendering Transformations, WPS Processes And SQL Views - FOSS4G 2016
The document discusses the integration of web processing services (WPS) with Geoserver to improve geospatial data processing and visualization. It covers the historical development of WPS, examples of its application for data transformation, and advanced features for managing large datasets. Additionally, it highlights parametric SQL views and custom querying processes tailored for specific geospatial analyses, emphasizing efficiency in data handling and rendering.
In this document
Powered by AI
Presentation by Ing. Andrea Aime and Ing. Simone Giannecchini from GeoSolutions, focusing on GeoServer and data processing.
Development in GIS applications since 2008, emphasizing the necessity of data processing in modern applications.
Introduction to WPS as an OGC standard for GIS calculations available on the internet.
Example of a WPS operation: buffering an L-shaped geometry.
Difference between synchronous (quick results) and asynchronous (longer computations) WPS operations.
Common architecture involving WPS, WCS, WFS in a server-client model.
Integration of WPS with GeoServer for data processing.
Demo of a request builder for WPS to set parameters and execute processes.
Conducting on-the-fly data transformations using WPS within rendering.
Advisory slide indicating scenarios where WPS might not be necessary.
Benefits of using Spatial DBMS for efficient data handling and parametric SQL views.
Introduction of parametric SQL views for dynamic data fetching in GeoServer.
SQL code example demonstrating complex data fetching and joins.
Transitioning from theory to practical applications of discussed concepts.
Soil Monitor tools for tracking soil indices with GeoServer and mapping front-end.
Brief mention of the soil sealing algorithm used in the Soil Monitor.
Results presentation of the change matrix from the Soil Monitor application.
Visualization of the changes tracked by the Soil Monitor.
Monitoring the processes used in the Soil Monitor application.
Mapping and analyzing tuna catches using filtering and aggregation techniques.
Example of regex expressions used in SQL queries for data filtering.
Using GeoServer for generating animations by varying parameters.
Discussion about the deficiencies of WFS/WCS in providing download services.
Overview of a community WPS module for downloading filtered large datasets.
Functionality of MapStore GUI for managing asynchronous WPS download tasks.
Architecture of the download service integrating WPS, GeoServer, and database interfaces.
Using GeoServer as a WPS broker for running remote asynchronous processes.
Detailed aspects of remote WPS processes and their functionalities, including change detection.
Computing road accident risks through integration of SQL views and WPS for effective assessments.
Techniques for efficient querying and filtering within large datasets for performance.
Strategies for optimizing multi-buffer computations in geographic data.
Introducing the CMRE-IDA project focused on assessing risks to marine mammals.
Overview of the sound propagation model and its integration into the IDA process.
Different raster algebra processes including simple and advanced calculations using Jiffle.
Wrapping up the presentation and inviting questions from the audience.
Crunching Data In GeoServer: Mastering Rendering Transformations, WPS Processes And SQL Views - FOSS4G 2016
- 1. Ing. Andrea Aime Ing.Simone Giannecchini GeoSolutions Processing Data In GeoServer With WPS And SQL Views
- 2. GeoSolutions Founded inItaly in late 2006 Expertise • Image Processing, GeoSpatial Data Fusion • Java, Java Enterprise, C++, Python • JPEG2000, JPIP, Advanced 2D visualization Supporting/Developing FOSS4G projects GeoServer, MapStore GeoNetwork, GeoNode, Ckan Clients Public Agencies Private Companies http://www.geo-solutions.it FOSS4G 2016, Bonn 22nd - 26th August 2016
- 3. Some historical perspective Year 2008: Publish maps on the net Nice styling, maybe time/elevation based selector, some little extra filtering Maybe some editing, some PDF printing Since 2013: All of the above, but… We can hardly make a new application without some data processing in it FOSS4G 2016, Bonn 22nd - 26th August 2016
- 4. WPS: some quickreminders FOSS4G 2016, Bonn 22nd - 26th August 2016
- 5. Web Processing Service Wikipedia introduces OGC WPS as: [A service] designed to standardize the way that GIS calculations are made available to the Internet. WPS can describe any calculation including all of its inputs and outputs, and trigger its execution The specific processes served up by a WPS implementation are defined by the owner of that implementation. Although WPS was designed to work with spatially referenced data, it can be used with any kind of data. FOSS4G 2016, Bonn 22nd - 26th August 2016
- 6. An Example Buffera L shaped geometry with distance “2” Get the result back as GML FOSS4G 2016, Bonn 22nd - 26th August 2016
- 7. Synchronous vs asynchronous WPS client WPS Launch process Sendback results Simple Suitable for fast executions Synchronous WPS client WPS Launch process Status URL Check progress 50% Check progress 100% Results inline Link to results More complex Suitable for longer computations Asynchronous FOSS4G 2016, Bonn 22nd - 26th August 2016
- 8. Common WPS setup WPS Remote WCS Remote WFS HTTP server WPS client Request + dataor links to data Result data Fetch data FOSS4G 2016, Bonn 22nd - 26th August 2016
- 9. GeoServer WPS integration WPS Remote WCS Remote WFS HTTP server WPS client AllGeoServer Layers WMS clientWMS GeoServer UI FOSS4G 2016, Bonn 22nd - 26th August 2016
- 10. Demo request builder List processes Describe Set parameters and execute All in one form FOSS4G 2016, Bonn 22nd - 26th August 2016
- 11. Rendering transformations On-the-flydata transformations inside rendering chain Calling WPS processes from SLD docs Optimized for performance FOSS4G 2016, Bonn 22nd - 26th August 2016
- 12. Let’s take onestep back… you don’t always need a WPS
- 13. Spatial DBMS! • Neverunder-estimate the processing power of your BDMS: • Designed to efficiently juggle large quantities of data • Efficient spatial primitives (at least, in PostGIS) • Doesn’t get more local to your data than this! • Passing params down? • Parametric SQL views! FOSS4G 2016, Bonn 22nd - 26th August 2016
- 14. Parametric SQL views WMS/WFSclient GeoServer Spatial database &viewparams=from:2000000;high:5000000 Expanded Query FOSS4G 2016, Bonn 22nd - 26th August 2016
- 15. Parametric SQL views FOSS4G2016, Bonn 22nd - 26th August 2016 SELECT Date_part('year'::text, t1.obs_datetime) AS obs_year, t1.storm_num, t1.storm_name, t1.wind, t2.wind AS wind_end, t1.press, t2.press AS press_end, t1.obs_datetime, t2.obs_datetime AS obs_datetime_end, St_makeline(t1.geom, t2.geom) AS geom FROM storm_obs t1 join(SELECT storm_obs.id, storm_obs.storm_num, storm_obs.storm_name, storm_obs.wind, storm_obs.press, storm_obs.obs_datetime, storm_obs.geom FROM storm_obs) t2 ON(t1.obs_datetime + '06:00:00'::interval) = t2.obs_datetime AND t1.storm_name::text = t2.storm_name::text WHERE Date_part('year'::text, t1.obs_datetime) BETWEEN %min_obs_year% AND %max_obs_year% ORDER BY Date_part('year'::text, t1.obs_datetime), t1.storm_num, t1.obs_datetime • Building lines of the fly from point data:
- 16. Enough theory Let’s getdown to business!
- 17. Pure local WPS:Soil Monitor
- 18. Soil Monitor • ComputingSoil Monitor Indeces at country level • Soil Sealing • Change Matrix • GeoServer as the W*S Server • MapStore as the mapping Front-End • Webmapping and WPS Orchestration • Charting • GPU processing wrapper as WPS Processes • JCuda • Custom code to bridge between GeoServer/GeoTools and Cuda • Direct Ingestion of the resulting raster data • Raster Algebra as rendering transformation using Jiffle http://www.soilmonitor.it/ FOSS4G 2016, Bonn 22nd - 26th August 2016 https://github.com/mbedward/jiffle
- 19. Soil sealing algorithm FOSS4G2016, Bonn 22nd - 26th August 2016
- 20. Change matrix results FOSS4G2016, Bonn 22nd - 26th August 2016
- 21. Charting what changed FOSS4G2016, Bonn 22nd - 26th August 2016
- 22. Tracking running processes FOSS4G2016, Bonn 22nd - 26th August 2016
- 23. Pure SQL views:Tuna Atlas
- 24. Mapping Tuna Catches •Multiple Filtering • Aggregation • Joining quartely stats against the grid FOSS4G 2016, Bonn 22nd - 26th August 2016
- 25. Filtering, joining andaggregation Some example control regexps: • Y_INTERV: • Default: 1 • Regex: ^(d)+$ • OP • Default: sum • Regex: ^[avg|sum]$ SELECT (T.TS_VALUE / %Y_INTERV%) AS TS_VALUE, T.CD_TA_OCEANAREA, G.GEOMETRY FROM (SELECT CD_TA_OCEANAREA, OP%(TS_VALUE) AS TS_VALUE FROM FIGIS.TS_FI_TA WHERE FIC_ITEM IN (%FIC_ITEM%) AND CD_GEAR IN (%CD_GEAR%) AND YR_TA IN (%YR_TA%) AND QTR_TA IN (%QTR_TA%) GROUP BY CD_TA_OCEANAREA ) t LEFT OUTER JOIN FIGIS_GIS.GRID_G5 g ON T.CD_TA_OCEANAREA = g.CD_OAREA ORDER BY T.CD_TA_OCEANAREA FOSS4G 2016, Bonn 22nd - 26th August 2016
- 26. Animation • Animator tool •GetMap + variying parameters + frame control => animated GIF! • http://docs.geoserver.org/stable/en/user/tutoria ls/animreflector.html FOSS4G 2016, Bonn 22nd - 26th August 2016
- 27.
- 28.
- 29. Advanced Clip andShip • Community WPS module plus MapStore UI • Requirements • Download large amounts of data • Generic data filtering • Clip on polygon/bbox/circle, both vector and raster • Reproject to target CRS • Band selection in raster • Work in a cluster • Solution: new WPS processes, asynch WPS call FOSS4G 2016, Bonn 22nd - 26th August 2016
- 30. MapStore GUI Buffer processcalled synchronously when buffer size changes FOSS4G 2016, Bonn 22nd - 26th August 2016
- 31. MapStore GUI Tracking downloadstatus (asynch WPS) FOSS4G 2016, Bonn 22nd - 26th August 2016
- 32. Download service architecture WPS GeoServer Layers MapStoreWMS GetCapabilities List of layers Buffer DownloadEstimator Download GetStatus Fetch data Status database Shared Hazelcast/DBMS database FOSS4G 2016, Bonn 22nd - 26th August 2016
- 33. Pure WPS calling toother computing nodes: WPS Remote
- 34. WPS Remote • UseGeoServer as WPS Broker Run Remote Processes Asynchronously • Support Python or command line tools • Relies on XMPP for discovery and messaging/logging • Supports Dismiss and basic load balancing for different executors • Automagic results ingestion in GeoServer FOSS4G 2016, Bonn 22nd - 26th August 2016
- 35. WPS Remote FOSS4G 2016,Bonn 22nd - 26th August 2016
- 36. WPS Remote FOSS4G 2016,Bonn 22nd - 26th August 2016
- 37. WPS Remote FOSS4G 2016,Bonn 22nd - 26th August 2016
- 38. WPS Remote FOSS4G 2016,Bonn 22nd - 26th August 2016
- 39. WPS Remote Change Detection FOSS4G 2016, Bonn 22nd - 26th August 2016
- 40. Mixing parametric SQLviews and WPS: The Destination Project
- 41. Intro Computing the riskof road accidents involving dangerous goods (chemicals, petrol, gases and so on) Road segments and stats about car accidents Human and environmental «targets» Involved area, depending on type of good and amount of damage FOSS4G 2016, Bonn 22nd - 26th August 2016
- 42. Large Data Volume •Road network of good part of northern Italy • Road divided into segments • 100m portions (500k of them) • 500m aggregation (120k of them) • 1 km square cells (few hundreds) • 51 buffer distances (depending on good, scenario, level of damage) • Several types of targets: schools, malls, hospitals, populated areas, superficial and underground acquifers, crops, woods, …. FOSS4G 2016, Bonn 22nd - 26th August 2016
- 43. The road arcrisk formula (s) • Adding togheter the risk caused by the different • Arc own propension to accidentds • Types of goods • Human and enviromental targets • The system allows to compute partial views of the formula, either by selection of targets/goods or by computing portions of it • Has a number of coefficients that can be hand- tuned by the caller FOSS4G 2016, Bonn 22nd - 26th August 2016
- 44. The results, visually •Rendering transformation • Read the arcs/polys from the DB, compute their risk based on the chosen formula, scenario, targets, and coefficients FOSS4G 2016, Bonn 22nd - 26th August 2016
- 45. How to computeit efficiently? • Using SQL Views? No, the possible aggregations variants are too many • Using a pure Java process? No, too much data to transfer from the DBMS • Fully on the fly? No, too much data involved • Pre-compute all buffers and locate all involved targets before hand (pre-cooked per buffer risk) • Use a process that builds a final aggregation query on the fly (dynamic sql views) FOSS4G 2016, Bonn 22nd - 26th August 2016
- 46. Parametric queries onsteroids • Find which queries are needed, replace params • Some queries have sub-queries as params FOSS4G 2016, Bonn 22nd - 26th August 2016
- 47. Efficient rendering tx Risk process Queries database Arcs/Buffer areasdb Map renderer Build overall query, replace params Compute risk for a batch of arcs Raw arcs Arcs + risk • Compute risk on the fly in the viewing area • Batch requests to the BDMS to minimize round-trip overhead FOSS4G 2016, Bonn 22nd - 26th August 2016
- 48. Efficient cross layerfiltering • Show only targets involved in the scenario under study, e.g., the ones crossing the buffer areas where there is significant risk • Limit query to the current bbox FOSS4G 2016, Bonn 22nd - 26th August 2016
- 49. Efficient cross layerfiltering SELECT v_geo_popolazione_residente_pl.* FROM v_geo_popolazione_residente_pl WHERE v_geo_popolazione_residente_pl.fk_bersaglio_umano_pl in ( SELECT distinct bersaglio.fk_bersaglio_umano_pl FROM v_geo_popolazione_residente_pl bersaglio join siig_geo_ln_arco_1 on st_dwithin(bersaglio.geometria, siig_geo_ln_arco_1.geometria, %distanzaumano%) WHERE siig_geo_ln_arco_1.geometria && st_makeenvelope(%bounds%, 32632) ) This is a job for a parametric sql view FOSS4G 2016, Bonn 22nd - 26th August 2016
- 50. Efficient computation ofmulti-buffers • Computing 51 buffers around each of 500k arcs… • Slow! • Buffering over the previous buffer proved to be much faster than computing each buffer from the original arc • Custom multibuffer process FOSS4G 2016, Bonn 22nd - 26th August 2016
- 51.
- 52. Intro • The IntegratedDecision Aid (IDA) has been developed at the Centre for Maritime Research and Experimentation (CMRE) in order to provide naval exercise planners with an effective tool to assess risk to marine mammals from acoustic transmissions. • Assessment of potential hazard is performed through sound propagation modeling available in the interface. Sound Propagation Model FOSS4G 2016, Bonn 22nd - 26th August 2016
- 53. Sound Propagation Model •External Octave Matlab Process • Logs the asynchrouns process status to the DB as a Feature Octave Matlab Script for SPM WFS Follows the process status through the WFSWFSLog Process allows to insert and update the status of SPM into the DB Dynamically builds an external Octave command line using the input parameters through a FreeMarker Template (FTL) FOSS4G 2016, Bonn 22nd - 26th August 2016
- 54. Sound Propagation Model •The list of model runs is available on the bottom Data Grid • It is possible to get information on the input parameters and execution from the WFS FOSS4G 2016, Bonn 22nd - 26th August 2016
- 55. Raster Algebra Processes •Simple Raster Algebra • Gets an OGC Filter containing logical operations between raster layers and produces a binary (0/1) layer FOSS4G 2016, Bonn 22nd - 26th August 2016
- 56. Raster Algebra Processes •Advanced Raster Algebra using Jiffle • Wraps Jiffle (http://code.google.com/p/jiffle/) in order to execute complex Raster Algebra scripts against the input layers Jiffle is a scripting language for creating and analysing raster images. Rather than having to write and test lots of JAI or Java AWT boiler-plate code to access and manipulate images, Jiffle lets you concentrate on the interesting bit: your algorithm. Jiffle is being developed as part of the JAITools project. Update 16 January 2013: Jiffle sources have moved to GitHub FOSS4G 2016, Bonn 22nd - 26th August 2016
- 57. Raster Algebra Processes •Advanced Raster Algebra using Jiffle • Gets an OGC Filter containing logical operations between raster layers and produces a binary light layer FOSS4G 2016, Bonn 22nd - 26th August 2016
- 58. Raster Algebra Processes •Raster Algebra detailed info • Input parameters and outcomes are logged into WFS • WPS Raster Stats Process provides statistics on the coverages Raster Statistics and Area in sq Km Raster Algebra Process user inputs FOSS4G 2016, Bonn 22nd - 26th August 2016
- 59. Utility Processes • WPSRaster Stats process • WPS Temp Cleaner async process The WPS Raster Stats process can be invoked at run-time. It computes the Raster Stats on the current viewport area. A cleaner process has been created to cleanup all the IDA temporary files. Is invoked asynchronously and makes use of the IDA config parameters. Also a WFS Delete Transaction listener is available cleaning up all the input and temp files when removing a row from the Data Grid. FOSS4G 2016, Bonn 22nd - 26th August 2016
- 60. The End Questions? andrea.aime@geo-solutions.it simone.giannecchini@geo-solutions.it FOSS4G 2016,Bonn 22nd - 26th August 2016