2024-04-04
Version description of the v4.3.2 release of ICESat-2 SlideRule.
New Features¶
v4.3.2 - Container Runtime Environment (Feature Preview) - ability to run Python containers from SlideRule endpoint
v4.3.2 - CSV output from ParquetBuilder supported
v4.3.1 - eb1085d - GEDI APIs for L1B and L2A datasets now point to LP DAAC S3 bucket.
v4.3.1 - GeoParquet output now supports ancillary fields
v4.3.1 - Added support for staging GeoParquet files in public S3 bucket via setting the output asset to “sliderule-public”.
v4.3.1 - ICESat-2 APIs support filtering data by beam (e.g. gt1l, gt1r, gt2l, gt2r, gt3l, gt3r)
v4.3.1 - The Python client
sliderule.toregionfunction now supports shapely Polygonsv4.3.1 - Added
regionto output ofatl06pandatl03spAPIs; makes it easier to determine direction of travel for each measurement in addition to easily reconstructing which granule a measurement comes fromv4.3.1 - 878a48b - Python client can now be configured to rethrow exceptions for applications that want to manage exceptions themselves
v4.3.1 - SlideRule web APIs now support CORS as well as JavaScript clients that can only provide data payloads on POST requests
Issues Resolved¶
v4.3.2 - 8c8fe17 - docker compose restart policy changed to “always” for production
v4.3.2 - 5d893d0 - fixed bug in atl08 processing that caused crash when last batch was a full batch
v4.3.2 - 35c071f - changed health check to go to AppServer and removed ProbeServer; this fixes the issue where the AppServer goes down but the ProbeServer remained healthy and therefore the node was deemed healthy
v4.3.2 - 35c071f - fixed bug in S3 curl library where S3 get request immediately failed and debug message attempted to be printed with no data
v4.3.2 - 08a3eaf - implemented proxy (EnpointProxy) retries for failed proxied requests; the lock for the proxied node is held until the request is successful or the retries are exhausted (this prevents the retry always going back to the same node); polling for available nodes was slowed down to 1Hz (from 5Hz); bug in cleaning up ancillary data fixed)
v4.3.2 - 01f5b8b - pinned v15.0.2 of Apache Arrow library to avoid memory bug in latest tag; (the arrow repo does not have releases)
v4.3.2 - 5338ef6 - fixed CurlLib memory leak when client request is prematurely terminated
v4.3.2 - cf5290a - added memory usage limit to health check to avoid situation where a node with not enough memory to service a processing request keeps registering itself (and consequently always has no active locks because it rejects all requests and therefore keeps having requests routed to it)
v4.3.1 - 24a6c79 - fixed hanging issue when terminator is dropped due to queue being full
v4.3.1 - 25640d1 - fixed corner-case bug in ATL08 processing when no ancillary data is available.
v4.3.1 - 599f4d7 - fixed memory leak with atl08 ancillary fields
v4.3.1 - 407da7d - fixed bug in ABoVE classifier code that was causing out of bounds memory access
v4.3.1 - e94161a - fixed bug in GeoParquet geometry code where only the first latitude,longitude of each batch was being used for the batch; it caused the data to stack up on top of each other and look decimated
v4.3.1 - 368 - fixed bug in PhoREAL implementation ancillary field interpolation
Development Updates¶
v4.3.2 - f68aa36 - number of threads in proxy (EndpointProxy) is calculated based on a user provided hint or the number of nodes in the cluster currently registered
v4.3.2 - 08a3eaf - added support for stack traces to have symbols from dynamically loaded modules (via
__no_unload__which is defaulted in Debug builds)v4.3.2 - 5338ef6 -
sys.lsobjLua API added to list global LuaObjects and total number of LuaObjectsv4.3.1 - e9ec583 - reordered build and deploy action so that temporary token is still valid when uploading terraform
v4.3.1 - 381 - static website move from ECS to CloudFront
v4.3.1 - GeoParquet output no longer includes latitude and longitude columns that duplicate what is provided in the geometry column
v4.3.1 - e055bc6 - meta data for each record (e.g. coordinates, index, etc) is now indicated in the definition of the record and no longer provided in each endpoint lua file
v4.3.1 - 00ecdb3 - server.lua script waits to authenticate to DAACs until the IAM role credentials have been established; this prevents errors at startup when trying to fetch the netrc file
v4.3.1 - 08b56f7 - fixed disk usage metric display in Grafana
v4.3.1 - 0b00277 - fixed prometheus endpoint to return valid table
v4.3.1 - 373 - removed
maxifrom all examples
Getting This Release¶
https://
Benchmarks¶
atl06_aoi
output: 625445 x 16 elements
total: 26.377888 secs
gdf2poly: 0.000519 secs
toregion: 0.163608 secs
__parse_native: 21.453290 secs
todataframe: 0.406155 secs
merge: 0.000003 secs
flatten: 2.858293 secs
atl06p: 26.116402 secs
atl06_ancillary
output: 1180 x 17 elements
total: 2.853569 secs
gdf2poly: 0.000519 secs
toregion: 0.163608 secs
__parse_native: 0.733802 secs
todataframe: 0.006865 secs
merge: 0.016739 secs
flatten: 0.035448 secs
atl06p: 2.649313 secs
atl03_ancillary
output: 1180 x 17 elements
total: 2.746311 secs
gdf2poly: 0.000519 secs
toregion: 0.163608 secs
__parse_native: 0.159768 secs
todataframe: 0.007559 secs
merge: 0.017402 secs
flatten: 0.037174 secs
atl06p: 2.744864 secs
atl06_parquet
output: 1577 x 17 elements
total: 2.649226 secs
gdf2poly: 0.000519 secs
toregion: 0.163608 secs
__parse_native: 0.210186 secs
todataframe: 0.007559 secs
merge: 0.017402 secs
flatten: 0.041889 secs
atl06p: 2.648749 secs
atl03_parquet
output: 22833 x 22 elements
total: 1.534764 secs
gdf2poly: 0.000519 secs
toregion: 0.163608 secs
__parse_native: 0.011159 secs
todataframe: 0.007559 secs
merge: 0.017402 secs
flatten: 0.041889 secs
atl06p: 2.648749 secs
atl03sp: 1.507323 secs
atl06_sample_landsat
output: 914 x 20 elements
total: 6.381781 secs
gdf2poly: 0.000519 secs
toregion: 0.163608 secs
__parse_native: 0.373254 secs
todataframe: 0.008152 secs
merge: 0.015304 secs
flatten: 0.037495 secs
atl06p: 4.657887 secs
atl03sp: 1.507323 secs
atl06_sample_zonal_arcticdem
output: 1651 x 27 elements
total: 5.275135 secs
gdf2poly: 0.000451 secs
toregion: 0.006655 secs
__parse_native: 0.326858 secs
todataframe: 0.007310 secs
merge: 0.023985 secs
flatten: 0.050125 secs
atl06p: 5.266107 secs
atl03sp: 1.507323 secs
atl06_sample_nn_arcticdem
output: 1651 x 20 elements
total: 4.773236 secs
gdf2poly: 0.000421 secs
toregion: 0.006201 secs
__parse_native: 0.092055 secs
todataframe: 0.008839 secs
merge: 0.026527 secs
flatten: 0.051369 secs
atl06p: 4.764687 secs
atl03sp: 1.507323 secs
atl06_msample_nn_arcticdem
output: 1596050 x 20 elements
total: 125.249892 secs
gdf2poly: 0.000456 secs
toregion: 0.007406 secs
__parse_native: 89.351120 secs
todataframe: 1.128149 secs
merge: 15.596906 secs
flatten: 29.527919 secs
atl06p: 124.077317 secs
atl03sp: 1.507323 secs
atl06_no_sample_arcticdem
output: 1596050 x 16 elements
total: 64.487345 secs
gdf2poly: 0.000436 secs
toregion: 0.007185 secs
__parse_native: 53.242468 secs
todataframe: 0.967663 secs
merge: 0.000003 secs
flatten: 7.009740 secs
atl06p: 63.322487 secs
atl03sp: 1.507323 secs
atl03_rasterized_subset
output: 51968 x 22 elements
total: 3.796079 secs
gdf2poly: 0.000436 secs
toregion: 0.007185 secs
__parse_native: 1.657236 secs
todataframe: 0.038700 secs
merge: 0.000003 secs
flatten: 0.321405 secs
atl06p: 63.322487 secs
atl03sp: 3.282703 secs
atl03_polygon_subset
output: 50799 x 22 elements
total: 3.085772 secs
gdf2poly: 0.000436 secs
toregion: 0.007185 secs
__parse_native: 1.390404 secs
todataframe: 0.041605 secs
merge: 0.000003 secs
flatten: 0.349935 secs
atl06p: 63.322487 secs
atl03sp: 3.056956 secs