2021-10-13
Version description of the v1.1.5 release of ICESat-2 SlideRule.
New Features¶
v1.1.5 - Default asset switched from project’s s3 bucket (atlas-s3) to NSIDC’s Cumulus bucket (nsidc-s3).
v1.1.5 - Non-blocking read API added to H5Coro to support greater parallelization of reads. When configured, H5Coro creates a thread-pool, assigns read requests to the next available thread, and returns a future to the calling application. The Atl03Reader in ICESat-2 plugin was updated to use the non-blocking API. Concurrent reads when from a maximum of 18 reads at a time to 128 reads at a time. A roughly 30% performance improvement was measured as a result of this change.
v1.1.5 - HttpServer (via the LuaEndpoint module) now monitors memory usage on the local system and will return a 503 response to any streaming requests made when the current memory usage exceeds a preconfigured threshold.
v1.1.5 - Test framework (pytest) and GitHub actions added to sliderule-python repository: #56
v1.1.5 - Polygon information extracted from icepyx region when using the ipxapi.py APIs: #60
v1.1.3 - In the
atl03recrecord (icesat2.atl03s and icesat2.atl03sp APIs), theinfofield has been replaced by theatl08_classandatl03_cnffields which hold the ATL08 photon classification, and ATL03 photon confidence level respectively.v1.1.3 - The default confidence level for all APIs that accept a confidence level has been changed to background (0) from high (4).
v1.1.2 - The api_widgets_demo updated to provide interface for creating polygon/bounding boxes, and to select a land class.
v1.1.1 - Added
ipxapi.pymodule to Python client to support ICEPyx users.v1.1.1 - GeoDataFrames are now sorted by time. Time is also used as the index. (APIs affected:
atl06,atl06p,atl03s,atl03sp).v1.1.0 - ATL08 classifications are now supported in the
atl06,atl06p,atl03s,atl03spAPIs: sliderule#71when the request parameters supply a list of ATL08 classifications to use, the server code will read the corresponding ATL08 data and only use the supplied classifications in the calculation
the following classifications are supported: noise, ground, canopy, top of canopy, unclassified
for the
atl03s,atl03sp, the presence of the ATL08 classification list also enables the returned photon data to include each photons classification
v1.1.0 - The following APIs now return GeoDataFrames instead of dictionaries:
atl06,atl06p,atl03s,atl03sp.this standardizes the return structure at no cost to performance
each GeoDataFrame has a “time” column which is a Python
datetimevalueeach GeoDataFrame uses the geometry.x and geometry.y to represent the “longitude” and “latitude” fields respectively.
the “delta_time” column now represents the time from the ATLAS Standard Data Product (SDP) epoch (January 1, 2018)
The GeoDataFrames returned by
atl03sandatl03spcontain a row for each photon that is returned
v1.1.0 - All APIs default to using version 004 of the data products.
v1.1.0 - Added the ground track field (“gt”) to the
atl06andatl06pelevation returns.added the following constants to the icesat2.py module: GT1L, GT1R, GT2L, GT2R, GT3L, GT3R
you can now do things like:
new_gdf = gdf[gdf["gt"] == icesat2.GT1L]
v1.1.0 - Added STRONG_SPOTS and WEAK_SPOTS constants to the icesat2.py module
you can no do things like:
new_gdf = atl06[atl06["spot"].isin(icesat2.WEAK_SPOTS)]
v1.1.0 - Python client published to conda-forge: sliderule
Major Issues Resolved¶
v1.1.5 - Unexpected termination of response data is now gracefully handled in the client with the request being retried: #40
v1.1.5 - Client updated to handle server-side memory issues gracefully - worker scale factor reduced to 3, back off (via sleep function) added to error handling, explicit handling of 503 error response: c47f0f0
v1.1.4 - The XMIT_SIGMA parameter of the ATL06 algorithm was updated to the correct value. cc47c5f
v1.1.2 - H5Coro memory usage was optimized to support runs against larger datasets
v1.1.1 - The issue with the server crashing in version 1.1.0 was resolved.
In version v1.0.6, support for NSIDC Cumulus was added; but it had a dormant bug in the implementation.
Each server maintains up-to-date AWS credentials for NSIDC resources in S3 by authenticating to the earthdata login servers and getting API tokens that last for 1 hour. Every 30 minutes, each server re-authenticates and gets new tokens. For performance reasons, AWS clients are pre-allocated and re-used. The AWS clients are only cycled when new tokens arrive. The code that created new clients with the updated tokens had race-conditions in it where a deleted client could still be in use.
The introduction of ATL08 processing in version 1.1.0 lengthed the amount of time a client was in use and therefore made the race-condition much more likely to occur.
The race condition has been fixed in this version.
v1.1.0 - Resolved delta_time calculation issue: sliderule#48
v1.1.0 - Resolved latitude and longitude calculation issue: sliderule#7
v1.1.0 - HttpServer class fatal exception bug that causes server to hang was fixed: sliderule#77
v1.1.0 - Health check made more robust to handle case when server connection hangs: devops#45
Minor Issues Resolved¶
v1.1.5 - HSDS support removed from codebase
v1.1.5 - Fixed count returned in Grand Mesa demo: #42
v1.1.5 - Ipyleaflet updates and fixes in io.py and ipysliderule.py modules: #54
v1.1.5 - PyPI dependencies listed in requirements.txt reduced to minimal set: #57
v1.1.5 - Fixed typo in CMR time query: 1e77117
v1.1.5 - Fixed geometry warning in creation of geodataframe: #52
v1.1.3 - An out-of-bounds vulnerability was fixed in the processing of ATL08 photon classifications. 658409c
v1.1.2 - GeoDataFrame index (which is a datetime calculated from the delta_time column) is now set to ns resolution so that all rows are unique. ef2abce
v1.1.2 - Fixed multiple memory leaks in h5coro, in the use of an asset object in reader classes, and in various other modules throughout the server code.
v1.1.2 - Fixed startup bug where client that needs credentials would never be able to get those credentials if created before the first set of credentials were available. da139e6
v1.1.1 - ATL08 classification can be supplied as a table or a string. 04bc5fb
v1.1.1 - Fixed out of bounds access caught by valgrind. 5af94a5
v1.1.1 - Changed name of Python bindings to
srpybinto avoid naming conflicts with the SlideRule Python client. 0525c0bv1.1.1 - Signed integer support added to Python bindings sliderule#84
v1.1.1 - Fixed bugs in s3cache I/O driver for H5Coro, and added support in Python bindings. 0170be8
v1.1.0 - TCP/IP and HTTP errors are caught and user-friendly error messages are printed: sliderule#26
v1.1.0 - Memory leak in Asset class fixed: sliderule#32
v1.1.0 - Server generated log messages logged by the Python client are logged at the level specified in the server log message: sliderule-python#1
v1.1.0 - The Python client logging is now completely turned off if the verbose setting is set to False: sliderule-python#26
v1.1.0 - Fixed bug in code where the along track spread and the number of photon checks were not being made in the correct place in the code. In certain circumstances the final elevation could have been calculated on the set of photons that failed validation instead of the preceding set that did not fail the validation. cb5948c
v1.1.0 - CMake build fails configuration if dependency not found: sliderule#40
v1.1.0 - Signals issued while mutex held in MsgQ class code: sliderule#79
v1.1.0 - Fixed race conditions in SlideRule’s Python bindings; specifically fixed the H5Coro API.
v1.1.0 - Added
set_rqst_timeoutAPI to icesat2.py module to allow user to control what the timeouts are for connecting to and reading from SlideRule servers.
Getting this release¶
https://
https://
https://