Conversation
|
I will automatically update this comment whenever this PR is modified
|
jules32 left a comment
jules32 left a comment There was a problem hiding this comment.
Hi! Great work on this Danny! A few commits and some suggestions to consider.
| | ||
| Several deliberate design decisions shape the library: | ||
| | ||
| **Build on, don't replace, existing libraries.** `earthaccess` composes existing |
There was a problem hiding this comment.
I know some decisions of what not to do were really important. If I remember correctly, it was features that were discussed then developed elsewhere rather than earthaccess. I'd suggest emphasizing this here, it's a big deal!
There was a problem hiding this comment.
I put in a blurb about this below ("Contribute upstream, don't accumulate") – could really use more eyes on it!
Co-authored-by: Julia Stewart Lowndes <julia@openscapes.org>
There was a problem hiding this comment.
We could symlink this in to our docs!
I would say let's not wait. We've demonstrated impact and I think that matters more. Alternatively, let's just go 1.0.0 in the short term and be OK with quickly moving to a 2.0.0 release with breaking changes. I think both are fine, but the latter sets more a precedent of maintainers taking the user impact of breaking changes too lightly. |
Co-authored-by: Matt Fisher <3608264+mfisher87@users.noreply.github.com>
I'm fine with either too. I also think the decision could be on hold until one of the two things – (i) co-author reviews/revisions, (ii) development for v1.0.0 – is completely ready-to-go. |
danielfromearth assigned betolink, itcarroll, chuckwondo, jhkennedy, jrbourbeau, JessicaS11, battistowx, asteiker and Sherwin-14 and unassigned betolink, itcarroll and chuckwondo 
itcarroll left a comment
itcarroll left a comment There was a problem hiding this comment.
Thanks @danielfromearth for all the excellent work on this!
| # Acknowledgements | ||
| | ||
| The development of `earthaccess` was supported by NASA's Earth Science Data Systems | ||
| (ESDS) program through the Openscapes project (NASA award **______**, PIs Julia |
There was a problem hiding this comment.
don't forget to dig up the award number
There was a problem hiding this comment.
@jules32, would you be able to include this?
Co-authored-by: Ian Carroll <carroll.ian@gmail.com>
| } | ||
| | ||
| @software{virtualizarr, | ||
| title = {{VirtualiZarr}: Create virtual {Zarr} stores from archival data using {xarray} syntax}, |
There was a problem hiding this comment.
| title = {{VirtualiZarr}: Create virtual {Zarr} stores from archival data using {xarray} syntax}, | |
| title = {{VirtualiZarr}: Create virtual {Zarr} stores from archival data using {Xarray} syntax}, |
| # Statement of need | ||
| | ||
| NASA's Earth science data archive is one of the largest and most diverse collections of | ||
| Earth observation data in the world, used by tens of thousands of researchers, educators, |
There was a problem hiding this comment.
| Earth observation data in the world, used by tens of thousands of researchers, educators, | |
| Earth observation data in the world, used by tens of millions of researchers, educators, |
ESDS metrics, accessed 25 Mar 2026
| `earthaccess` was created to address this gap: it provides uniform access to NASA | ||
| Earthdata regardless of data storage location, enabling researchers to focus on science | ||
| rather than data engineering. | ||
| |
There was a problem hiding this comment.
| `earthaccess` was created to address this gap: it provides uniform access to NASA | |
| Earthdata regardless of data storage location, enabling researchers to focus on science | |
| rather than data engineering. | |
| `earthaccess` was created to address this gap: it provides uniform access to NASA | |
| Earthdata regardless of data storage location and handles authentication, credentials, and tokening behind the scenes, enabling researchers to focus on science rather than data engineering. |
Not sure what would be better ("data access and authentication"? "API engineering"?), but I'm wanting to change the "data engineering" at the end of the sentence. It might be pedantic, but to me data analysis doesn't necessarily exclude data engineering in the modern world (though I think that's the ideal we're working towards).
| The target audience includes Earth scientists, remote sensing researchers, climate modelers, | ||
| hydrologists, ecologists, and any researcher, application developer, or educator who needs | ||
| to work with NASA Earth science data. The library is designed to be approachable for those new to Python -- with a |
There was a problem hiding this comment.
| The target audience includes Earth scientists, remote sensing researchers, climate modelers, | |
| hydrologists, ecologists, and any researcher, application developer, or educator who needs | |
| to work with NASA Earth science data. The library is designed to be approachable for those new to Python -- with a | |
| The target audience includes Earth scientists, remote sensing researchers, modelers, and any researcher, data user, application developer, or educator who needs | |
| to work with NASA Earth science data. The library is designed to be approachable for those new to Python -- with a |
We could add a more specific list of Earth scientists (hydrologists, ecologists, oceanographers, cryospheric scientists, climate modelers, some other broad categories, etc.) as another sentence (not sure how close to the word limit we are).
| supporting environment variables, `.netrc` files, and interactive prompts. Once | ||
| authenticated, the library creates HTTP sessions that correctly handle NASA's | ||
| cross-domain redirects and retrieves temporary AWS S3 credentials for in-region | ||
| cloud access. |
There was a problem hiding this comment.
Do the tokens still expire after an hour? And if so, I think earthaccess will renew them? If that's the case, I'd suggest adding this to the last sentence ("retrieves and renews temporary AWS...").
| 3. **Access**: Detects at runtime whether the process is running within AWS `us-west-2` | ||
| and automatically selects the optimal access path -- direct S3 reads for in-region | ||
| access or HTTPS downloads otherwise. Files can be opened as `fsspec`-compatible | ||
| file-like objects for streaming into libraries such as xarray [@xarray], or |
There was a problem hiding this comment.
| file-like objects for streaming into libraries such as xarray [@xarray], or | |
| file-like objects for streaming into libraries such as Xarray [@xarray], or |
| hard-coded URLs and custom authentication logic. | ||
| | ||
| **Multi-institutional development.** Contributors span NASA's Distributed Active Archive Centers (DAACs) — including ASDC, ASF, GES DISC, LP DAAC, NSIDC, OB.DAAC, ORNL DAAC, and PO.DAAC — as well as other federal and academic institutions (USGS, | ||
| University of New Hampshire), private industry (Coiled, Development Seed), |
There was a problem hiding this comment.
| University of New Hampshire), private industry (Coiled, Development Seed), | |
| Goddard, University of New Hampshire, University of Maryland), private industry (Coiled, Development Seed), |
| @danielfromearth Love this - thank you so much for putting it together! Sorry it looks like a lot of edits - most of them are pretty minor (grammatical or editorial), with a few suggestions for the text. All that said, none of them are non-starters for me. |
Co-authored-by: Jessica Scheick <JessicaS11@users.noreply.github.com>
Co-authored-by: Jessica Scheick <JessicaS11@users.noreply.github.com>
Co-authored-by: Jessica Scheick <JessicaS11@users.noreply.github.com>
Co-authored-by: Jessica Scheick <JessicaS11@users.noreply.github.com>
12 participants
Manuscript draft
This PR is intended for revisions and improvements to the manuscript draft being prepared for submission to the Journal of Open Source Software (JOSS).
Paper format: The manuscript is prepared as a Markdown (
paper.md) file with references in apaper.bibfile, following the JOSS formatting guidelines.For a PDF preview: With docker installed locally, a PDF preview of the draft manuscript can be generated, by running the following from the earthaccess root directory (as described in the JOSS guidelines's docker section):
📚 Documentation preview 📚: https://earthaccess--1249.org.readthedocs.build/en/1249/