Merge Release 21.3.2 into Release 21.3.x#4318
Open
acyu wants to merge 75 commits intorelease-21.3.xfrom
Open
Conversation
ruff fixes Generated pydantic schemas Updated uv.lock
…mulus into mikedorfman/CUMULUS-4382
…ed placeholder deploy tf filesAdded placeholder deploy tf filesAdded placeholder deploy tf filesAdded placeholder deploy tf filesAdded placeholder deploy tf filesAdded placeholder deploy tf files
for more information, see https://pre-commit.ci
…mulus into mikedorfman/CUMULUS-4382
mikedorfman/CUMULUS-4382
…GranuleIdInputFile (#4215) * getGranulesForPayload can get granules from s3 file * update endpoints and functions to take granuleIds not granuleId+collectionId * add new unit tests * add changelog entries * fix lint * override tar for audit * update request validation refactor * add integration test * add typing to lib granules * fix err->error * fix typing * update request validation and bulk description * remove tar change * remove getUniqueGranuleByGranuleId and PR feedback more typing * update test titile * update changelog there is release cut
* Update lerna to v8, add tar override shim * Add missing bamboo script to 4313
for more information, see https://pre-commit.ci
Mason localstack ci
…4248) * CUMULUS-451:Create s3search Module for DuckDB with S3-Backed Tables
* increment version to 21.3.0 * increment async-operation to 55 * update CL * update contributors * update package api docs * add website documentation * Release 21.3.0 (#4227) * increment version to 21.3.0 * increment async-operation to 55 * update CL * update contributors * update package api docs * add website documentation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Tim Clark <tim.clark@nasa.gov> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Release 21.3.0 (#4233) * Update lerna to v8, add tar override shim (#4230) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Tim Clark <tim.clark@nasa.gov> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix CL 4458 * add tar_shim to npm publish * fix compare in CL * update async_operation_image_version to 55 * update async_operation_image_version to 55 * update ecs async-operation Dockerfile to node 22 * increment version in website package.json * Release 21.3.1 (#4255) * added states:StartExecution action to the <prefix>-steprole IAM role * Overriding vulnerable dependency in @aws-client/xml-builder (cherry picked from commit eeefc18) * Moved fast-xml-parser override to the root package.json (cherry picked from commit d32ab15) * CUMULUS-4514 cherry pick * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * version bump to 21.3.1 --------- Co-authored-by: Tim Clark <tim.clark@nasa.gov> Co-authored-by: mikedorfman <michael.dorfman@colorado.edu> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * update changelog for v21.3.1 * update CL --------- Co-authored-by: Tim Clark <tim.clark@nasa.gov> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: mikedorfman <michael.dorfman@colorado.edu>
* First pass at an aws-api-proxy task * Added a readme Updated schema generators * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated Changelog * Updated old granule-invalidator references * Added partial failure compatibility * Added package_uv to bin root from a different feature branch * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates based on PR feedback * Updated ajv from 6.12.3 to 8.18.0 to test dependency vulnerability fix * Updated readonly->readOnly in schemas after strict-mode check * Updates per PR feedback * Reverting config validation to use a static list of literals with a bit comment disclaimer * Fixed broken output validation * Reverted ajv version bump --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…um core task (#4211) * implemented multipart checksum computation based on threshold and partition values * fixed ESLint errors and warnings * add retries to s3 getObject calls using aws-client, cleaned up unit tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * cleaned up granule size variables and arguments * cleaned up typing, removed env variables for setting retries * updated CHANGELOG with CUMULUS-4352 changes * updated unit tests to check updateHashFromBody branches --------- Co-authored-by: Brandon Lokey <brandon.t.lokey@nasa.gov> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…rements Bug (#4238) * first commit * adding unit tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixing units * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adding lambda unit + fixing var * jsdoc + test fixes * fixing booleans * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * small test * switching var name * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * PR feedback to be continued * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * testing * testing * grammar changes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * small nit fix * PR feedback * adding back in something accidentally deleted * small bamboo script for fix * removing line * PR feedback * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * some doc/wording fixes * wording/grammar fixes * wording/grammar changes * adding checks for DataGranule to int tests * int test fixes * fixing int tests * switching DayNightFlag to lowercase in all instances * better DataGranule testing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * reverting pre-commit leftover changes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* First pass at an aws-api-proxy task * Added a readme Updated schema generators * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated Changelog * Updated old granule-invalidator references * Added partial failure compatibility * Added package_uv to bin root from a different feature branch * Took a swag at adding an module that helps create consistent python tasks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated incorrect zip path * Added aws_api_proxy_task as an output to both the ingest and cumulus module * Updated changelog * Removed accidentally-added CHANGELOG entry * Fixed incorrect reference in outputs * Updated so handler is a variable with a default of main.handler * Moved python_lambda and renamed it Added terraform and aws-provider specs to cumulus-task module per tflint * Updates per PR feedback * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates per PR comments * Updated to merge tags in a local and reference them in both the cloudwatch log group and the lambda * Fixed issue with aws region reference --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…rkflow (#4269) * added private_api_lambda_arn as an output, refactor granule_invalidator_workflow.tf to use the new output * updated changelog * move CUMULUS-4564 changelog entry under unreleased fixed section * added changelog entry for private_api_lambda_arn output in the added section of the changelog * moved private_api_lambda_arn output to after archive_api_redirect_uri output * updated incorrect package versions to 21.3.1 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* First pass at an aws-api-proxy task * Added a readme Updated schema generators * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated Changelog * Updated old granule-invalidator references * Added partial failure compatibility * Added package_uv to bin root from a different feature branch * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Took a swag at adding an module that helps create consistent python tasks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated incorrect zip path * Updates based on PR feedback * Updated ajv from 6.12.3 to 8.18.0 to test dependency vulnerability fix * Updated readonly->readOnly in schemas after strict-mode check * Updates per PR feedback * Reverting config validation to use a static list of literals with a bit comment disclaimer * Fixed broken output validation * Reverted ajv version bump * First stab at aws_api_proxy integration tests * Added aws_api_proxy_task as an output to both the ingest and cumulus module * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated changelog * Removed accidentally-added CHANGELOG entry * Fixed incorrect reference in outputs * Updated so handler is a variable with a default of main.handler * Second pass at integration tests * Fixed example-stack reference to new cumulus output * Added AWS proxy integration tests and supporting updates to aws-client * Moved python_lambda and renamed it Added terraform and aws-provider specs to cumulus-task module per tflint * Updates per PR feedback * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated the ChangeLog * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixed linting issue * Added aws_api_proxy outputs in ingest and cumulus * Updates per PR comments * Updated to merge tags in a local and reference them in both the cloudwatch log group and the lambda * Fixed issue with aws region reference * Added outputs.tf to aws-api-proxy deploy * Updated broken outputs * Updates per PR feedback --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…ssion (#4231) * added refresh endpoint for launchpad saml * added token extension logic for EDL * Add iat claim to JWT tokens for better client-side session management - Enable the Issued At (iat) claim in createJwtToken by removing the noTimestamp: true option. - Refactor unit tests in test-token.js to handle non-deterministic JWT strings by verifying decoded claims instead of performing direct string comparisons. - This change allows client-side applications to accurately track session age and trigger proactive refreshes. * removed disabled comment since this endpoint is used * Modified oauth refresh token function to decode the existing token and so we can reuse it's iat claim if the token is verified. We then check that the oauth user is authorized before creating a new token and returning it to the client. * Modified saml refresh token function to decode the existing token and so we can reuse it's iat claim if the token is verified * Added iat parameter to createJwtToken function to support reusing iat during token refresh. iat will be used when if the parameter is not null or undefined * added tests to verify expected behavior of JWT refresh * fixed linter issues * added MAX_SESSION_DURATION environment variable and logic to enforce how many times a cumulus dashboard user can extend a session. Added tests to verify this behavior * updated changelog * updated unit tests * updated formatting * consolidated shared logic for token refresh functions in token.js and launchpadSaml.js * fixed linting errors * added unit tests for new token helper functions * set default token refresh expiration time extension to one hour * set default token refresh expiration time extension to one hour --------- Co-authored-by: Austin Wisdom <austin.j.wisdom@nasa.gov>
* Fix/Update ElasticSearch Metrics parameter
* Replace CnmResponse in example with cumulus core version * Include exception messages in workflow failures * Add logging to CnmResponse task * Replace CumulusProcess with run_cumulus_task in task implementation * Create task README.md * Rework task output to return 'granules' key directly This allows the output of the task to be fed directly into a following task using the default CMA output mapping
* Add pdr-cleanup and initialize task dir * Updates README.md * Adjusts styling for pdr-cleanup task * Adds unit tests to pdr-cleanup task * Adds check for s3 provider in pdr-cleanup * Ruff format fixes * Adds package.json * Improvements to cleanup logic * Improves testing and adds tooling * Add archive location of pdr to the task output * Add schemas * Changes file organization to meet task layout standards * Changes to Python 3.13 * Adds terraform deployment * Updates CHANGELOG.md * Update README.md to fit template * WIP: README * WIP: output.json fix * ruff check pass * Added sample terraform.tfvars * Remove cache file * WIP: Format fixes * Change logging statements to remove f-strings and adds lint check * Change doc style to restructuredText * Adjust task name * Removes task level bin * PDR cleanup adjustments
* upgrade lerna workspaces uuid update * remove prepare command from package,npm install ignore-scripts still run them * fix bootstrap * fix deployment * granule-invalidator-task 21.2.0 to 21.3.1 can it fix lint * fix another version * delete prepare * changelog * audit * webpack config * audit only root,npm install no concurrency * markdownlint-cli upgrade fix lint precommit * why point to another packages dependency * update dependencies @Cumulus package * minimatch has multiple issue * resolve package path * audit issue * remove double bootstrapping * snyk * fix cleanup script * serialize-javascript audit * add generate-task-schemas permission for ci * Timed out for ecs/async-operation unit tests * querystring replace * fix /cumulus/node_modules/.bin/generate-task-schemas: Permission denied * Cannot find module lock-stack integration with release * lock-stack needs package * release integration test no file or module issues * generate-task-schemas needs schema directory * add uuid to message package * remove package script from example to avoid race condition * lerna 905 * update package versions * remove prepare * 21.0.1 * changelog entry * optional section
… operations endpoints (#4263) * CUMULUS-4472: utilize concurrency and db connection pool for all bulk operations endpoints and lambda functions * fix missing concurrency from applyWorkflowToGranules pMap * enable passing concurrency and maxDbConnections for all granule bulk operations endpoints and lambda functions * update validations for bulk operations endpoints * narrow knex connections spawned in bulk operations pMap * update tests * CUMULUS-4472: update tests and JSDocs * CUMULUS-4472: add changelog entry * CUMULUS-4472: fix types used, set up knex client in case not passed in * CUMULUS-4472: change knex signature and param again, add tests for pMap continuing process for non error granules, update to nil check * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * CUMULUS-4472: modify test for pMap processing w errors * CUMULUS-4472: pr comment and lint fixes * CUMULUS-4472: fix test * CUMULUS-4472: update CL * CUMULUS-4472: fix lint error --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add depends_on for lambda log groups * Rename granule invalidator lambda to match the rest of cumulus core tasks * Rename cnm to cma lambda to match the rest of cumulus core tasks * Refactor tasks to use shared tf module * Update CHANGELOG.md * Refactor output ordering
* First stab at requirements export * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added fetch-depth of 0 for git checkout * Updated uv.lock * Updated requirements export * Updated github actions requirements pull * Fixed issues with requirements export * More tweaks to requirements generation * Update requirements.txt from uv.lock changes * Revert "Update requirements.txt from uv.lock changes" This reverts commit 25c02d6. * Revert "Update requirements.txt from uv.lock changes" This reverts commit 25c02d6. * More tweaking of gh actions * Update requirements.txt from uv.lock changes * Revert "Update requirements.txt from uv.lock changes" This reverts commit 5c8b0d8. * Reverted gh actions change * More testing * Dummy update to trigger GH actions * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update requirements.txt from uv.lock changes * Revert "Update requirements.txt from uv.lock changes" This reverts commit 9048afe. * Update requirements.txt from uv.lock changes * Specified to only run requirements rebuild when uv.lock files are changed * Regenerated all uv.lock files to trigger requirements.txt generation * Update requirements.txt from uv.lock changes * Bumped gh workflow file to trigger CI * Updated changelog * Updated paths parameter in gh workflow * Revert "Update requirements.txt from uv.lock changes" This reverts commit d533386. * Update requirements.txt from uv.lock changes * Revert "Regenerated all uv.lock files to trigger requirements.txt generation" This reverts commit b4cd208. * Regenerated all uv.lock files to trigger requirements.txt generation * Updates per PR feedback * Update requirements.txt from uv.lock changes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* First stab at get_cnm task code and unit tests * Added a README for task * Added integration tests Fixed ruff issues * Removed an unnecessary logging statement * Several small fixes from copy-pasting Adding Terraform deploy code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated Changelog Added get_cnm terraform for example tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated integration tests Updated task based on issues in integration tests * Updated get_cnm.py unit tests * Updates to resolve eslint issues * Updated spec names to correspond to the what we're actually testing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Temporarily allowlisted vulnerable packages in order to run integration tests * Fixed a bug in integration tests * Fixed eslint issues * Update tasks/get-cnm/deploy/main.tf Co-authored-by: Rohan Weeden <reweeden@alaska.edu> * Remvoed temporarily-added audit-ci.json exceptions * Updated to use the nasa version of cumulus-api Updated lockfiles for granule-invalidator and get-cnm * Updated lint gh actions to include setup-uv * Removed `prepare` npm target for get-cnm Removed not-needed setup-uv actions * Updated task readme Updated pyproject.toml version * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates per PR feedback * Update requirements.txt from uv.lock changes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update requirements.txt from uv.lock changes * Updates per PR feedback * Updated python package to match precommit * Update tasks/get-cnm/src/get_cnm/get_cnm.py Co-authored-by: Rohan Weeden <reweeden@alaska.edu> * Removed commented-out code --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Rohan Weeden <reweeden@alaska.edu> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* initialize blokey CI stack * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Brandon Lokey <brandon.t.lokey@nasa.gov> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* WIP: Initial setup for atisdale CI stack * Fixes atisdale Github name in select stack
* First stab at versioning python projects so they match the package.json file * Updates per feedback * Committed VERSION file * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update requirements.txt from uv.lock changes * Updated python tasks with VERSION file Added dynamic version references to pyproject.toml files * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update requirements.txt from uv.lock changes * Updated requirements generation file * Update requirements.txt from uv.lock changes * Added appropriate requirements generation to package.json * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update requirements.txt from uv.lock changes * Updated cnm-to-cma and cnm-response * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update requirements.txt from uv.lock changes * Updated changelog * Updated to use uv version * Updated the npm `update` target to also update python version --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
for more information, see https://pre-commit.ci
… release-21.3.2
Cut new version of Cumulus Documentation
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary: Summary of changes
Addresses CUMULUS-XX: Develop amazing new feature
Changes
PR Checklist
📝 Note:
For most pull requests, please Squash and merge to maintain a clean and readable commit history.