feature: bridge to SILO – adding transform_upload_to_SILO#177
feature: bridge to SILO – adding transform_upload_to_SILO#177gordonkoehn wants to merge 7 commits intorubiconfrom
transform_upload_to_SILO#177Conversation
There was a problem hiding this comment.
Pull Request Overview
The PR integrates a new rule into V-Pipe's workflow for executing sr2silo to process DNA and amino acid alignments before their upload to SILO.
- Added a new environment file (workflow/envs/sr2silo.yaml) specifying the channels and dependency (sr2silo version 0.0.4).
Files not reviewed (1)
- workflow/rules/publish.smk: Language not supported
| @DrYak would you have suggestions to where I can find the
I need all of these files only to generate the metadata. |
🦙 MegaLinter status: ✅ SUCCESS
See detailed report in MegaLinter reports
|
DrYak force-pushed the 65-sr2silo-integration-into-v-pipe-workflow branch from 5b3cb85 to 777ce7e Compare 
gordonkoehn left a comment
gordonkoehn left a comment There was a problem hiding this comment.
I've made the changes to work with the new sr2silo v1.1.0 where I re-worked the interface. Now authentication and metadata are handled much, much better.
See bioconda PR here:
And the working example snakemake workflows here:
| password=config["loculus"]["password"], | ||
| username=config["loculus"]["username"], | ||
| group_id=config["loculus"]["group_id"], |
There was a problem hiding this comment.
Note, this is the authentication that we need to submit to Loculus.
They can also be set as environmental variables, so that they don't show up in snakemake logs.
Named: GROUP_ID, USERNAME, PASSWORD
I am not sure how this is best done here in Snakmake and VPipe.
| {params.SR2SILO} submit-to-loculus | ||
| --processed-file "{input.silo_input}" \ | ||
| --keycloak-token-url "{params.keycloak_token_url}" \ | ||
| --submission-url "{params.submission_url}" \ | ||
| --group-id {params.group_id} \ | ||
| --username "{params.username}" \ | ||
| --password "{params.password}" && \ |
There was a problem hiding this comment.
note also no more batch_id here
| "keycloak_token_url": { | ||
| "type": "string", | ||
| "default": "", | ||
| "description": "Loculus keycloak authentication URL to use during submission", | ||
| "example": ["https://auth.example.com/token"] | ||
| }, | ||
| "submission_url": { | ||
| "type": "string", | ||
| "default": "", | ||
| "description": "Loculus submission URL to use during submission", | ||
| "example": ["https://backend.example.com/api"] | ||
| }, | ||
| "lapis-url": { | ||
| "type": "string", | ||
| "default": "", | ||
| "description": "Lapis URL to for fetching the reference from LAPSI/SILO.", | ||
| "example": ["https://lapis.example.com"] | ||
| }, | ||
| "group_id": { | ||
| "type": "integer", | ||
| "default": 0, | ||
| "description": "Group ID to use for submission, for authentication wiht Loculus", | ||
| "example": [123] | ||
| }, | ||
| "username": { | ||
| "type": "string", | ||
| "default": "", | ||
| "description": "Username to use for authentication wiht Loculus.", | ||
| "example": ["your-username"] | ||
| }, | ||
| "password": { |
There was a problem hiding this comment.
Adding the parameters to the docs.
| @DrYak, let's implement this together. I am not sure how I can test this. |
gordonkoehn force-pushed the 65-sr2silo-integration-into-v-pipe-workflow branch 2 times, most recently from 5973f2a to 74fbe99 Compare 
- handle preprocessing and uploading as two separate rules
gordonkoehn force-pushed the 65-sr2silo-integration-into-v-pipe-workflow branch from fb4c59b to 0345bda Compare DrYak force-pushed the rubicon branch 2 times, most recently from db1530b to 934e088 Compare 
This PR aims to integrate a rule at the end of V-Pipe's workflow to execute sr2silo for the upload of DNA alignments, enriched with amino acid alignments to SILO.
The snakemake rule that is meant as a guiding layout is here.