7

For over a week I've been fighting with passing a variables between jobs in multi-project pipeline in GitLab CI and got lots of weird errors. The mechanism looks like very basic and it drives me crazy that such an obvious thing still doesn't work for me, if somebody ran into similar issues - I would appreciate your help!

So what I've been trying to make: I have two projects on gitlab and I'm trying to link them in a single multi-project pipeline, the jobs schema looks like this: In project A:

variables: BUILD_PATH:"" build: script: - $BUILD_PATH="some-path" #the important point here that this value sets inside the job, it's not static bridge: variables: PATH: $BUILD_PATH RUN_TYPE: test #this value is a static and it passes correctly, no issues here trigger: project: project-B-path

In project B:

variables: PATH: "" RUN_TYPE: "" test: script: echo "From upstream pipeline dynamic: $PATH" echo "From upstream pipeline static: $RUN_TYPE" ...

When I run it on CI I have the $RUN_TYPE variable correctly passed and empty value in $PATH variable (even though, $BUILD_PATH has the correct value during run of the build job). Tried many approaches - to set the $BUILD_PATH value in before script, to pass environment value (like CI_JOB_ID) to job in project B, to not create this variable at all in project B, etc. Nothing helped, dynamic variable always has empty value.

Then I've tried to save the dynamic var $BUILD_PATH in .env file and to publish it as artefact, so the bridge job could read it from there. I did it like that:

build: script: - some code here - echo "BUILD_VERSION=hello" >> vars.env artifacts: reports: dotenv: vars.env

When I run it on CI job always fails with errors like :

Uploading artifacts... 825vars.env: found 1 matching files and directories 826WARNING: Failed to load system CertPool: crypto/x509: system root pool is not available on Windows 827WARNING: Uploading artifacts as "dotenv" to coordinator... failed id=1877748 responseStatus=500 Internal Server Error status=500 token=some-token-here 828WARNING: Retrying... context=artifacts-uploader error=invalid argument 829WARNING: Uploading artifacts as "dotenv" to coordinator... failed id=1877748 responseStatus=500 Internal Server Error status=500 token=some-token-here 830WARNING: Retrying... context=artifacts-uploader error=invalid argument 831WARNING: Uploading artifacts as "dotenv" to coordinator... failed id=1877748 responseStatus=500 Internal Server Error status=500 token=some-token-here 832FATAL: invalid argument

I've also tried uploading .env file without name as I saw somewhere, like

 - echo "BUILD_VERSION=hello" >> .env

but again no luck, same 500 error. I keep researching this error, but so far - it's with me.

So the point - none of the ways of passing variables to the downstream pipeline in multi-project pipeline worked for me. If anyone met same issues or made it work in a different ways - please help

UPDATE: Resolved this issue in a different way - with cUrl trigger from project A like:

 - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=branchName --form "variables[PATH]=$BUILD_PATH" "https://gitlab/api/v4/projects/projectID/trigger/pipeline"
Improve this question
1
  • 1
    can you post the full solution including project A and B? I am facing a similar issue as you and I still have some confusion, hence that can help. Thanks in advance! Commented Jan 12, 2023 at 15:40

5 Answers 5

Reset to default
7

I was able to do this by using build.env

upstream project:

stages: - build - deploy build_job: stage: build script: - echo var="defined in the job" >> $CI_PROJECT_DIR/build.env artifacts: reports: dotenv: build.env trigger: stage: deploy trigger: project: path/to/downstream

downstream project:

downstream: script: - echo $var needs: - project: path/to/upstream job: build_job

as a bonus, you could also pass along the upstream project's path and branch by adding this block to the trigger job:

variables: UPSTREAM_REF_NAME: $CI_COMMIT_REF_NAME UPSTREAM_PROJECT_PATH: $CI_PROJECT_PATH

and then change the downstream project's needs block to

needs: - project: $UPSTREAM_PROJECT_PATH ref: $UPSTREAM_REF_NAME job: buildinfo
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

3

Your downstream project job needs to declare needs: on the upstream project job.

upstream project:

build_vars: stage: build script: - echo "BUILD_VERSION=hello" >> build.env artifacts: reports: dotenv: build.env deploy: stage: deploy trigger: my/downstream_project

downstream project:

test: stage: test script: - echo $BUILD_VERSION needs: - project: my/upstream_project job: build_vars ref: main artifacts: true
Improve this answer

4 Comments

I've tried this one too, the downstream job fails right away not even staring with error "This job depends on other jobs with expired/erased artifacts". Upstream job doesn't specify any artefacts expire timeout, looks like this error message not fully correct, but I didn't find the reason yet. Did you have this error using this approach?
@OksanaVozniuk the person who triggers the upstream job needs appropriate permission in both projects. But I was able to use this exact same test without issues.
The interesting thing is that I've got wide permissions in both projects, but still researched it, looks like the root cause is somewhere in permissions but a bit deeper. Anyway, it all worked with curl calls, thank you for help
@sytech Maybe about.gitlab.com/releases/2022/04/22/gitlab-14-10-released/… would be relevant here? (for GitLab 14.10+, Apr. 2022)
0

To pass variable values from upstream pipeline to downstream pipeline we can use 'forward' together with 'trigger'

#pass MY_VARIBALE value via CI/CD pipeline as user input #variables that define in upstream pipeline variables: GLOBAL_VARIABLE: "value" upstream-job: needs: - someOtherJobIfNeeded trigger: include: - local: "downstream_gitlab-ci.yml" forward: pipeline_variables: true strategy: depend only: - master downstream_Job: needs: - someOtherJobIfNeeded before_script: - | if [[ -z "$MY_VARIBALE" ]]; then echo "Error: MY_VARIBALE variable not set as CI/CD varibale." exit 1 else echo "$MY_VARIBALE" fi if [[ -z "$GLOBAL_VARIABLE" ]]; then echo "Error: GLOBAL_VARIABLE variable not set at upstream pipeline." exit 1 else echo "$GLOBAL_VARIABLE" fi script: - someOtherScriptIfNeeded.sh only: - master

Ref : https://docs.gitlab.com/ee/ci/yaml/#triggerforward

Improve this answer

1 Comment

please fix the indentation. forward keyword and its pipeline_variables should fall within the trigger block whereas strategy should fall outside forward, but within the trigger block.
0

Have you tried forwarding the pipeline variables? It solved my issue.

deploy: stage: deploy trigger: my/downstream_project forward: pipeline_variables: true
Improve this answer

Comments

0

I found that defining another env var also works, and I tested the trigger:forward keyword mentioned above, it works and it feels like the official correct way to do it ( No need to define extra CI variable if using forward.

stages: - setup - run variables: CUSTOM_VAR: yeah CI_CUSTOM_VAR: $CUSTOM_VAR # it has to be a different variable name generate: stage: setup script: - env | grep CUSTOM # - some script to generate child pipeline yaml artifacts: expire_in: 5 minutes paths: - .child_pipeline.yaml run pipeline: stage: run needs: [generate] # variables: # CUSTOM_VAR: $CUSTOM_VAR # this leads to in child job CUSTOM_VAR=$CUSTOM_VAR not evaluated but if using gitlab push options or manually trigger with CI variblae passed it works trigger: include: - artifact: .child_pipeline.yaml job: generate strategy: depend # This is the official way of doing it # forward: # pipeline_variables: true

And to fake generated child pipeline yaml

stages: - run child job: stage: run script: # both variables appear here (down side), with CI_CUSTOM_VAR which could be the default one or overriden one ( from gitlab push options or gitlab api) - env | grep CUSTOM

This way the CI_CUSTOM_VAR is resolved first to the static value in parent pipeline yaml and it could be override by

  • gitlab push options from command line or

  • from gitlab api

This works in my test in gitlab v16.77-ee

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.