322

I'd like to use GitLab CI with the .gitlab-ci.yml file to run different stages with separate scripts. The first stage produces a tool that must be used in a later stage to perform tests. I've declared the generated tool as an artifact.

Now how can I execute that tool in a later stage job? What is the correct path, and what files will there be around it?

For example the first stage builds artifacts/bin/TestTool/TestTool.exe and that directory contains other required files (DLLs and others). My .gitlab-ci.yml file looks like this:

releasebuild: script: - chcp 65001 - build.cmd stage: build artifacts: paths: - artifacts/bin/TestTool/ systemtests: script: - chcp 65001 - WHAT TO WRITE HERE? stage: test

The build and test jobs run on Windows if that's relevant.

Improve this question

5 Answers 5

Reset to default
256

Use dependencies. With this config, the test stage will download the untracked files that were created during the build stage:

build: stage: build artifacts: untracked: true script: - ./Build.ps1 test: stage: test dependencies: - build script: - ./Test.ps1
Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Finally got it to work! The key point here is dependencies should be used along with artifacts. Only the artifacts that were included would be available for consumption in subsequent stage. Needless to say, be conservative on what is being uploaded. I would say use expire_in. Otherwise we could end up wasting lot of storage. These artifacts are uploaded to gitlab in the build job/stage/step and downloaded in test.
Do you really have to use dependencies? Gitlab documentation states Note that artifacts from all previous stages are passed by default.. The question is when do you need to use dependencies.
The documentation clears this up quite well: docs.gitlab.com/ee/ci/yaml/#dependencies
@Josef artifacts from all previous stages are passed by default (Not from the previous jobs)
the documentation is confusing - what does it mean that artifacts of "all previous stages" are being passed? If I understand it correctly, if all previous stages are always passed, then all the data from all the jobs will be available and I do not need to use the "dependencies" tag. The only case I can think of is, if I DO NOT want that behavior and only pass artifacts from one or a selected few jobs.
|
252

Since artifacts from all previous stages are passed by default, we just need to define stages in correct order. Please try the example below, which could help understanding.

image: ubuntu:18.04 stages: - build_stage - test_stage - deploy_stage build: stage: build_stage script: - echo "building..." >> ./build_result.txt artifacts: paths: - build_result.txt expire_in: 1 week unit_test: stage: test_stage script: - ls - cat build_result.txt - cp build_result.txt unittest_result.txt - echo "unit testing..." >> ./unittest_result.txt artifacts: paths: - unittest_result.txt expire_in: 1 week integration_test: stage: test_stage script: - ls - cat build_result.txt - cp build_result.txt integration_test_result.txt - echo "integration testing..." >> ./integration_test_result.txt artifacts: paths: - integration_test_result.txt expire_in: 1 week deploy: stage: deploy_stage script: - ls - cat build_result.txt - cat unittest_result.txt - cat integration_test_result.txt

enter image description here

And in case to pass artifacts between jobs in different stages, we can use dependencies together with artifacts to pass the artifacts, as described from the document.

And one more simpler example:

image: ubuntu:18.04 build: stage: build script: - echo "building..." >> ./result.txt artifacts: paths: - result.txt expire_in: 1 week unit_test: stage: test script: - ls - cat result.txt - echo "unit testing..." >> ./result.txt artifacts: paths: - result.txt expire_in: 1 week deploy: stage: deploy script: - ls - cat result.txt
Improve this answer

9 Comments

Very clear explanation, thank you. If a stage names an artifact by the same name as an artifact from a previous stage, does the original artifact get overwritten?
@MichaelOsofsky You can name the artifact by the same name, the original artifact will not get overwritten by the one from next stage with the same name. The next stage only downloads the artifact from the former stage, it is a copy of it. I name them differently in the example mainly due to unit test and integration will be executed in parallel. If we remove .e.g integration test job, all jobs will execute in sequence, then we can use the same name for all the artifacts without any confusion. FYI, I update the answer with one more example.
In your example I see you're appending to result.txt. If you overwrote result.txt in the job unit_test, I assume the job deploy would never have access to the contents of result.txt from the job build. I'm just asking to ensure I never cause this type of bug in my scripts.
According to the log, the deploy stage will download both result.txt from build and test stages, but the later one will overwrite the former one.
BTW, the original artifact is not touched and always available for downloading from CI/CD -> Pipelines, then click the dropdown button for Artifacts on the right, you will find all the artifacts of all stages.
|
14

If you want foo/ to be available in the next stage AND it is in your .gitignore you'll need to list it in the artifacts of the stage that creates it, or as explained at here use untracked: true. As I wanted just a subset I didn’t use untracked: true.

The following approach worked for me (with NO dependencies specified in the following stage)

 artifacts: paths: - foo/ expire_in: 1 hour

BTW regarding the : expire_in: 1 hour part:
I read at https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2133 there's no way to get artifacts to automatically expire at the conclusion of pipeline and the default retention was surprisingly long (30 days by default) - hence the time-based kludge to get rid of them - see https://docs.gitlab.com/ee/ci/yaml/

Improve this answer

1 Comment

➕1️⃣ for mentioning the expiration info 🙂
8

If you use the needs: keyword, the default artifact management behavior changes. The only artifacts available are from "upstream" jobs on the needs graph. Also, the dependencies: keyword cannot be used with the needs: keyword.

To address the pipeline snippet from the question, adding a needs relationship to the job that creates the required artifacts is all that is needed:

releasebuild: script: - chcp 65001 - build.cmd stage: build artifacts: paths: - artifacts/bin/TestTool/ systemtests: script: - chcp 65001 stage: test needs: - job: releasebuild artifacts: true

NOTE: The needs:artifacts: keyword defaults to true and can be omitted. When set to false, the job won't load the upstream artifacts.

Improve this answer

Comments

4

You can use needs section. Docs needs.

releasebuild: needs: [] # Pass there dependencies that needs to start this job or leave it empty ... artifacts: paths: - artifacts/bin/TestTool/ systemtests: needs: ["releasebuild"] script: # Now you can do whatever you want with artifacts/bin/TestTool/ here # Example: - cp artifacts/bin/TestTool/* test/* ...

The same for needs:artifacts, just another syntax

releasebuild: needs: [] # Pass there dependencies that needs to start this job or leave it empty ... artifacts: paths: - artifacts/bin/TestTool/ systemtests: needs: - job: releasebuild artifacts: true script: # Now you can do whatever you want with artifacts/bin/TestTool/ here # Example: - cp artifacts/bin/TestTool/* test/*

This is pretty same that dependencies ones.

Improve this answer

1 Comment

Actually it's not the same and you need to be very careful when you mixing them. I even made a blog about difference in their behaviour - dev.to/zavoloklom/…

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.