4

My pipeline consists of very long jobs that build docker images by compiling a set of packages; all of them require ~ 3 hours on my runner.

I set up my CI so that only one job can run at a time, in order to reduce the disk I/O overhead. But in this way many jobs fail, supposedly because they hit a job start timeout which seems to be bout 1h, with this message:

There has been a timeout failure or the job got stuck. Check your timeout limits or try again

I have not been able to find how to increase the job start timeout, since all the documentation I found is relative to the job duration timeout.

So I’d need help with this, thanks. I'm using GitLab CE 12.8.1 and Gitlab Runner 11.5.1.

Improve this question
1
  • I know what you are talking about. We can paraphrase it using gitlab specific terminology. You want to specify timeout that controls conversion between pending and started job. For now it seems impossible in gitlab. Commented Mar 9, 2024 at 12:08

1 Answer 1

Reset to default
5

There are three types of timeouts in Gitlab CI:

  • Project timeout
  • Runner timeout
  • Jobs timeout

Project timeout

According to GitLab docs:

Timeout defines the maximum amount of time in minutes that a job is able run. This is configurable under your project’s Settings > CI/CD > General pipelines settings. The default value is 60 minutes. Decrease the time limit if you want to impose a hard limit on your jobs’ running time or increase it otherwise. In any case, if the job surpasses the threshold, it is marked as failed

Runner timeout

According to GitLab docs:

For each Runner you can specify a maximum job timeout.

Runner timeout settings are defined in runner's edit page

Jobs timeout

GitLab CI/CD Pipeline Configuration Reference | GitLab

Job's timeout allows you to configure a timeout for a specific job. For example:

build: script: build.sh timeout: 3 hours 30 minutes test: script: rspec timeout: 3h 30m

Precedence of different types of timeout

The job-level timeout can exceed the project-level timeout but can not exceed the Runner-specific timeout.

If runner timeout smaller than project defined timeout, will take the precedence.

Examples of precedence of timeout directive

See Configuring GitLab Runners | GitLab

name: **Example 1 - Runner timeout bigger than project timeout** project_timeout: 2h runner_timeout: 24h job_timeout: 4h resulting_timeout: 4h name: **Example 2 - Runner timeout not configured** project_timeout: 2h job_timeout: 4h resulting_timeout: 4h name: **Example 3 - Runner AND job timeout are not configured** project_timeout: 24h resulting_timeout: 24h name: **Example 4 - Runner timeout smaller than project timeout** project_timeout: 2h runner_timeout: 30m resulting_timeout: 30m name: **Example 5 - Runner timeout smaller than Project timeout, Job timeout is bigger than Runner timeout** project_timeout: 2h runner_timeout: 30m job_timeout: 1h resulting_timeout: 30m
Improve this answer
6
  • thanks for your reply, but as far as I understand these are all runtime timeout, limiting the time a job can effectively run after it is started. But my problem, as I tried to explain in my original post, is that my jobs die because they are not started due to unavailable runners, since my only runner runs one job at each time and the others are killed while waiting to be run. Maybe your reply effectively provides a solution for this case but I misunderstood it? Commented Mar 24, 2020 at 18:07
  • Hi, @nicola-mori. Just set Project timeout to value more than 3h. And leave Runner timeout value empty. See Example 3 Commented Mar 24, 2020 at 18:28
  • Hi, my Project timeout (i.e. that configured in Settings -> CI / CD Settings -> General pipelines -> Timeout) is currently 2d, and my runner does not have a timeout. Does this correspond to what you are suggesting? Or do have to put the settings you are suggesting somewhere (I guess in .gitlab-ci.yml)? Sorry for the noob questions but I'm a beginner... Commented Mar 24, 2020 at 18:57
  • Hi. Please show your .gitlab-ci.yml Commented Mar 25, 2020 at 5:52
  • You can find it here: basket.fi.infn.it:443/f/0de7f11b48/?dl=1 Commented Mar 25, 2020 at 9:11

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.