0

Consider one Azure agent (self-hosted in my case). Multiple pipelines have access to it and everything works quite well as long as the load is not high. That is, the oldest jobs are executed first, keeping everything a nice FIFO.

However, when the pipelines are under heavy load I see that one pipeline seems to take preference, starving the other pipeline runs. I can manually fix it by clicking "Run next" so a starved pipeline gets to run next. But I'd like to avoid manual procedure, as it's hard to reconcile with the idea of an automated build.

If multiple pipelines have jobs available for one azure agent, which job is run first? Can I change this scheduling somehow?

Research

There's a similar question about which agent is used, if it's the other way around: multiple agents available for one job. It's the oldest agent according to some Reddit thread. In my case, the more recently created pipeline takes precedence.

The docs mentions that agents poll for new jobs. So I grep for poll, job, etc. in _work and _diag, but the matches seem not to be related with the job mentioned in the docs. So I'll next start inspecting network traffic, but I assume the decision for the job is done by the backend of Azure anyways.

Improve this question

1 Answer 1

Reset to default
1

There is no weighting mechanism that you can declaratively apply to your pipelines to change the priority within the queue. Regarding preference for which pipeline is queued next, keep in mind that the queue shows the list of jobs (not pipelines). As you've mentioned the pipeline is first-in-first-out, and perhaps your newer pipeline has multiple jobs that are queued at the same time. This may give the appearance that the pool has a preference to run a single pipeline when it's actually running related jobs from a single pipeline run.

Turning your question on it's head, I assume you're asking because you have a single build agent in a pool and under high load you're seeing important work in the queue being delayed or blocked by other pipelines.

Perhaps a better question to your problem is, how can I prevent important work from sitting idle in the queue when there is heavy load? The most practical answer is to add additional hosts to the pool so that more than one job can be executed in parallel. By increasing the throughput of the pool there is less opportunity for pipelines to be monopolized by a single pipeline.

Obviously, adding additional hosts comes with an infrastructure price tag, and Azure DevOps has a few offerings that can provide cost-effective burstable agents based on load:

  • Running agents in a virtual machine scale set: you can spin up additional hosts based on the queue length
  • Managed DevOps Pools: Microsoft hosts your self-hosted image in the cloud in your virtual network. There are lots of options here for tearing down agents that aren't needed. In some cases, because you only pay for infrastructure while it is in use and agents can be spun down to zero when there are no jobs, this option may be cheaper than self-hosted agents.

A simple alternative you can implement without a major configuration change is to add an additional VM to the pool and keep it in a stopped state when it's not required. The virtual machine will auto-start the agent when the machine boots, making it an easy option to provide additional capacity when needed.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.