2
$\begingroup$

I went through a lot of Vulkan tutorials and each of them has talked about fences, which is good, but each of them does not clearly tell what exectly happen on GPU. It is so frustrating trying to learn Vulkans synchronization....

As far as I understand, I have CommandBuffer which can be executed by vkQueueSubmit. In this submit I can add a fence.

The fence is signaled when the execution of all commandBuffers is finished. So I can check the fence status on the CPU to see the progress of the submit with vkGetFenceStatus or vkWaitForFences.

But what happens to the GPU execution flow? Let's take a look at the following scenarios:

Scenario1:

Suppose I have recorded multiple commandBuffers and submit them in a single operation.

  • vkQueueSubmit(commandBuffer1, commandBuffer2, commandBuffer3, commandBuffer4, fence1)

On the GPU, the queue is filled with commandBuffer1-commandBuffer4. I have read that the start of execution of the commandBuffer is ordered. So commandBuffer1 is executed before commandBuffer2. CommandBuffer2 before CommandBuffer3... But the GPU can do this in parallel, so it is not guaranteed that commandBuffer1 has finished its work before commandBuffer2 starts executing. It is therefore also possible that commandBuffer2 has finished its work before commandBuffer1 has finished.

Fence1 is signaled when commandBuffer1,2,3 and 4 have finished their work. See Figure 1

enter image description here

Figure1: Example of the execution of commands within a GPU with a fence

Scenario2:

As in Scenario1, we have 4 commandBuffers that are to be transmitted. This time, however, they are transmitted in separate transmission commands, each with a fence.

  • vkQueueSubmit(commandBuffer1, fence1)
  • vkQueueSubmit(commandBuffer2, fence2)
  • vkQueueSubmit(commandBuffer3, fence3)
  • vkQueueSubmit(commandBuffer4, fence4)

The start of execution is still ordered, so commandBuffer1 starts before commandBuffer2... same goes for commandBuffer3 and commandBuffer4.

But what happens on the GPU? Is commandBuffer2 blocked by the fence until commandBuffer1 is finished? Or does a fence not block at all?

In case that Fences do not block GPU commands: In the case of Figure 1, does this mean that Fence2 signals first, then Fence4, Fence1, Fence3?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

The GPU does not block on a fence, it can only update the status of the fence, and even then it will not block, it just updates it and keeps going.

The CPU does block on a fence though. The calls to wait on a fence like vkWaitForFences always blocks but can have a timeout so the CPU can make forward progress.

In Vulkan commands start in order, but can finish out of order. Any fence that has been set to signal the completion of work will be signaled in the order that command finishes. So in the given example the fences will be signaled in the order 2,4,1,3. But none of those fences will block the GPU, it will just keep on going until it hits one of the GPU blocking commands like a semaphore, waitidle, ect.

Some fence sitting:

I like to think of fences as a way to keep tabs on the progress of the GPU. The CPU can choose to wait for a fence, or just check its status. (in both cases I always use the vk fence wait command with different timeouts), but I am guaranteed the GPU isn't going to block on any fence. This can be handy for debugging as we can spin loop checking fences and printing out the completion order and other GPU progress checks. It can also be handy for learning to work with fences and semaphores. For example I have intentionally created deadlocks on the GPU, when the program deadlocks I break it and look at the state of data, or just use the deadlock to say ugh, I got into a state that should never have happened, like fence A finished before fence B indicating a problem with a semaphore for example.

Fences sound fabulous for doing real work, but as program complexity goes up we end up juggling fences in ways that can become wildly complicated. The solution for managing the fence nightmare the ensures is to create a counter that increases over time. Then wait for every fence associated with a counter to complete. Then we can just write the code so that we can wait on counters instead of the associated fence. But that becomes pointless since timeline semaphores do that as part of their own built-in functionality. This has relegated fence operations to swap image management, and even then only because we can't use timeline semaphores.

Unfortunately there is a ton of tutorials that use fence/semaphore combos to do real work, and there are probably as many programs that are copying those tutorials. While there are almost no good tutorials on using timeline semaphores.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you very much! now I finally understand how to synchronize my program properly without the GPU being idle =) +1 $\endgroup$ Commented Jul 10, 2024 at 11:09
  • 1
    $\begingroup$ "This has relegated fence operations to swap image management, and even then only because we can't use timeline semaphores" I find this decision baffling. I guess they didn't want to have the timeline semaphores extension interact with the swapchain extension, but honestly it's so frustrating. $\endgroup$ Commented Jul 10, 2024 at 21:15

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.