0
$\begingroup$

I am writing a program that uses a transfer queue and a compute queue. As the names suggest, the transfer queue loads data from the CPU to the GPU and from the GPU to the CPU and the compute queue uses this data to perform some calculations and also to update them.

The general concept is that I have two staging buffers:

  1. stagingBufferToGPU (VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT) with TRANSFER_SRC_BIT. This buffer should load data from CPU to GPU.
  2. stagingBufferToCPU (VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT) with TRANSFER_DST_BIT. This Buffer should load data from GPU to CPU.

The data stored in the stage buffers is copied to memory with VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT for faster calculation.

Since the data I want to calculate comprises tons of GigaBytes, the entire calculation is split up. The staging buffers are therefore used several times.

Each of my buffers has a VkSemaphore that signals whether the buffer is currently in use. So the idea is that when a calculation command is sent, it waits for the signal from the semaphore. After the calculation command has completed its work, it will send a signal to the semaphore again. The same applies if the buffer is used by a copy-submit...

Now the problem:

The staging buffers are also used by the host. The CPU loads data into the stagingBufferToGPU in the following way:

void* mappedData; vkMapMemory(m_device, m_bufferMemory, memoryOffsetGPU, dataSize, 0, &mappedData); memcpy(mappedData, data, dataSize); vkUnmapMemory(m_device, m_bufferMemory);

same with the stagingBufferToCPU:

void* mappedData; vkMapMemory(m_device, m_bufferMemory, memoryOffsetGPU, dataSize, 0, &mappedData); memcpy(data, mappedData, dataSize); vkUnmapMemory(m_device, m_bufferMemory);

How can I ensure that no operations are performed while these assignments are taking place? Do I need a VkFence for this? Or can I use the VkSemaphore that I already have?

Something like:

waitForSemaphoreAndDesignalIt Do memory mapping signal semaphore

BTW: I'm new to Vulkan, so please bear with me =)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Vulkan timeline semaphores replace binary semaphores very nicely when a semaphore is needed. However in this situation memory barriers are probably the synchronization mechanism of choice. The flow of data would be: copy data to mapped memory->memory barrier->compute shader->memory barrier-> copy back.

The CPU would control breaking the data into chunks to be operated on, and each chunk then goes through the data flow shown above. Once that flow is done, another chunk can be started.

Mapping the memory would typically be done at the start of the operation, the memory stays mapped for the duration and when the entire operation is done, the memory is unmapped.

The chunk of mapped memory for the GPU can be large and broken down into smaller chunks on the CPU to optimize data flow and compute shader run times. So you could map 128 megs of memory for example, then break that down into several multi meg sub buffers, like 8 16 meg buffers. Then get 8 threads running the data through compute shaders.

oops, almost forgot the most important part:

At the end of each chunk operation is where a semaphore would come into play. For debugging I recommend a vkqueuewaitidle just to get the entire above process debugged to life. Then when that is working as expected replace the wait idle with a timeline semaphore. The timeline semaphore will allow each chunk to effectively have an ID and the timeline semaphore just waits for whatever ID it is assigned. In a multi threaded system this would make the ID counter the only point of contention between the threads so should work nicely.

One last note, using a timeline semaphore like I just outlined could end up serializing that data operations, if the data is all independent then using multiple timeline semaphores, one per thread for example, would allow individual threads to operate on individual chunks independent of other data.

So the choice depends on how that data can be processed.

Here is a decent intro to timeline semaphores.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks a lot! So the next I'll do is check out memory barriers... $\endgroup$ Commented Jul 6, 2024 at 10:13

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.