All Questions
Tagged with parallel-computing or parallel-processing
21,953 questions
Tooling
1 vote
2 replies
49 views
using persistent-memory gawk how variables can created to be local and issolated from other execution instances?
The idea of Persistent-Memory gawk is fabulous because it improves the performance, size, and clarity of many scripts on static and reference data. However, I have a significant problem in adopting ...
1 vote
1 answer
68 views
How to share a large CustomObject to workers in Python multiprocessing on Windows (spawn)?
I'm trying to run calculations using multiple cores in Python on multiple platforms (Linux, macOS, Windows). I need to pass a large CustomClass Object and a dict (both readonly) to all workers. So far ...
0 votes
0 answers
31 views
Attribution Error when using Huggingface transformers Trainer with FSDP
I am now trying to use FSDP in Huggingface transformers Trainer. The training script is something like train_dataset = Mydataset(...) args = TrainingArguments(...) model = LlamaForCausalLM....
0 votes
0 answers
19 views
OptimisticLockingException when using multiInstanceLoopCharacteristics for parallel execution of subprocess
I have the following process definition I try to execute on Camunda 7.24 / CibSeven 2.1 which currently logs during execution many OptimisticLockingException. I could already trace it down that it ...
0 votes
1 answer
123 views
Why are items not written to console immediately after being processed?
I have the following C# code : var rand = new Random(1); var range = Enumerable.Range(1, 8); var partition = Partitioner.Create(range, EnumerablePartitionerOptions.NoBuffering); foreach (var x in ...
0 votes
0 answers
46 views
Taking advantage of memory contiguousness in HLSL
This is a bit of a slog so bare with me. I'm currently writing a 3D S(moothed) P(article) H(ydrodynamics) simulation in Unity with a parallel HLSL backend. It's a Lagrangian method of fluid simulation,...
Tooling
0 votes
0 replies
25 views
ComfyUI + Flux 1 dev + limited RAM + same workflow: With 2 GPUs?
I am running Flux 1 dev text to image model through ComfyUI in Kaggle. Everything works but I noticed that Kaggle offers a second GPU inside the notebook. If I try to run two instances of the ComfyUI ...
1 vote
0 answers
74 views
Intuition over TBB parallel scan/parallel prefix requirements
I am reading a paragraph about the tbb::parallel_scan algorithm from the book Intel Threading Building Blocks, and I understood what the operation does serially, but I am not understanding what are ...
0 votes
0 answers
70 views
Simple TBB example where tbb::affinity_partitioner gives a measurable speedup
While looking at this TBB guide webpage: https://www.intel.com/content/www/us/en/docs/onetbb/developer-guide-api-reference/2021-9/bandwidth-and-cache-affinity.html, they mention this ...
Best practices
0 votes
2 replies
126 views
Looping Datasets in R
Essentially I am trying to create a dataset that is dependent on prior rows to generate values for any given row. I then would like to run this loop over many IDs for an entire dataset. Current set up ...
0 votes
0 answers
50 views
Parallel Equations Expansion in TFORM
TFORM is considered a great tool for manipulating large and symbolic equations. In this thread, I’d like to share my optimization problem, which concerns a very simple operation — equations expansion. ...
Advice
1 vote
1 replies
31 views
Is there a way to transfer MBTiles formatted maps faster?
I need to transfer a MBTiles map tiles from a disk to another, is there a faster way then just mv? It is huge and takes time.
0 votes
1 answer
93 views
C++ segmentation fault when throwing in ordered OMP parallel for
the code below crashes with terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >' Aborted ...
0 votes
1 answer
51 views
Parallel Processing in Sclang
I am doing some calculations on spectra, doing some maths on each partial. This is taking a lot of time, but since the partials are all calculated independently, I wonder how to do parallel processing ...
1 vote
0 answers
34 views
run single notebook with different objects lists dict parallel on spark, when called from master notebook in synapse
Anyone know how to call single notebook with different parameters parallel, and all notebooks should appear on spark UI to make the trouble shooting easier? I have one child notebook, calling from ...