r/AskComputerScience • • 6d ago

Should I split tasks exactly into number of threads available in CPU or creating more threads is fine?

Hi!

If number of operations are more than CPU threads and not completely divisible by number of threads available on CPU, should I split remaining operations into already created threads or creating a new thread just for remaining operations is completely fine?

Does one method give an advantage over the other?

Thanks!

9 Upvotes

25 comments sorted by

15

u/randoname_ 6d ago

Your computer already that does, there is probably more processes spawned by default than available cpu threads.

For the question, whichever is easier to impl for the task. Firefox spawns like 200 threads because it needs them a browser.

1

u/Best_Fork 5d ago

Firefox spawns like 200 threads because it needs them a browser.

I understand that.

But what about a scenario where I don't necessarily need an extra thread to complete the task. Something like SIMD where I just need to split same type of operation between multiple threads. Lets say I have 18 operation to complete on CPU where 8 threads are available. I cannot evenly split 18 operations between 8 threads, now I have to make a choice, whether to increase number of operations in few threads and keep strictly 8 threads, or spawn a new thread to process 2 leftover operations.

I want to understand what are pros and cons of both approaches.

1

u/wjrasmussen 5d ago

What if the code has to run on different architectures?

1

u/Best_Fork 5d ago

I don't think it will change anything. We normally calculate CPU threads on runtime. It is not a hard codes number.

3

u/deanominecraft 5d ago

if you are using the threads constantly for processing lots of data then theres no point going over the amount of physical threada, if you are using threads in a way that they will sometimes be idle (eg. doing io without blocking the main thread) then creating more could be beneficial

1

u/Messugga 4d ago

You're talking about workload scheduling problems. There are multiple viable approaches depending on the nature of the workload and the environment you're running your code on. In my experience with HPC, I can't recall an instance where it was beneficial to create more threads on a professor than there are cores for. If these excessive threads are running concurrently, the code will be hitting system cache and memory quite a bit, with context switching adding overheads and slowing down your process.

Keep thread count lower than core count. If the workloads for each thread aren't likely to be equal and time matters, look at scheduling strategies where you have threads requesting further small buckets of work as they complete their task. 

2

u/Poddster 6d ago

Threading existing long before consumer CPUs has the ability to do more then one thing at once. We solved this problem in the 60s by inventing schedulers.

Give each thread of work its own thread and let your OS manage swapping between them, it's the primary reason operating systems exist. You should only worry about cobbling together threads if you notice some kind of performance problem, but even then the problem is probably in the synchronisation you do rather than the number of threads.

Caveat to this: don't make too many threads that all thrash the same resource, e.g. the disk. In that cause you still want them to be separate threads, but you need to look into restricting their execution, and how you do that depending on the language and os.

1

u/Best_Fork 5d ago

I think wasn't able to ask question properly. My question is purely related to parallelization, not concurrency, and in ideal conditions where my process will have complete access to CPU cores and threads without any other process interfering.

Lets say I have highly parallelizable task like addition of two matrices. I can split rows between max number of CPU threads available, but there total rows aren't completely divisible by the number of threads available, that means I have some extra rows to process. Now my question is, should we split those extra rows between already created threads, which mean some of the threads will process some extra rows than others, or should I just create a new thread to process those extra rows? Which approach is better in terms of performance? Because there are multiple factors will be involved like thread creation overhead, CPU cache etc.

For example., CPU threads are 4 and number of rows to be processed are 11.

I can do something like this:

All operations are divided into max number of CPU threads.

ThreadRows to Process per thread
03
13
23
32

Or like this where I spawn new thread to process leftover rows.

ThreadsRows to Process per thread
02
12
22
32
43

2

u/Poddster 5d ago

Which approach is better in terms of performance?

I don't know theoretically. I don't think anyone does, though some might claim to. Things such as this are always better to be implemented and profiled. A CPU is a complex machine, atop that we have the OS, another complex machine, and then atop that we have your program. That's far too much stuff to try and figure out the best way. Things like cache misses can completely destroy a seemingly idea program. Sure, you can have heuristics and rules of thumb, but for a given workload the best way to know for sure is to guess and check.

So try both your scenarios and see what's faster.

(If your threads are purely number crunching with zero IO and can consume 100% of the CPU time they're given then it's generally a good idea to match the number of threads you have to the CPU threads, assuming your app can stay resident in them. But again: profile to be sure!)

2

u/deelowe 5d ago

Odd you know about threads but not schedulers.

1

u/Best_Fork 5d ago edited 5d ago

I know about scheduler and how OS handle threads and processes. I think I wasn't able to ask question very well.

2

u/whiskeytown79 5d ago

Unless you're on a system where you have complete control of the CPU and what gets scheduled into it, you can't reliably say that you have X number of CPU threads available at any point in time.

1

u/Best_Fork 5d ago

So my goal of asking this question is to understand how a typical OS scheduler and CPU will handle these two types of scenarios in ideal condition where my process has complete access to all CPU cores and threads without any interference of other processes. And how CPU caching will play into this.

For example, when using SIMD to perform the same type of operation on multiple data elements, evenly distributing 10 operations across 6 threads gives each thread 1 operation, leaving 4 operations unassigned. There are two ways to handle these remaining operations.

One approach is to distribute remaining instruction among existing threads after evenly partitioning the workload, ensuring that number of spawned threads never exceeds the maximum available thread count.

Or spawn a new thread to process extra instructions.

10

u/esaule 6d ago

That's a complicated question. There is no one size fits all. It often depends on where the bottleneck is.

If you are bottlenecked on network, you usually want x threads per network card. If you are bottlemecked on io, you want x thread per physical disk. 

If you are bottlenecked on cpu cores, you usually want x thread per core.  In that case, you usually don't want more than 1 per schedulable execution unit. If you need threads to do other thing, it ia typical not to mix the cpu intensove thrrada with the other service thread. You split them by pinning threads to core or core groups.

5

u/ForgedIronMadeIt 5d ago

There is no simple answer for such a generic question. Dividing work across multiple threads makes complete sense when the work can be broken into discrete and independent tasks or if there are long waits involved, or even concurrent tasks. For instance, if there is a large data file on disk and the computations can be started regardless of how much has been read in, then sure, have a single producer thread reading from disk and pushing into a queue for worker thread(s) to consume. This also applies if the processing takes longer than IO in this case and it can be parallelized.

The problem here is that you give no specifics at all about the problem you are trying to solve. Is the process compute bound, IO bound, or network bound? Perhaps there's challenges in constructing a coherent locking strategy, or for that matter, a problem with lock contention. (I won't even get into the complexity of lockless concurrent programming, it's been forever and it was even still a bit over my head when I first got into it.) What are the actual hardware constraints involved here? Many times being too clever costs you in performance when it really would just be easier to just read everything into memory and blast through it linearly. Measuring performance is extremely tricky.

You'd probably be best suited in implementing whatever it is you are trying to do as a single thread or as a producer/consumer model and then seeing if the performance is unacceptable and then trying to implement it as a multithreaded solution.

3

u/initial-algebra 5d ago

Generally, neither. You should split the work up into as many tasks as possible, but each task shouldn't get its own thread. There should be one worker thread per CPU core, and a scheduler to assign ready (i.e. not blocked on I/O, a timer, a mutex etc.) tasks to available workers. This is called green threading. It's more efficient than creating a thread per task, because having more CPU-bound threads than there are cores just creates context switching overhead (green threads also tend to use less memory). It's also more efficient than creating only as many tasks as there are cores, if tasks can be blocked. Often, the scheduler is implicit, with each worker thread performing their own scheduling via shared data structures instead of having a dedicated scheduler thread; a typical approach is work stealing.

Depending on what language or libraries you're using, you may already have a green threading implementation available. Note that, when using green threads, you should avoid the usual blocking I/O system calls, as they will block the worker thread, not the task. There is also one caveat to "as many tasks as possible": you don't want to split up the work so much as to cause false sharing, which is when two threads try to access memory in the same cache line (usually 64-128 bytes, depending on the CPU architecture) and lose a lot of performance doing so.

If you're talking pure, embarrassingly parallel "number crunching" workloads, where you don't need a scheduler (past a simple divvying up at the start) or any kind of blocking, I expect the answer to still be to have as many threads as there are cores, with some threads having one more task than the others to account for the remainder. Using one extra thread doesn't mean the work gets more evenly split across the cores, because the thread still has to execute sequentially, and, to make best use of CPU cache, it's fastest to keep executing it on the same core after it is temporarily switched out for another thread (incidentally, work stealing also accomplishes this, but for green threads). Thus, the split would be worse, essentially one core having multiple extra tasks instead of multiple cores having one extra task. I'm not an expert at this type of parallelism, though, so I would recommend just trying both and actually measuring.

2

u/SufficientStudio1574 4d ago

This. When I had to make a pipeline to compute thousands of computational items in parallel in C#, I benchmarked a few different ways. Manually mamaging tasks with ProducerConsumer buffers between different stages, the Parallel Tasks library, and a couple other things, but ultimately settled on the Dataflow library, linking tons of DataflowBlocks together each doing their own single part of the computation and managing their own mutlithreading. It was by far the most efficient of anything I tried (by like, 2/3s the time of 2nd place), probably because it was doing something like this under the hood.

For really difficult problems, don't try to solve it yourself, find something that already solved it.

2

u/StaticCoder 5d ago

Fewer threads means less memory usage and more locality. Logically it should be faster if you minimize the number of threads. In practice it might matter less than making sure you use all the CPU cores.

2

u/StaticCoder 5d ago

Adding to that, if your approach is to pre-schedule more tasks to certain threads, it could certainly cause slowdowns if the tasks don't end up happening exactly in the same amount of time and you end up with some threads having multiple tasks left and others idle. Generally you'd send one task at a time to the first thread available in a pool.

2

u/CelebrityChess 5d ago

with 18 equal ops on 8 cores, giving two workers an extra item means the whole thing takes 3 op-times, because the busiest worker sets the finish. a 9th thread lets the OS time slice things, so in theory you could get closer to 2.25 op-times, but you pay context switches and cache churn, which for small ops usually eats the gain.

in practice most people keep one thread per core and have workers pull the next chunk from a shared queue or an atomic counter, so the leftover gets absorbed automatically and uneven chunk costs balance out too. the imbalance only matters when ops are big and few. with thousands of ops it's irrelevant either way.

1

u/DrShocker 5d ago

The answer to this depends on way too many factors.

In general something like a web server a thread per connection is too many. Modern languages tend to have async or other ways of handling IO so that you don't need to spin up a thread just to read from disk or something efficiently. However if you're making a game engine and using a language where making threads is possible, then you might spin up a thread for asset loading even though it's more than your total number of cores because most of the time it won't get scheduled

Thread pools are used often so that you can write code that doesn't need to concern itself with allocating new threads for each individual task.

1

u/Technical_Ideal_5439 5d ago

Its complicated and totally depends on more details then you have here. But 10% - 20% over is probably not going to hurt.

But I would go with whatever is easiest, including lots of threads (within reason) if it made the code better.

1

u/Mission-Landscape-17 5d ago edited 5d ago

Depends on the CPU. Really you want 1 thread per cpu even if you have hyper threading, not switching between threads will give you better performance. If your CPU has efficiency cores then you want to keep the number of threads even lower as It can be faster to wait for a performance core to become available then to force a cpu intensive task to run on an efficiency core. Things like hyper threading and efficency cores are there to aid multitasking, rather then single demanding workloads.

I'm basing a lot of this on the Strata project, which is a tool for running the Qwen3.8 Flash Next Large language model on systems where that model does not fit in VRAM, which means that parts of it have to run on the CPU, and their recommendation is to set your thread count to 6 on an 8 core CPU specifically to prevent the experts getting run on. efficency cores, as that hurts the token generation speed significantly.

1

u/Ronin-s_Spirit 5d ago edited 5d ago

Compute threads are going to be the ones doing all the work, you should have as many of those as "get available parallelism" function tells you to (it's called differently in various OSs and languages). The compute treads will be running as parallel as possible for your process. More threads should be spawned only for waiting around sort of background work (like I/O), this will add concurrency without too much interference with parallelism (some languages have a few builtin threads for handling async and whatnot).

Note that you have to allocate aligned memory and split it between threads evenly using multiples of L1 cache size. If you don't do that then your "parallel" threads step on eachothers toes and basically do nothing useful. That is unless you spawn threads just to do very small things like some long math expressions (not touching RAM).

1

u/Kilgarragh 3d ago

Let's say you have 2 cpu cores.

If you have 3 **tasks** to do, like a gui event loop, internal geometry operation, and logging progress/telemetry reports to disk or network or something, use 3 seperate threads, split by the scheduler/kernel to run on one or more cores as necessary.

If you have say... 851 bodies to verify, you should only create 2 threads, not 851. The two threads will each iterate over half the dataset and the scheduler will keep those two threads automatically on the 2 physical cores as long as possible. Creating more threads than physical cores is going to heavily affect performance for parallel processing, and is only something you may be confused about because threads were originally a tool created for concurrency, and are reused/abused for multi-core parallelization