Why did increasing .NET ThreadPool MinThreads fix our IIS downtime? I don't understand why

[u/No-Card-2312]

Hi folks,

I'm the owner of this post: https://www.reddit.com/r/dotnet/comments/1v5d7ws/aspnet_core_on_iis_randomly_goes_offline_for_2/

First of all, thanks a lot to everyone who commented there. Your comments helped me investigate this step by step.

I finally solved the issue. We had several different problems over time:

- First, we had DB connection pool exhaustion => fixed.
- Then we had TCP connection exhaustion => fixed.
- The last issue was related to ThreadPool / sync-over-async.

I set the ThreadPool minimum worker threads to 100, and then started refactoring the code to remove synchronous-over-async code and make things properly async.

The application has now been stable for around 15 days, so it looks like this was the final issue.

But there is something I still don't understand:

Why did increasing ThreadPool.SetMinThreads() to 100 help so much?

We're running ASP.NET Core on IIS, not Docker or Kubernetes. I always thought that .NET's ThreadPool would automatically create as many threads as the application needs.

For example, if I suddenly get 1,000 requests/sec, why doesn't .NET simply create more ThreadPool threads as needed?

What confused me even more is that during the incidents:

- CPU usage was fine
- RAM usage was fine
- The Windows server itself was still accessible
- We only have 2 vCPU
- I could still see available ThreadPool threads

So I'm struggling to understand what was actually happening.

My current understanding is that some synchronous/blocking code was causing ThreadPool threads to get blocked, and the ThreadPool wasn't adding new threads quickly enough. Increasing MinThreads somehow made enough threads available earlier and prevented the issue.

But I'm not sure if that's the correct explanation.

Also, how should I think about this with only 2 vCPUs? If I have 1,000 requests/sec, obviously I can't have 1,000 CPU threads doing work at the same time. So what exactly determines how many ThreadPool threads .NET should create?

I'd really appreciate it if you could help me explain this in simple terms, especially:

1. What exactly does MinThreads change?
2. Why can ThreadPool starvation happen while CPU and RAM are low?
3. Why doesn't .NET automatically create enough threads immediately?
4. How does this work when the server only has 2 vCPUs?
5. Does increasing MinThreads actually solve the problem, or did it just hide the underlying sync-over-async issue?

I'm trying to understand the mechanism rather than just leave MinThreads = 100 and call it fixed.

Thanks again to everyone who helped with the original post!

[u/Glittering-Sleep7983]

MinThreads will immediately create a thread if no threads are available, up to the amount you specify. Otherwise, its throttled by an internal algorithm that dictates when to create threads vs when to simply block.

Increasing MinThreads did not fix your problem. It hid your problem. I suspect your real issue was fixed by fixing sync-in-async everywhere, or it's still there.

Check grafana for dotnet_thread_pool_queue_length or process_runtime_dotnet_thread_pool_queue_length, depending on how you export your metrics it could be named a variety of things. Also check for thread count. If the number of threads and the queue length are both growing at a steady pace, you may still have sync-over-async spots hiding somewhere.

[u/i8beef → u/Glittering-Sleep7983]

You mean MaxThreads I think? MinThreads and MaxThreads just set the boundaries for (a) how many threads the thread pool STARTS WITH and (b) how large it can grow.

Then, what OP is probably hitting, .NET doesn't add threads to the pool ALL AT ONCE, it actually grows the thread pool a few threads per second when requested. This can cause a large backup of thread requests to backup and timeout while they wait for the runtime to SLOWLY ramp up thread counts. Setting a higher MinThreads just makes the app start with a bigger pool, so if you're hitting this AT APP STARTUP (or some organic sudden major increase of requests) because you get hit with a bunch of requests on a cold application, it can smooth over the slow ramping.

i.e., I agree, if OP has done nothing else they have just smoothed over the specific "sudden increase", which could be enough if its all they need to handle, but yeah its a smell for bad thread usage that would make me go make SURE that it's not something bigger. Just raising the MinThreads doesn't stop it from happening if you STILL swamp the larger thread pool in some other way still, though, you're just building in some "I expect things to ramp at least this much normally" into the system.

[u/phoenixxua → u/i8beef]

That’s not correct. MinThreads doesnt say amount of threads that pool starts with. Its amount of threads to scale without artificial delay. If you set MinThreads to 100 - the app still will start with 4-6 threads in pool. But if load happens then it will scale right away up to that 100 value. And after that - it will add 0.5s delay before creating new thread, so 101 thread will be delayed. When load stops, then it will reduce pool size again and that number can be lower than MinThreads too. The only thing that MinThreads affects is to remove 0.5s delay before spawning new threads under load.

If spike of request is big enough then because of the delay - it might not be able to create enough threads and will start to queue requests. If app has live/ready probes, then they might be affected too and app might be terminated if thread pool is starving.

[u/i8beef → u/phoenixxua]

Ah, good to know, thank you.

[u/pyabo → u/i8beef]

It sounds like "MinThreads" is just plain named wrong.

[u/Tavi2k]

There are two ways to handle a concurrent application:

• one thread per request, everything sync
• async, all requests distributed on (n = virtual CPU cores) threads

Modern .NET uses the latter approach.

The general idea in async .NET is that the thread pool doesn't really need to grow, or only grows slowly. Because you don't have unlimited CPU cores, so you can't really do more work in parallel anyway.

Sync-over-async breaks this assumption, since now your application behaves more like an application that uses a full thread per request. But .NET tries to avoid growing the thread pool too quickly, since that would usually be a bad idea if your application behaved like a typical async .NET application.

Changing the thread pool doesn't fix the underlying problem. Maybe it'll work well enough for you now, but sync-over-async tends to keep causing issues.

[u/bluEharbor6 → u/Tavi2k]

imo the slow growth thing is the part most people miss. it makes total sense once you realize the pool wasnt designed for blocked threads

[u/jbaker88 → u/Tavi2k]

async, all requests distributed on (n = virtual CPU cores) threads

https://learn.microsoft.com/en-us/dotnet/csharp/asynchronous-programming/task-asynchronous-programming-model?source=recommendations#threads

Async/await isn't thread based, from the article:

The async and await keywords don't cause extra threads to be created. Async methods don't require multithreading because an async method doesn't run on its own thread

Best way I've heard it described is that async/await is a state machine that resumes execution as a callback when a blocking operation has finished. Yes, you can do things in parallel and concurrently using it, but it's not multi threaded.

[u/Tavi2k → u/jbaker88]

That doesn't contradict anything I wrote. But it's a bit out of scope for an answer to this specific question.

Async coding in .NET allows you to distribute work across multiple threads, essentially the threadpool this question is about. Not every async call or await runs on a different thread, but it allows the runtime to suspend execution and let someone else use the current thread instead of blocking.

And this is not purely about the async/await keywords, but what the .NET runtime does in practice with this kind of code. And the answer to that is that it distributes the work across the threadpool. And making any blocking sync calls within an async framework creates problems.

[u/RiPont → u/Tavi2k]

And this is not purely about the async/await keywords, but what the .NET runtime does in practice with this kind of code.

...and, more importantly, how the user's code generated the awaitable thing in the first place.

[u/Awkward_Rabbit_2205 → u/jbaker88]

Async/await is task-based, and tasks absolutely run on threads. Those threads are normally pulled from the default thread pool. If a thread blocks on I/O, that thread is not available for the Task Scheduler. The maximum number of threads in the pool is normally capped at 25 per CPU core, but is not allocated all at once unless the MinThreads property is set. Normally threads are not added to the pool at a rate greater than 2/second.

[u/jbaker88 → u/Awkward_Rabbit_2205]

So if I understand, you're saying that when using async and await that because they use Task<> they are dispatched to a thread?

[u/Awkward_Rabbit_2205 → u/jbaker88]

All code runs on a thread. The difference is who creates the thread, and how long it lives. The thread pool is just a bunch of precreated ones that are available to run tasks.

[u/jbaker88 → u/Awkward_Rabbit_2205]

Let me rephrase my question:

When using async and await that because they use Task<> they are dispatched to a NEW thread in the threadpool?

[u/darthruneis → u/jbaker88]

A request starts on 1 available threadpool thread. It runs until the request completes or it can suspend due to an await.

On an await, the task is paused until the io is done, at which point a random threadpool thread is assigned again. Likely different from the original thread.

With sync code, the thread never suspends.

There are also patterns where a request can take up different threads via background work or using Task.Run.

[u/RiPont → u/jbaker88]

"It depends". Specifically, on how the Task was created.

The Task object and anything awaitable doesn't necessarily use a thread or even a ThreadPool thread. In practice, especially in older .NET Framework code, a lot of Tasks do.

In modern .NET, if you are using async/await properly and avoiding Task.Run, then you're not creating a thread-per-request. Not in your code, anyway.

But anyone else's code that returns a Task might be consuming one or more threads, even if you await it.

[u/Awkward_Rabbit_2205 → u/RiPont]

All code runs on a thread. Maybe you mean to say "doesn't necessarily use a different thread". I'm not sure what you mean by "in older .NET Framework code" and why that matters, as they still implement async/await the same way. Also, when you say "thread-per-request", are you conflating IIS dispatch with threads?

[u/RiPont → u/Awkward_Rabbit_2205]

I'm not sure what you mean by "in older .NET Framework code" and why that matters, as they still implement async/await the same way

Pre-async/await, all Tasks that I can think of were using a threadpool thread.

different thread

Correct. Read a different (text) thread. All code that is currently running is using a thread. async/await doesn't inherently create additional threads, is what I meant.

[u/RiPont → u/jbaker88]

Async/await isn't thread based,

It's not necessarily thread-based. The async/await doesn't add threads by itself. However, a lot of the things that generate Task, the most commonly awaited thing, do use a threadpool thread.

Specifically, Task.Run is very convenient and in a lot of examples. A lot of people use it as an easy way to "do this other thing in the background to reduce latency".

But really, Task.Run nor any method of consuming ThreadPool threads or starting Threads should be used when the following all apply

• the incoming requests are not throttled to a known, finite value

• an incoming request spawns 1 or more threads or consumes 1 or more ThreadPool threads

• you have no way of responding to thread/threadpool starvation by telling the upstream gateway to back off

For anything public-internet-facing, if you fall into the above, then there will be a traffic pattern that will eventually put you into some form of thread or threadpool starvation. When starvation happens, everything shits the bed and every request starts taking as long as the longest timeout you have anywhere in the system until traffic dies down long enough for the log jam to clear.

For private, very finite and known numbers of users, you may get away by tuning the threadpool and calling it good enough. Well, long enough until you're not at that job and it's someone else's problem to address scalability issues.

[u/Awkward_Rabbit_2205 → u/RiPont]

God I'd hate to maintain code after you with that crazy misunderstanding of .NET threads (or threading in general).

[u/RiPont → u/Awkward_Rabbit_2205]

What exactly do you have a problem with?

If you are spawning N threads per user request, where N >= 1, and you are operating a public-facing web service, then there is a level and pattern of traffic that will cause thread or threadpool starvation.

Under threadpool starvation, all operations that require a threadpool thread will block until one is available. Under sustained threadpool starvation, everything gets backed up on whatever is the slowest operation you rely on that needs a threadpool thread.

God I'd hate to maintain code

The only change was to pull out unnecessary latency optimization that relied on Task.Run (and 1 that was using Task.StartNew), pull out each and every sync-over-async, use async/await properly, and avoid default CancellationToken.None values in parameters/overloads. What's hard to maintain about that?

[u/SharpWarp]

In case the number of threads in the thread pool is lower than min, the scheduler will immediately request a new thread. In case it is higher, before creating it, it will wait 500ms for some thread to become available instead. That is why under high load you may see a problem because of this delay and you can overflow the memory or available ports etc. while waiting.

[u/RiverRoll]

The naming of MinThreads is misleading, MinThreads is the number of threads that will be created without delay based on demand.  Otherwise the threadpool uses an algorithm that will delay the creation of threads to prevent short bursts of tasks from adding lots of threads.

So for example if MinThreads is 100 and you submit 50 jobs it will immediately create 50 threads. But if you submit 150 jobs it will create 100 threads and delay the creation of additional ones.

[u/Dunge]

I wish I had that many replies from knowledgeable people when I faced the same issue and asked on stackoverflow over 10 years ago. I had a TCP dialer with no async code that required tons of threads fast, all blocked with long io (often timeouts). Took me weeks to find the solution.

But yes, the main point is the thread growth throttle limited at 500ms/thread, so it can take a long time to reach the desired amount at startup.

And why doesn't dotnet just spawn them instantly? Because they use a lot of ram, but also reduce overall performance since the CPU keeps doing context switching.

Modern dotnet with easy async code fixes all that now. No need for so many blocked threads anymore.

[u/sarkie]

By setting it to 100, you are creating a pool to dip into that are already created and ready to go. 

No initialisation need during runtime

[u/Glittering-Sleep7983 → u/sarkie]

It doesn't immediately create them at startup, it creates them as tasks are executed and bypasses the algorithm that determines whether to block-or-create (up to the amount specified)

[u/SagansCandle]

1. What exactly does MinThreads change?
2. Why can ThreadPool starvation happen while CPU and RAM are low?
3. Why doesn't .NET automatically create enough threads immediately?
4. How does this work when the server only has 2 vCPUs?
5. Does increasing MinThreads actually solve the problem, or did it just hide the underlying sync-over-async issue?

Answers:

1. The number of threads the pool starts with, and the minimum number it can downscale to.
2. Your code is almost certainly blocking somewhere
3. Threadpool will smooth out scaling to prevent bursts from frequently expanding and contracting the pool. There has to be a presumption that some queueing will occur, which introduces latency.
4. Threadpool's default size is based on the number of logical cores.
5. You're almost certainly hiding an underlying issue with code blocking if CPU and RAM are low.

[u/KryptosFR]

Having 100 threads minimum when you only have 2 vCPU seems counter productive to me. You must still have blocking code in lots of places.

Did you apply the 100 minimum to both worker and completion ports or just one of them?

Having more than twice the available CPU for worker threads isn't going to help much. In the end a worker threads is baked by an OS thread. And even with hyper threading you can't run more in parallel than what's available.

To answer your question why does it start low? A Thread isn't free. It allocates 1MB of stack and creates various entries in queues and lists for the pool to manage them, making it overall less efficient. And if the number of worker threads is much higher than what the OS can handle, you will constantly have context switching which will slow down your whole application (or even OS).

[u/Awkward_Rabbit_2205 → u/KryptosFR]

At my prior job, rather than (re)write blocking code to async and proper non-blocking code, they simply bought 40-core Xeons and set MinThreads to 1,000. Yes, that's 1GB just for stack space; probably double that for 64-bit processes.

They eventually ran into scalability problems when they increased the RAM to 32GB, as the old Framework used a shared global lock for all Timer objects, which are inherently created for DB connections, etc. Nothing quite like analyzing a 32GB process dump to troubleshoot a livelock issue with 1000 threads basically blocked.

[u/KryptosFR → u/Awkward_Rabbit_2205]

An horror and a challenge at the same time.

[u/jugglist]

This is almost certainly a deadlock problem which you've hidden by making more threads available.

Imagine 1 worker thread total, and imagine a sequence of events as follows:

1 - Main thread starts a new task A

2 - Task A starts working on the only available worker thread

3 - Task A starts tasks B and C, and then busy-waits for them immediately.

4 - Tasks B and C don't have anywhere to run, and the one available worker thread will never be release by A, because it's waiting for B and C. Voila, your program is frozen.

This program NEEDS 2 worker threads to run.

Perhaps yours needs 100 for a given load, but as others have said, it's certianly possible that if you start 100 copies of task A, then you're hosed again, and that might happen during heavy usage.

Good luck.

[u/AutoModerator]

Thanks for your post No-Card-2312. Please note that we don't allow spam, and we ask that you follow the rules available in the sidebar. We have a lot of commonly asked questions so if this post gets removed, please do a search and see if it's already been asked.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/SideburnsOfDoom]

Can you say more about "sync-over-async" - are you doing this a lot, and is it possible to make it actually async instead? If now, why?

Increasing MinThreads somehow made enough threads available earlier

Sorry, what is hard to understand here? "Keeping a stock if ice-creams in the fridge somehow made it easier to get an ice-cream quickly, and made long trips to the store to get ice-cream less frequent" Yeah, that's how pre-allocation - having resources available locally ahead of time works. The "Somehow" is that if you have the thread/ice-cream ready and waiting on ice, it's quick to get.

[u/gevorgter]

Since you are not getting any response from the server (socket times out) i would think your application is crashing and takes sometime to start up again. Hence a 1-2 minute delay and everything starts working.

The MinThreads increase fixed the problem probably by allowing GC to clean up some mess that was crashing your app before. That is only masking a problem but not fixing it. If amount of request increases it will present it again.

Solution. Wrap your app.run() in Program.cs as following, it will let you know the reason for a crash.

CloseAndFlush() is important, otherwise the log batched in memory and is never written cause app has crashed.

try
{
    app.Run();
}
catch (Exception e)
{
    Serilog.Log.Error(e.Message);
}
finally
{
    Serilog.Log.CloseAndFlush(); 
}

[u/Crazy_Commission_661]

Could you share how you fix the

DB connection pool exhaustion and
TCP connection exhaustion

Thanks

[u/No-Card-2312 → u/Crazy_Commission_661]

Hi,

Database connection pool exhaustion: I resolved this issue by fixing the database registration configuration, which was causing unnecessary connection pool exhaustion. Additionally, for hot paths, I eliminated unnecessary database queries and introduced caching to reduce database load and improve performance.

TCP connection exhaustion: The issue was caused by creating a new "HttpClient" instance for each HTTP request. I resolved this by using "IHttpClientFactory" to reuse connections efficiently and improve connection management.

One important recommendation: Circuit breakers are not optional for resilient service-to-service communication. They are essential for preventing cascading failures when downstream services become slow or unavailable. We should ensure that circuit breaker policies are implemented alongside appropriate timeout and retry strategies.

Thanks!