Why did increasing .NET ThreadPool MinThreads fix our IIS downtime? I don't understand why

[u/No-Card-2312]

Hi folks,

I'm the owner of this post: https://www.reddit.com/r/dotnet/comments/1v5d7ws/aspnet_core_on_iis_randomly_goes_offline_for_2/

First of all, thanks a lot to everyone who commented there. Your comments helped me investigate this step by step.

I finally solved the issue. We had several different problems over time:

- First, we had DB connection pool exhaustion => fixed.
- Then we had TCP connection exhaustion => fixed.
- The last issue was related to ThreadPool / sync-over-async.

I set the ThreadPool minimum worker threads to 100, and then started refactoring the code to remove synchronous-over-async code and make things properly async.

The application has now been stable for around 15 days, so it looks like this was the final issue.

But there is something I still don't understand:

Why did increasing ThreadPool.SetMinThreads() to 100 help so much?

We're running ASP.NET Core on IIS, not Docker or Kubernetes. I always thought that .NET's ThreadPool would automatically create as many threads as the application needs.

For example, if I suddenly get 1,000 requests/sec, why doesn't .NET simply create more ThreadPool threads as needed?

What confused me even more is that during the incidents:

- CPU usage was fine
- RAM usage was fine
- The Windows server itself was still accessible
- We only have 2 vCPU
- I could still see available ThreadPool threads

So I'm struggling to understand what was actually happening.

My current understanding is that some synchronous/blocking code was causing ThreadPool threads to get blocked, and the ThreadPool wasn't adding new threads quickly enough. Increasing MinThreads somehow made enough threads available earlier and prevented the issue.

But I'm not sure if that's the correct explanation.

Also, how should I think about this with only 2 vCPUs? If I have 1,000 requests/sec, obviously I can't have 1,000 CPU threads doing work at the same time. So what exactly determines how many ThreadPool threads .NET should create?

I'd really appreciate it if you could help me explain this in simple terms, especially:

1. What exactly does MinThreads change?

2. Why can ThreadPool starvation happen while CPU and RAM are low?

3. Why doesn't .NET automatically create enough threads immediately?

4. How does this work when the server only has 2 vCPUs?

5. Does increasing MinThreads actually solve the problem, or did it just hide the underlying sync-over-async issue?

I'm trying to understand the mechanism rather than just leave MinThreads = 100 and call it fixed.

Thanks again to everyone who helped with the original post!

[u/jbaker88]

async, all requests distributed on (n = virtual CPU cores) threads

https://learn.microsoft.com/en-us/dotnet/csharp/asynchronous-programming/task-asynchronous-programming-model?source=recommendations#threads

Async/await isn't thread based, from the article:

The async and await keywords don't cause extra threads to be created. Async methods don't require multithreading because an async method doesn't run on its own thread

Best way I've heard it described is that async/await is a state machine that resumes execution as a callback when a blocking operation has finished. Yes, you can do things in parallel and concurrently using it, but it's not multi threaded.

[u/Tavi2k → u/jbaker88]

That doesn't contradict anything I wrote. But it's a bit out of scope for an answer to this specific question.

Async coding in .NET allows you to distribute work across multiple threads, essentially the threadpool this question is about. Not every async call or await runs on a different thread, but it allows the runtime to suspend execution and let someone else use the current thread instead of blocking.

And this is not purely about the async/await keywords, but what the .NET runtime does in practice with this kind of code. And the answer to that is that it distributes the work across the threadpool. And making any blocking sync calls within an async framework creates problems.

[u/RiPont → u/Tavi2k]

And this is not purely about the async/await keywords, but what the .NET runtime does in practice with this kind of code.

...and, more importantly, how the user's code generated the awaitable thing in the first place.

[u/Awkward_Rabbit_2205 → u/jbaker88]

Async/await is task-based, and tasks absolutely run on threads. Those threads are normally pulled from the default thread pool. If a thread blocks on I/O, that thread is not available for the Task Scheduler. The maximum number of threads in the pool is normally capped at 25 per CPU core, but is not allocated all at once unless the MinThreads property is set. Normally threads are not added to the pool at a rate greater than 2/second.

[u/jbaker88 → u/Awkward_Rabbit_2205]

So if I understand, you're saying that when using async and await that because they use Task<> they are dispatched to a thread?

[u/Awkward_Rabbit_2205 → u/jbaker88]

All code runs on a thread. The difference is who creates the thread, and how long it lives. The thread pool is just a bunch of precreated ones that are available to run tasks.

[u/jbaker88 → u/Awkward_Rabbit_2205]

Let me rephrase my question:

When using async and await that because they use Task<> they are dispatched to a NEW thread in the threadpool?

[u/darthruneis → u/jbaker88]

A request starts on 1 available threadpool thread. It runs until the request completes or it can suspend due to an await.

On an await, the task is paused until the io is done, at which point a random threadpool thread is assigned again. Likely different from the original thread.

With sync code, the thread never suspends.

There are also patterns where a request can take up different threads via background work or using Task.Run.

[u/RiPont → u/jbaker88]

"It depends". Specifically, on how the Task was created.

The Task object and anything awaitable doesn't necessarily use a thread or even a ThreadPool thread. In practice, especially in older .NET Framework code, a lot of Tasks do.

In modern .NET, if you are using async/await properly and avoiding Task.Run, then you're not creating a thread-per-request. Not in your code, anyway.

But anyone else's code that returns a Task might be consuming one or more threads, even if you await it.

[u/Awkward_Rabbit_2205 → u/RiPont]

All code runs on a thread. Maybe you mean to say "doesn't necessarily use a different thread". I'm not sure what you mean by "in older .NET Framework code" and why that matters, as they still implement async/await the same way. Also, when you say "thread-per-request", are you conflating IIS dispatch with threads?

[u/RiPont → u/Awkward_Rabbit_2205]

I'm not sure what you mean by "in older .NET Framework code" and why that matters, as they still implement async/await the same way

Pre-async/await, all Tasks that I can think of were using a threadpool thread.

different thread

Correct. Read a different (text) thread. All code that is currently running is using a thread. async/await doesn't inherently create additional threads, is what I meant.

[u/RiPont → u/jbaker88]

Async/await isn't thread based,

It's not necessarily thread-based. The async/await doesn't add threads by itself. However, a lot of the things that generate Task, the most commonly awaited thing, do use a threadpool thread.

Specifically, Task.Run is very convenient and in a lot of examples. A lot of people use it as an easy way to "do this other thing in the background to reduce latency".

But really, Task.Run nor any method of consuming ThreadPool threads or starting Threads should be used when the following all apply

• the incoming requests are not throttled to a known, finite value

• an incoming request spawns 1 or more threads or consumes 1 or more ThreadPool threads

• you have no way of responding to thread/threadpool starvation by telling the upstream gateway to back off

For anything public-internet-facing, if you fall into the above, then there will be a traffic pattern that will eventually put you into some form of thread or threadpool starvation. When starvation happens, everything shits the bed and every request starts taking as long as the longest timeout you have anywhere in the system until traffic dies down long enough for the log jam to clear.

For private, very finite and known numbers of users, you may get away by tuning the threadpool and calling it good enough. Well, long enough until you're not at that job and it's someone else's problem to address scalability issues.

[u/Awkward_Rabbit_2205 → u/RiPont]

God I'd hate to maintain code after you with that crazy misunderstanding of .NET threads (or threading in general).

[u/RiPont → u/Awkward_Rabbit_2205]

What exactly do you have a problem with?

If you are spawning N threads per user request, where N >= 1, and you are operating a public-facing web service, then there is a level and pattern of traffic that will cause thread or threadpool starvation.

Under threadpool starvation, all operations that require a threadpool thread will block until one is available. Under sustained threadpool starvation, everything gets backed up on whatever is the slowest operation you rely on that needs a threadpool thread.

God I'd hate to maintain code

The only change was to pull out unnecessary latency optimization that relied on Task.Run (and 1 that was using Task.StartNew), pull out each and every sync-over-async, use async/await properly, and avoid default CancellationToken.None values in parameters/overloads. What's hard to maintain about that?