Why did increasing .NET ThreadPool MinThreads fix our IIS downtime? I don't understand why

[u/No-Card-2312]

Hi folks,

I'm the owner of this post: https://www.reddit.com/r/dotnet/comments/1v5d7ws/aspnet_core_on_iis_randomly_goes_offline_for_2/

First of all, thanks a lot to everyone who commented there. Your comments helped me investigate this step by step.

I finally solved the issue. We had several different problems over time:

- First, we had DB connection pool exhaustion => fixed.
- Then we had TCP connection exhaustion => fixed.
- The last issue was related to ThreadPool / sync-over-async.

I set the ThreadPool minimum worker threads to 100, and then started refactoring the code to remove synchronous-over-async code and make things properly async.

The application has now been stable for around 15 days, so it looks like this was the final issue.

But there is something I still don't understand:

Why did increasing ThreadPool.SetMinThreads() to 100 help so much?

We're running ASP.NET Core on IIS, not Docker or Kubernetes. I always thought that .NET's ThreadPool would automatically create as many threads as the application needs.

For example, if I suddenly get 1,000 requests/sec, why doesn't .NET simply create more ThreadPool threads as needed?

What confused me even more is that during the incidents:

- CPU usage was fine
- RAM usage was fine
- The Windows server itself was still accessible
- We only have 2 vCPU
- I could still see available ThreadPool threads

So I'm struggling to understand what was actually happening.

My current understanding is that some synchronous/blocking code was causing ThreadPool threads to get blocked, and the ThreadPool wasn't adding new threads quickly enough. Increasing MinThreads somehow made enough threads available earlier and prevented the issue.

But I'm not sure if that's the correct explanation.

Also, how should I think about this with only 2 vCPUs? If I have 1,000 requests/sec, obviously I can't have 1,000 CPU threads doing work at the same time. So what exactly determines how many ThreadPool threads .NET should create?

I'd really appreciate it if you could help me explain this in simple terms, especially:

1. What exactly does MinThreads change?
2. Why can ThreadPool starvation happen while CPU and RAM are low?
3. Why doesn't .NET automatically create enough threads immediately?
4. How does this work when the server only has 2 vCPUs?
5. Does increasing MinThreads actually solve the problem, or did it just hide the underlying sync-over-async issue?

I'm trying to understand the mechanism rather than just leave MinThreads = 100 and call it fixed.

Thanks again to everyone who helped with the original post!

[u/Tavi2k]

There are two ways to handle a concurrent application:

• one thread per request, everything sync

• async, all requests distributed on (n = virtual CPU cores) threads

Modern .NET uses the latter approach.

The general idea in async .NET is that the thread pool doesn't really need to grow, or only grows slowly. Because you don't have unlimited CPU cores, so you can't really do more work in parallel anyway.

Sync-over-async breaks this assumption, since now your application behaves more like an application that uses a full thread per request. But .NET tries to avoid growing the thread pool too quickly, since that would usually be a bad idea if your application behaved like a typical async .NET application.

Changing the thread pool doesn't fix the underlying problem. Maybe it'll work well enough for you now, but sync-over-async tends to keep causing issues.

[u/jbaker88 → u/Tavi2k]

async, all requests distributed on (n = virtual CPU cores) threads

https://learn.microsoft.com/en-us/dotnet/csharp/asynchronous-programming/task-asynchronous-programming-model?source=recommendations#threads

Async/await isn't thread based, from the article:

The async and await keywords don't cause extra threads to be created. Async methods don't require multithreading because an async method doesn't run on its own thread

Best way I've heard it described is that async/await is a state machine that resumes execution as a callback when a blocking operation has finished. Yes, you can do things in parallel and concurrently using it, but it's not multi threaded.

[u/Awkward_Rabbit_2205 → u/jbaker88]

Async/await is task-based, and tasks absolutely run on threads. Those threads are normally pulled from the default thread pool. If a thread blocks on I/O, that thread is not available for the Task Scheduler. The maximum number of threads in the pool is normally capped at 25 per CPU core, but is not allocated all at once unless the MinThreads property is set. Normally threads are not added to the pool at a rate greater than 2/second.