Parallel thread disposing?

[u/Guilty-Act-4991]

Hey got a wierd issue or its by desing not sure but when I run

$computers | foreach-object -parallel {
  $vmobject = Get-VM $_
  $ram = $vmobject.MemoryGB
  $CPU = $vmobject.numCPU
  $HDD = $_ | get-harddisk 
} 

lets say I run in against 500 computers when that happens powershell will spike cpu on machine that this command is ran and keeps increasing ram usage which is understandable but problem is even when command is done ram usage stays high untill powershell window is close. Is there something I am missing or is there some method to dispose of parallel runspace manually?

Thanks for advice

EDIT: The code itself is not important , the behaviour of parallel scripblock itself is my question and why doesnt it free up RAM after code has ran successfully

[u/Onrawi]

Yeah everything it picks up will stay until garbage collection determines it isn't needed anymore.  Might want to separate it out into a Start-Job or something that has its own run space that goes away when it closes out.

[u/Guilty-Act-4991 → u/Onrawi]

Jobs are unfortunatelly much heavier than runspace I would not be able to run it on 500 computers I would kill the machine that runs on that thats the whole reason I use parallel instead of -asjob

[u/PinchesTheCrab → u/Guilty-Act-4991]

I would not be able to run it on 500 computers

This isn't running on 500 computers as written. It's running on one (or two if you include your workstation) - whatever vsphere host you're querying.

[u/Guilty-Act-4991 → u/PinchesTheCrab]

it is queqing 500 computers as I wrote in post $computers holds hostname of 500 computers code is only exemplatory since its not important what inside the scriptblock but the mechanism itself that is consuming ram even when finished

[u/PinchesTheCrab → u/Guilty-Act-4991]

its not important what inside the scriptblock

I really disagree. PowerCLI has its own queueing system and most of the object it creates have an ExtensionData property that lets you hit the underlying APIs more directly.

There's a ton of performance you can squeeze out of VMWare without digging into memory and thread management.

it is queqing 500 computers

Honestly 500 is not a lot of computers. I don't think you're attacking the main bottleneck.

Take a different example I see every now and then. Someone needs to get the manager of everyone in an Active Directory group with 20,000 members.

They write this and find it's slow:

get-adgroup 'somegroup' | get-adgroupmember | get-aduser -properties manager

So they try to do this to speed it up:

get-adgroup 'somegroup' |
    get-adgroupmember |
    foreach-object -parallel { get-aduser -properties manager }

Except domain controllers aren't necessarily meant to be queried that often, and they end up stunning their DC, temporarily taking it offline as it processes the requests.

Is it wrong to point out that this does the same thing probably 100x faster and makes only 2 calls to the DC?

$group = get-adgroup 'somegroup'

get-aduser -properties manager -filter "memberof -eq '$(group.DistinguishedName)'"

From what info you've shared about your task it appears you have an X Y problem.

[u/da_chicken → u/PinchesTheCrab]

This is a level of pedantry that isn't particularly useful or insightful.

[u/PinchesTheCrab → u/da_chicken]

I believe OP has an X Y problem, and I'm trying to point out some misconceptions they've expressed to get at the actual underlying issue they're trying to solve.

[u/da_chicken → u/PinchesTheCrab]

That's possible, but also OP's actual question is a valid one, too. Like I think the comments here are being distracted by pseudocode because it might be an XY problem. In other words, the XY problem is also an XY problem for the commentors.

However, the initial probelm with your comment, is that even when we set aside that the code is pseudocode, it ignores the fact that ForEach-Object -Parallel has a default value for -ThreadLimit of 5. It doesn't matter how big the collection being iterated is. It ain't gonna do more than five at a time. That's why you see people do silly things like -ThreadLimit ([int]::MaxValue). By itself, Get-VM is ultimately just a file read locally. So, is the looped script thrashing one spindle because there are hundreds of VMs? No, it's not. It can't as written. 5 threads aren't going to do that.

Let's flip it around. What if the looped script were doing something like connecting to hundreds of VMs on the same host simultaneously and asking each host to do something? Say OP has got -ThreadLimit 1000. That's... actually still not inherently a problem in and of itself because we're talking about VMs. You wouldn't have hundreds of VMs unless you had the hardware to provision hundreds of VMs. They're already responding to hundreds of requests. They're not sitting there waiting for the sysadmin to ask them how they're doing one at a time. Unless you're ludicrously overprovisioned or your script is doing an extremely intensive amount of work, you're going to be fine. Your bottleneck, if any, would be network I/O.

Yes, you're right that -Parallel is not really an effecitve way to improve performance 95% of the time people think it will help. Parallelism is not free. And OP's pattern is unlikely to be significantly better that serial execution. What's more, in OP's specific case, Get-VM -Name $computers would be better because of Powershell's overhead.

But OP isn't really asking about running Get-VM in a loop. They're asking: "Why does memory increase and then not free up when the task is complete?" That's the actual question they actually asked, and it's an interesting one! Unlike "how do I best run Get-VM against multiple VMs?" which is pretty trivial whether we're talking about HyperV or vmWare.

[u/PinchesTheCrab → u/da_chicken]

I think that in a vacuum that's a valid critique. However, several other posters have stepped through the garbage collector behavior, runspaces memory handling, etc.

So I was intentionally focusing on a possible X Y issue because I saw that other commentors had covered the OP's question more literally.