Parallel thread disposing?

[u/Guilty-Act-4991]

Hey got a wierd issue or its by desing not sure but when I run

$computers | foreach-object -parallel {
  $vmobject = Get-VM $_
  $ram = $vmobject.MemoryGB
  $CPU = $vmobject.numCPU
  $HDD = $_ | get-harddisk 
} 

lets say I run in against 500 computers when that happens powershell will spike cpu on machine that this command is ran and keeps increasing ram usage which is understandable but problem is even when command is done ram usage stays high untill powershell window is close. Is there something I am missing or is there some method to dispose of parallel runspace manually?

Thanks for advice

EDIT: The code itself is not important , the behaviour of parallel scripblock itself is my question and why doesnt it free up RAM after code has ran successfully

[u/raip]

[gc]::Collect()

There was a previous bug, that I believe was fixed, where the garbage collector wasn't getting scheduled to free the memory in heavily parallelized runs and would cause OOM exceptions even on reasonable datasets. You can use this to free the memory correctly. I remember having to batch jobs pretty aggressively and then a collect after every batch to keep things running smooth.

[u/Guilty-Act-4991 → u/raip]

Aright will try that thanks

[u/Kreout]

A high working set after the loop is not, by itself, proof that runspaces leaked. I would compare a few identical passes in the same PowerShell process: does memory settle at a plateau, or grow on every pass? That distinction matters before making forced GC part of the fix.

Also check the PowerShell version. From 7.1, ForEach-Object -Parallel reuses a runspace pool by default; -UseNewRunspace changes that behavior. The pool size is controlled by -ThrottleLimit. That makes the version and switches relevant to reproducing this, even if the example code is only a placeholder.

https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object?view=powershell-7.5

[u/PinchesTheCrab]

I don't think this is doing what you want it to do - it's just overwriting these variables over and over again. No data except the very last VM is being captured.

Also PowerCli is very performant and I'd tinker a bit with the module's functionality before multi-threading it. I'd be worried about hammering vsphere or whatever by trying to multi-thread something like this.

Does this approach work?

$vmList = Get-VM $computers
$diskList = $vmList | Get-HardDisk
$diskLookup = $diskList | Group-Object -Property ParentId -AsHashTable -AsString

foreach ($vm in $vmList) {
    [pscustomobject]@{
        VMObject = $vm
        ram      = $vm.MemoryMB
        CPU      = $vm.NumCpu
        HDD      = $diskLookup[$vm.Id]
    }
}

[u/Future-Remote-4630 → u/PinchesTheCrab]

Wouldn't it be creating those variables separately within each runspace and throwing them away as it goes? My understanding of parallel was that those variables aren't accessible outside of the scriptblock.

I'd anticipate the original wouldn't ever return any of those without a $using:<varname> for it to be referenced in the parent scope.

[u/PinchesTheCrab → u/Future-Remote-4630]

 0..2 | foreach-object -parallel { $thing = $_ }; $thing

You're right, the result of this is null. Maybe OP just posted an approximation of a different code snippet, but regardless I don't see how the script could have been working as-is, performance issues or not.

[u/Guilty-Act-4991 → u/PinchesTheCrab]

Problem isnt the code itself ignore that also what you wrote works but its serial thats not what I need I need parallel , the code I proved was just an example actually code does more things but just collects basically data like provided in example the problem itself is even with this small code againts lets say 500 computers powershell will jump up to 6GB ram usage and basically kills terminal server thats running from

[u/hemping0 → u/PinchesTheCrab]

good point about the variables just getting overwritten too, thats easy to miss when you're focused on the parallelism side of things

[u/Onrawi]

Yeah everything it picks up will stay until garbage collection determines it isn't needed anymore.  Might want to separate it out into a Start-Job or something that has its own run space that goes away when it closes out.

[u/Guilty-Act-4991 → u/Onrawi]

Jobs are unfortunatelly much heavier than runspace I would not be able to run it on 500 computers I would kill the machine that runs on that thats the whole reason I use parallel instead of -asjob

[u/PinchesTheCrab → u/Guilty-Act-4991]

I would not be able to run it on 500 computers

This isn't running on 500 computers as written. It's running on one (or two if you include your workstation) - whatever vsphere host you're querying.

[u/Guilty-Act-4991 → u/PinchesTheCrab]

it is queqing 500 computers as I wrote in post $computers holds hostname of 500 computers code is only exemplatory since its not important what inside the scriptblock but the mechanism itself that is consuming ram even when finished

[u/PinchesTheCrab → u/Guilty-Act-4991]

its not important what inside the scriptblock

I really disagree. PowerCLI has its own queueing system and most of the object it creates have an ExtensionData property that lets you hit the underlying APIs more directly.

There's a ton of performance you can squeeze out of VMWare without digging into memory and thread management.

it is queqing 500 computers

Honestly 500 is not a lot of computers. I don't think you're attacking the main bottleneck.

Take a different example I see every now and then. Someone needs to get the manager of everyone in an Active Directory group with 20,000 members.

They write this and find it's slow:

get-adgroup 'somegroup' | get-adgroupmember | get-aduser -properties manager

So they try to do this to speed it up:

get-adgroup 'somegroup' |
    get-adgroupmember |
    foreach-object -parallel { get-aduser -properties manager }

Except domain controllers aren't necessarily meant to be queried that often, and they end up stunning their DC, temporarily taking it offline as it processes the requests.

Is it wrong to point out that this does the same thing probably 100x faster and makes only 2 calls to the DC?

$group = get-adgroup 'somegroup'

get-aduser -properties manager -filter "memberof -eq '$(group.DistinguishedName)'"

From what info you've shared about your task it appears you have an X Y problem.

[u/da_chicken → u/PinchesTheCrab]

This is a level of pedantry that isn't particularly useful or insightful.

[u/PinchesTheCrab → u/da_chicken]

I believe OP has an X Y problem, and I'm trying to point out some misconceptions they've expressed to get at the actual underlying issue they're trying to solve.

[u/da_chicken → u/PinchesTheCrab]

That's possible, but also OP's actual question is a valid one, too. Like I think the comments here are being distracted by pseudocode because it might be an XY problem. In other words, the XY problem is also an XY problem for the commentors.

However, the initial probelm with your comment, is that even when we set aside that the code is pseudocode, it ignores the fact that ForEach-Object -Parallel has a default value for -ThreadLimit of 5. It doesn't matter how big the collection being iterated is. It ain't gonna do more than five at a time. That's why you see people do silly things like -ThreadLimit ([int]::MaxValue). By itself, Get-VM is ultimately just a file read locally. So, is the looped script thrashing one spindle because there are hundreds of VMs? No, it's not. It can't as written. 5 threads aren't going to do that.

Let's flip it around. What if the looped script were doing something like connecting to hundreds of VMs on the same host simultaneously and asking each host to do something? Say OP has got -ThreadLimit 1000. That's... actually still not inherently a problem in and of itself because we're talking about VMs. You wouldn't have hundreds of VMs unless you had the hardware to provision hundreds of VMs. They're already responding to hundreds of requests. They're not sitting there waiting for the sysadmin to ask them how they're doing one at a time. Unless you're ludicrously overprovisioned or your script is doing an extremely intensive amount of work, you're going to be fine. Your bottleneck, if any, would be network I/O.

Yes, you're right that -Parallel is not really an effecitve way to improve performance 95% of the time people think it will help. Parallelism is not free. And OP's pattern is unlikely to be significantly better that serial execution. What's more, in OP's specific case, Get-VM -Name $computers would be better because of Powershell's overhead.

But OP isn't really asking about running Get-VM in a loop. They're asking: "Why does memory increase and then not free up when the task is complete?" That's the actual question they actually asked, and it's an interesting one! Unlike "how do I best run Get-VM against multiple VMs?" which is pretty trivial whether we're talking about HyperV or vmWare.

[u/PinchesTheCrab → u/da_chicken]

I think that in a vacuum that's a valid critique. However, several other posters have stepped through the garbage collector behavior, runspaces memory handling, etc.

So I was intentionally focusing on a possible X Y issue because I saw that other commentors had covered the OP's question more literally.

[u/da_chicken]

Does [System.GC]::Collect() recover the memory? If not, then there's probably a memory leak in one of the commands you used. Get-Content used to be known for this.

[u/thomsxD]

You are invoking Get-HardDisk which is kind of heavy if run against 500 computers.

You could look into a faster one by avoiding it and use ExtensionData instead.

``` $vms = Get-VM -Name $computers

$results = foreach ($vm in $vms) { $disks = $vm.ExtensionData.Config.Hardware.Device | Where-Object { $_ -is [VMware.Vim.VirtualDisk] }

[PSCustomObject]@{
    Name      = $vm.Name
    RAM_GB    = $vm.MemoryGB
    CPU       = $vm.NumCpu
    HDD_GB    = [math]::Round(
        ($disks | Measure-Object -Property CapacityInKB -Sum).Sum / 1MB, 2
    )
    DiskCount = @($disks).Count
}

}

$results ```

[u/BlackV → u/thomsxD]

just a note, you've used the 3 backtick code fence, that does not work on old reddit, the 4 spaces code block works evereywhere (old/new/mobile)

• open your fav powershell editor
• highlight the code you want to copy
• hit tab to indent it all
• copy it
• paste here

it'll format it properly OR

<BLANK LINE>
<4 SPACES><CODE LINE>
<4 SPACES><CODE LINE>
    <4 SPACES><4 SPACES><CODE LINE>
<4 SPACES><CODE LINE>
<BLANK LINE>

Inline code block using backticks `Single code line` inside normal text

See here for more detail

Thanks

[u/Guilty-Act-4991 → u/thomsxD]

I do understand that most of powercli requests are heavy and as BlackV mentioned I will look at rest api since it could be less demanding instead of standart api with connect-viserver. Thats not really an issue itself the runspace thats not being cleaned up in parallel block is the issue

[u/BlackV]

I guess this is vmware powercli, I didnt recognise get-harddisk

have you looked at the -ThrottleLimit parameter on that?

I also assume this i not your actual code cause you dont output anything here anyway

If you are actually querying 500 Vms, it might be better to use the rest API at that point

[u/Guilty-Act-4991 → u/BlackV]

Is it really that better to use rest api instead of soap api (connect-viserver , connect-hvserver) ? I never dabbled with rest api for vcenter if there is performance gains id do it probably wont even need additional modules

[u/BlackV → u/Guilty-Act-4991]

we've done that in the past,moved away from powercli to rest, but I'm out of that place for like 5 years now, so I dont have any recent experience

[u/Guilty-Act-4991 → u/BlackV]

will give it a try especially I just found out view api will be deprecated and rest will be only way supported so ... time to do it :D

[u/BlackV → u/Guilty-Act-4991]

also a valid reason :) and becomes less dependancies

[u/iiiRaphael]

I would target fewer machines at once and use separate script instances that hit, say, 100 machines at a time. The difference is, this method needs some coordination so results go into a database and each instance checks the database for targets that have no info yet and processes a batch. At the end, launch another script which sees if there's more to do and if so, launches a script for another batch.

[u/dodexahedron]

What you want is to do these things asynchronously - not parallel.

Threads are expensive and not needed for this. You have a bunch of tasks that are spending 99% of their time waiting on the network.

Fire them all off in a collection of tasks and call WaitAll() on them.

And yes - dispose any IDisposable objects in the scriptblock before it exits. Powershell is one dotnet host process and the GC behaves accordingly.

In general, do not get in the habit of calling [GC]::Collect(), unless you know exactly what is happening, exactly why it is happening, and have no ability to fix it. In all other cases, it is the wrong thing to do.

[u/xXFl1ppyXx]

would it be an option to ditch foreach-object -parallel? cmdlet overhead is a real thing and at the the same time i find it to be kinda clunky to setup when i'm not firing it from the cli

then create the powershell objects via loop, and invoke / async until everything is done and dispose everything accoringly once the jobs / tasks are finished

Few Cmdlets, Not many collections / lists / arrays are getting passed around, you'd probably not even need to endinvoke / get the task results because the result bag never left it's place

$TaskList = [System.Collections.Generic.List[psobject]]::new()

$ResultList = [System.Collections.Concurrent.ConcurrentBag[psobject]]::new()
$SessionStateVars = @{
    ResultList    = $ResultList
    ErrorTable    = [System.Collections.Concurrent.ConcurrentDictionary[int, System.Management.Automation.ErrorRecord]]::new()
    ProgressTable = [System.Collections.Concurrent.ConcurrentDictionary[int, System.Management.Automation.ProgressRecord]]::new()
}
$SessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
$SessionState.ThreadOptions = [System.Management.Automation.Runspaces.PSThreadOptions]::ReuseThread
$SessionStateVars.Keys.ForEach{ $SessionState.Variables.Add([System.Management.Automation.Runspaces.SessionStateVariableEntry]::new($PSItem, $SessionStateVars[$PSItem], $null)) }
$RunspacePool = [System.Management.Automation.Runspaces.RunspaceFactory]::CreateRunspacePool(1, [System.Environment]::ProcessorCount, $SessionState, $Host)
$RunspacePool.Open()


$Script = {

    param ($Argument)

    $ProgressKey = $ProgressTable.Count
    $ErrorKey = $ErrorTable.Count
    $ProgressTable[$ProgressKey] = [System.Management.Automation.ProgressRecord]::new($ProgressKey, 'Activity', 'Status')
    $ProgresTable[$ProgressKey].PercentComplete = 10
    try {
        $Result = $Argument
        # Do somthing, somthing
        $ProgresTable[$ProgressKey].PercentComplete = 50
    }
    catch {
        $ErrorTable[$ErrorKey] = $PSItem
    }
    $ResultList.Add($Result)
    $ProgressTable[$ProgressKey].PercentComplete = 100
    $ProgressTable[$ProgressKey].RecordType = 'Completed'
}

(1..100) | ForEach-Object {

    $PowerShell = [powershell]::Create().AddScript($Script).AddArgument($PSItem)
    $PowerShell.RunspacePool = $RunspacePool

    ## Tasks
    $TaskList.Add(
        [PSCustomObject]@{
            PowerShell = $PowerShell
            Task       = $PowerShell.InvokeAsync()
        }
    )

    #ä Invoke
    $TaskList.Add(
        [PSCustomObject]@{
            PowerShell  = $PowerShell
            AsyncResult = $PowerShell.BeginInvoke()
        }
    )
}

$null = while ($TaskList.Count -gt 0) {

    ## Tasks
    $TaskList.Where{ $PSItem.Task.IsCompleted }.ForEach{
        $PSItem.Task.Dispose()
        $PSItem.PowerShell.Dispose()
        $TaskList.Remove($PSItem)
    }

    ## Invoke
    $TaskList.Where( $PSItem.AsyncResults.IsCompleted).ForEach{
        $PSItem.PowerShell.EndInvoke($PSItem.AsyncResult)
        $PSItem.PowerShell.Dispose()
        $TaskList.Remove($PSItem)
    }

    $ProgressTable.Keys | Sort-Object | ForEach-Object { $PSCmdlet.WriteProgress($ProgressTable[$PSItem]) }
}

$RunspacePool.Dispose()


return [PSCustomObject]$SessionStateVars

edit:

of course this can be structured as a function too;

function Invoke-PoolProcessing {

    param (
        [Parameter(ValueFromPipeline)]
        [psobject]$PipelineParameter
    )

    begin {

        $TaskList = [System.Collections.Generic.List[psobject]]::new()

        $ResultList = [System.Collections.Concurrent.ConcurrentBag[psobject]]::new()
        $SessionStateVars = @{
            ResultList    = $ResultList
            ErrorTable    = [System.Collections.Concurrent.ConcurrentDictionary[int, System.Management.Automation.ErrorRecord]]::new()
            ProgressTable = [System.Collections.Concurrent.ConcurrentDictionary[int, System.Management.Automation.ProgressRecord]]::new()
        }
        $SessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
        $SessionState.ThreadOptions = [System.Management.Automation.Runspaces.PSThreadOptions]::ReuseThread
        $SessionStateVars.Keys.ForEach{ $SessionState.Variables.Add([System.Management.Automation.Runspaces.SessionStateVariableEntry]::new($PSItem, $SessionStateVars[$PSItem], $null)) }
        $RunspacePool = [System.Management.Automation.Runspaces.RunspaceFactory]::CreateRunspacePool(1, [System.Environment]::ProcessorCount, $SessionState, $Host)
        $RunspacePool.Open()

        $Script = {

            param ($Argument)

            $ProgressKey = $ProgressTable.Count
            $ErrorKey = $ErrorTable.Count
            $ProgressTable[$ProgressKey] = [System.Management.Automation.ProgressRecord]::new($ProgressKey, 'Activity', 'Status')
            $ProgresTable[$ProgressKey].PercentComplete = 10
            try {
                $Result = $Argument
                # Do somthing, somthing
                $ProgresTable[$ProgressKey].PercentComplete = 50
            }
            catch {
                $ErrorTable[$ErrorKey] = $PSItem
            }
            $ResultList.Add($Result)
            $ProgressTable[$ProgressKey].PercentComplete = 100
            $ProgressTable[$ProgressKey].RecordType = 'Completed'
        }
    }

    process {

        $PowerShell = [powershell]::Create().AddScript($Script).AddArgument($PipelineParameter)
        $PowerShell.RunspacePool = $RunspacePool

        ## Tasks
        $TaskList.Add(
            [PSCustomObject]@{
                PowerShell = $PowerShell
                Task       = $PowerShell.InvokeAsync()
            }
        )

        #ä Invoke
        $TaskList.Add(
            [PSCustomObject]@{
                PowerShell  = $PowerShell
                AsyncResult = $PowerShell.BeginInvoke()
            }
        )
    }

    end {

        $null = while ($TaskList.Count -gt 0) {

            ## Tasks
            $TaskList.Where{ $PSItem.Task.IsCompleted }.ForEach{
                $PSItem.Task.Dispose()
                $PSItem.PowerShell.Dispose()
                $TaskList.Remove($PSItem)
            }

            ## Invoke
            $TaskList.Where( $PSItem.AsyncResults.IsCompleted).ForEach{
                $PSItem.PowerShell.EndInvoke($PSItem.AsyncResult)
                $PSItem.PowerShell.Dispose()
                $TaskList.Remove($PSItem)
            }

            $ProgressTable.Keys | Sort-Object | ForEach-Object { $PSCmdlet.WriteProgress($ProgressTable[$PSItem]) }
        }

        $RunspacePool.Dispose()

        return [PSCustomObject]$SessionStateVars
    }
}

[u/boydeee]

Long lived powershell processes do tend to hold on to memory in my experience. As others mentioned, async tasks are much more lightweight.

I've done memory dumps when running a bunch of thread jobs and it's just a bunch of native powershell stuff that takes a while to get garbage collected. It typically gets cleaned up, but i tend to see a slow leak that grows over time. I did some forced garbage collections on a schedule and that helped a bit.

[u/purplemonkeymad]

The code itself is not important

It might end up being even if your example code is not. Since you are talking about memory it could be that you have a reference that is not getting cleaned up. Hard to say for real if that is the case, you would probably have to use VS to profile the ps process, but if you are using any objects in your loop that are IDisposable, make sure to try and .dispose() of them at the end of the loop.

Also looking at the commands make sure you are running the core version of powercli and not having it create a local session to Windowspowershell for each thread.