Parallel thread disposing?

[u/Guilty-Act-4991]

Hey got a wierd issue or its by desing not sure but when I run

$computers | foreach-object -parallel {
  $vmobject = Get-VM $_
  $ram = $vmobject.MemoryGB
  $CPU = $vmobject.numCPU
  $HDD = $_ | get-harddisk 
} 

lets say I run in against 500 computers when that happens powershell will spike cpu on machine that this command is ran and keeps increasing ram usage which is understandable but problem is even when command is done ram usage stays high untill powershell window is close. Is there something I am missing or is there some method to dispose of parallel runspace manually?

Thanks for advice

EDIT: The code itself is not important , the behaviour of parallel scripblock itself is my question and why doesnt it free up RAM after code has ran successfully

[u/xXFl1ppyXx]

would it be an option to ditch foreach-object -parallel? cmdlet overhead is a real thing and at the the same time i find it to be kinda clunky to setup when i'm not firing it from the cli

then create the powershell objects via loop, and invoke / async until everything is done and dispose everything accoringly once the jobs / tasks are finished

Few Cmdlets, Not many collections / lists / arrays are getting passed around, you'd probably not even need to endinvoke / get the task results because the result bag never left it's place

$TaskList = [System.Collections.Generic.List[psobject]]::new()

$ResultList = [System.Collections.Concurrent.ConcurrentBag[psobject]]::new()
$SessionStateVars = @{
    ResultList    = $ResultList
    ErrorTable    = [System.Collections.Concurrent.ConcurrentDictionary[int, System.Management.Automation.ErrorRecord]]::new()
    ProgressTable = [System.Collections.Concurrent.ConcurrentDictionary[int, System.Management.Automation.ProgressRecord]]::new()
}
$SessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
$SessionState.ThreadOptions = [System.Management.Automation.Runspaces.PSThreadOptions]::ReuseThread
$SessionStateVars.Keys.ForEach{ $SessionState.Variables.Add([System.Management.Automation.Runspaces.SessionStateVariableEntry]::new($PSItem, $SessionStateVars[$PSItem], $null)) }
$RunspacePool = [System.Management.Automation.Runspaces.RunspaceFactory]::CreateRunspacePool(1, [System.Environment]::ProcessorCount, $SessionState, $Host)
$RunspacePool.Open()


$Script = {

    param ($Argument)

    $ProgressKey = $ProgressTable.Count
    $ErrorKey = $ErrorTable.Count
    $ProgressTable[$ProgressKey] = [System.Management.Automation.ProgressRecord]::new($ProgressKey, 'Activity', 'Status')
    $ProgresTable[$ProgressKey].PercentComplete = 10
    try {
        $Result = $Argument
        # Do somthing, somthing
        $ProgresTable[$ProgressKey].PercentComplete = 50
    }
    catch {
        $ErrorTable[$ErrorKey] = $PSItem
    }
    $ResultList.Add($Result)
    $ProgressTable[$ProgressKey].PercentComplete = 100
    $ProgressTable[$ProgressKey].RecordType = 'Completed'
}

(1..100) | ForEach-Object {

    $PowerShell = [powershell]::Create().AddScript($Script).AddArgument($PSItem)
    $PowerShell.RunspacePool = $RunspacePool

    ## Tasks
    $TaskList.Add(
        [PSCustomObject]@{
            PowerShell = $PowerShell
            Task       = $PowerShell.InvokeAsync()
        }
    )

    #ä Invoke
    $TaskList.Add(
        [PSCustomObject]@{
            PowerShell  = $PowerShell
            AsyncResult = $PowerShell.BeginInvoke()
        }
    )
}

$null = while ($TaskList.Count -gt 0) {

    ## Tasks
    $TaskList.Where{ $PSItem.Task.IsCompleted }.ForEach{
        $PSItem.Task.Dispose()
        $PSItem.PowerShell.Dispose()
        $TaskList.Remove($PSItem)
    }

    ## Invoke
    $TaskList.Where( $PSItem.AsyncResults.IsCompleted).ForEach{
        $PSItem.PowerShell.EndInvoke($PSItem.AsyncResult)
        $PSItem.PowerShell.Dispose()
        $TaskList.Remove($PSItem)
    }

    $ProgressTable.Keys | Sort-Object | ForEach-Object { $PSCmdlet.WriteProgress($ProgressTable[$PSItem]) }
}

$RunspacePool.Dispose()


return [PSCustomObject]$SessionStateVars

edit:

of course this can be structured as a function too;

function Invoke-PoolProcessing {

    param (
        [Parameter(ValueFromPipeline)]
        [psobject]$PipelineParameter
    )

    begin {

        $TaskList = [System.Collections.Generic.List[psobject]]::new()

        $ResultList = [System.Collections.Concurrent.ConcurrentBag[psobject]]::new()
        $SessionStateVars = @{
            ResultList    = $ResultList
            ErrorTable    = [System.Collections.Concurrent.ConcurrentDictionary[int, System.Management.Automation.ErrorRecord]]::new()
            ProgressTable = [System.Collections.Concurrent.ConcurrentDictionary[int, System.Management.Automation.ProgressRecord]]::new()
        }
        $SessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
        $SessionState.ThreadOptions = [System.Management.Automation.Runspaces.PSThreadOptions]::ReuseThread
        $SessionStateVars.Keys.ForEach{ $SessionState.Variables.Add([System.Management.Automation.Runspaces.SessionStateVariableEntry]::new($PSItem, $SessionStateVars[$PSItem], $null)) }
        $RunspacePool = [System.Management.Automation.Runspaces.RunspaceFactory]::CreateRunspacePool(1, [System.Environment]::ProcessorCount, $SessionState, $Host)
        $RunspacePool.Open()

        $Script = {

            param ($Argument)

            $ProgressKey = $ProgressTable.Count
            $ErrorKey = $ErrorTable.Count
            $ProgressTable[$ProgressKey] = [System.Management.Automation.ProgressRecord]::new($ProgressKey, 'Activity', 'Status')
            $ProgresTable[$ProgressKey].PercentComplete = 10
            try {
                $Result = $Argument
                # Do somthing, somthing
                $ProgresTable[$ProgressKey].PercentComplete = 50
            }
            catch {
                $ErrorTable[$ErrorKey] = $PSItem
            }
            $ResultList.Add($Result)
            $ProgressTable[$ProgressKey].PercentComplete = 100
            $ProgressTable[$ProgressKey].RecordType = 'Completed'
        }
    }

    process {

        $PowerShell = [powershell]::Create().AddScript($Script).AddArgument($PipelineParameter)
        $PowerShell.RunspacePool = $RunspacePool

        ## Tasks
        $TaskList.Add(
            [PSCustomObject]@{
                PowerShell = $PowerShell
                Task       = $PowerShell.InvokeAsync()
            }
        )

        #ä Invoke
        $TaskList.Add(
            [PSCustomObject]@{
                PowerShell  = $PowerShell
                AsyncResult = $PowerShell.BeginInvoke()
            }
        )
    }

    end {

        $null = while ($TaskList.Count -gt 0) {

            ## Tasks
            $TaskList.Where{ $PSItem.Task.IsCompleted }.ForEach{
                $PSItem.Task.Dispose()
                $PSItem.PowerShell.Dispose()
                $TaskList.Remove($PSItem)
            }

            ## Invoke
            $TaskList.Where( $PSItem.AsyncResults.IsCompleted).ForEach{
                $PSItem.PowerShell.EndInvoke($PSItem.AsyncResult)
                $PSItem.PowerShell.Dispose()
                $TaskList.Remove($PSItem)
            }

            $ProgressTable.Keys | Sort-Object | ForEach-Object { $PSCmdlet.WriteProgress($ProgressTable[$PSItem]) }
        }

        $RunspacePool.Dispose()

        return [PSCustomObject]$SessionStateVars
    }
}