Multithreading Script

PowerShell Help
1

I am attempting to multithread a script and any time I increase the amount of threads, the performance of the script greatly decreases. For example, if I set a max of 5 threads I can write 635 objects/min. When I set to 10 threads, I can write only 361 objects/min. I have beefed up resources on the server I am running this from and never get close to maxing out the hardware, so I know it’s not a hardware limitation.

Here is the script:

$runspacecollection = @()
$runspacepool = [runspacefactory]::CreateRunspacePool(1, 5)
$runspacePool.Open()


$scriptblock = {
    param($folder)

    Script goes here

}


Write-Host ‘Parsing Folder Structure’
$folders = Get-ChildItem \\server\Datastore
Write-Host ‘Processing Files’


foreach ($folder in $folders) {
    $powershell = [powershell]::Create().AddScript($scriptblock).AddArgument($folder)
    $powershell.RunspacePool = $runspacepool
    [Collections.ArrayList]$runspacecollection += New-Object -TypeName System.Management.Automation.PSObject -Property @{
        runspace   = $powershell.BeginInvoke()
        powershell = $powershell
    }
}


while ($runspacecollection) {
    foreach ($runspace in $runspacecollection.ToArray()) {
        if ($runspace.Runspace.IsCompleted) {
            $runspace.Powershell.EndInvoke($runspace.Runspace)
            $runspace.Powershell.Dispose()
            $runspacecollection.Remove($runspace)
        }
    }
}
2

After checking various settings and syntax, my issue is with the server that I am reading files from. I can see that I am getting all the threads, however, I am being bottlenecked by the read speed on the server I am reading from.

The more threads the bigger the read speed bottleneck. Explains a lot.