Abandoned Mutex in Nested ForEach-Object -Parallel

Hi!

I’m using nested ForEach-Object -Parallel blocks to process resources in AWS Accounts\Region in parallel.

I added a Mutex in the second parallel block to prevent concurrent writes to my .CSV report.

However, the script throws numerous errors like this:

MethodInvocationException: 
Line |
   6 |          $Mutex.WaitOne() > $null;
     |          ~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception calling "WaitOne" with "0" argument(s): "The wait completed due to an abandoned mutex."

Is this a scoping issue?

It seems like the Mutex isn’t being released properly, but it’s hard to tell since the report generates properly (i.e., no malformed text, which is what prompted me to use the Mutex in the first place).

I tried to imitate Tolga’s answer here to implement this:

The relevant code portions are below:

$Regions | ForEach-Object -Parallel {                
                
        Import-Module -Name AWSPowerShell.NetCore, PSWriteColor #import the module for each Region runspace

        $Mutex = New-Object System.Threading.Mutex $false, 'CsvReportLock' #create Mutex to prevent concurrent .CSV writes
        $Mutex.WaitOne() > $null;       
        
        $AccountName = $using:AccountName    
        $Credential = $using:Credential
        $Region = $_.RegionName

        try {

            $EC2Instances = (Get-EC2Instance -Region $Region -Credential $Credential).Instances

            $EC2Instances | ForEach-Object {
                    
                $Instance = $_

                [PSCustomObject]@{           
                                                
                    Account    = $AccountName
                    Region     = $Region
                    VPC        = $Instance.VpcId
                    Subnet     = $Instance.SubnetId
                    InstanceId = $Instance.InstanceId
                    Platform   = $Instance.Platform
                    State      = $Instance.State.Name

                } | Export-Csv -Path "ec2_report.csv" -Append -Force

                $Mutex.ReleaseMutex();
            }
        }

        catch {

            $ErrorMessage = $Error[0]
            Write-Color -Text "$ErrorMessage ", "$AccountName ", "$Region" -Color Yellow, White, Cyan -LogFile "errors.txt"
        }
    
    } -ThrottleLimit $RegionThrottleLimit

} -ThrottleLimit $AccountThrottle

Instead of writing to the CSV multiple times and on top of that trying to fight concurrent operations, simply pipe the output to Export-Csv


$Regions | ForEach-Object -Parallel {                
                
    Import-Module -Name AWSPowerShell.NetCore, PSWriteColor #import the module for each Region runspace

    $AccountName = $using:AccountName    
    $Credential = $using:Credential
    $Region = $_.RegionName

    try {
        $EC2Instances = (Get-EC2Instance -Region $Region -Credential $Credential).Instances

        foreach($Instance in $EC2Instances){
            [PSCustomObject]@{                                         
                Account    = $AccountName
                Region     = $Region
                VPC        = $Instance.VpcId
                Subnet     = $Instance.SubnetId
                InstanceId = $Instance.InstanceId
                Platform   = $Instance.Platform
                State      = $Instance.State.Name
            }
        }
    }

    catch {
        $ErrorMessage = $Error[0]
        Write-Color -Text "$ErrorMessage ", "$AccountName ", "$Region" -Color Yellow, White, Cyan -LogFile "errors.txt"
    }

} -ThrottleLimit $RegionThrottleLimit | Export-Csv -Path "ec2_report.csv" -NoTypeInformation

I saw this suggested on other forums as well, but when I do this it does not generate the report properly. Keep in mind that the Region parallel loop is nested inside the Account parallel loop, so even though we are piping the output at the end of each region, it still might conflict with another account’s writes.

I think that this strategy requires writing to different files and combining them at the end.

Which should work, but I was curious to try the Mutex approach.

I suggest to write data into variables or an array of data, and then when done write data to file once, that’s better ie. more performant than writing multiple files.

System.Array is thread safe, so you should have no issues with simultaneous writes to array, then when done ex:

[array] $Data = @()

foreach ($Entry in $resources)
{
      # simultaneous writes to Data array
      $Data += $Entry
}

# when done
$Data | Export-Csv -Path "FileName.csv"

Are you sure that regular arrays are threadsafe?

I thought that one needed to use the ConcurrentBag class (https://docs.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentbag-1?view=net-6.0).

In any case thank you for the tip.

If I need to make it faster will re-visit, but for now I like having the separate reports and the merged reports :blush:

-Brandon

There is certainly a way to control the flow of logic to not require opening, writing, closing a file over and over. I’ll leave that up to you. The best alternative I could recommend would be using a synchronized hashtable or a concurrent dictionary so you could safely collect it all until you are ready to write the file once.

$sht = [hashtable]::Synchronized(@{})

$Regions | ForEach-Object -Parallel {                
                
    Import-Module -Name AWSPowerShell.NetCore, PSWriteColor #import the module for each Region runspace

    $AccountName = $using:AccountName    
    $Credential = $using:Credential
    $Region = $_.RegionName

    try {
        $EC2Instances = (Get-EC2Instance -Region $Region -Credential $Credential).Instances

        foreach($Instance in $EC2Instances){
            $obj = [PSCustomObject]@{                                         
                Account    = $AccountName
                Region     = $Region
                VPC        = $Instance.VpcId
                Subnet     = $Instance.SubnetId
                InstanceId = $Instance.InstanceId
                Platform   = $Instance.Platform
                State      = $Instance.State.Name
            }

            $sht.Add($Instance.InstanceID,$obj)
        }
    }

    catch {
        $ErrorMessage = $Error[0]
        Write-Color -Text "$ErrorMessage ", "$AccountName ", "$Region" -Color Yellow, White, Cyan -LogFile "errors.txt"
    }

} -ThrottleLimit $RegionThrottleLimit

# At the right time, write out the data
$sht.Values | Export-Csv -Path "ec2_report.csv" -NoTypeInformation

Note that each key in the hashtable has to be unique. I guessed that instanceID is unique. You can pick any property or combination of properties for use as the key since you really only care about the values.

There is also the option to use a generic integer index.

$sht = [hashtable]::Synchronized(@{})

$Regions | ForEach-Object -Parallel {                
                
    Import-Module -Name AWSPowerShell.NetCore, PSWriteColor #import the module for each Region runspace

    $AccountName = $using:AccountName    
    $Credential = $using:Credential
    $Region = $_.RegionName

    try {
        [array]$EC2Instances = (Get-EC2Instance -Region $Region -Credential $Credential).Instances

        for($i = 0, $i -lt $EC2Instances.count, $i++){
            $obj = [PSCustomObject]@{                                         
                Account    = $AccountName
                Region     = $Region
                VPC        = $EC2Instances[$i].VpcId
                Subnet     = $EC2Instances[$i].SubnetId
                InstanceId = $EC2Instances[$i].InstanceId
                Platform   = $EC2Instances[$i].Platform
                State      = $EC2Instances[$i].State.Name
            }

            $sht.Add($i,$obj)
        }
    }

    catch {
        $ErrorMessage = $Error[0]
        Write-Color -Text "$ErrorMessage ", "$AccountName ", "$Region" -Color Yellow, White, Cyan -LogFile "errors.txt"
    }

} -ThrottleLimit $RegionThrottleLimit

# At the right time, write out the data
$sht.Values | Export-Csv -Path "ec2_report.csv" -NoTypeInformation

Thank you for the help.

The multiple files works and will definitely give that a shot if I need better performance.

-Brandon

I’m not 100% sure but following 2 links may help to establish thread safe array:
Array.SyncRoot Property (System) | Microsoft Docs
Array Class (System) | Microsoft Docs

All it takes to SyncRoot your array, in PS for example:
SyncRoot property in Powershell - Stack Overflow