Removing all instances of duplicates

PowerShell Help
1

I used the below script to export a csv with computer names and object id’s from azure

There are a number of duplicate files that I would like to remove.

My csv looks like so

name, object id

comp1. id1

comp1, id1

comp2. id2

comp3, id3

I would like to remove these duplicates. I want to remove both the instances so when I output the csv we would only see comp2 and comp3. I have seen plenty scripts to remove just the duplicate but I want to remove any and all instances of duplicates.

Could someone assist with the coding for this I am fairly new and can’t find info online.

Get-MsolDevice -All -LogonTimeBefore ‘may, 2019 12:00:00 AM’ | select-object -Property ObjectID, DisplayName | export-csv “C:\temp\test\List.csv” -NoTypeInformation

2

This will do what you need:

$csv = Import-Csv -Path 'C:\temp\test\List.csv' -NoTypeInformation

$result = foreach ($row in $csv) {

    $duplicateCount = ($csv -match $row).Count

    if (-not ($duplicateCount -gt 1)) {
        Write-Output $row
    }
}

# Call $result or pipe back to the CSV file path to update it
$result

Edit: Replaced [pscustomobject] with $row.

3

You can use Group-Object to filter out duplicates … 

Import-Csv -Path ‘Your csv file’ |

Group-Object -Property Name |

Where-Object -Property Count -eq -Value 1 |

Select-Object -ExpandProperty Group

4

Just an FYI in case it may not be clear:

Olaf has provided an excellent example of how to filter out duplicates based on the Name, and groups them by the name (comp1, comp2, etc.) regardless of what the object id is. So if duplicates in the object id don’t matter, go this nice, clean route.

If you need to find duplicates based on the Name and object id, something like my original post would work.

Depending on your need, either solution should suffice.

5

If you really like to remove the duplicates of both properties you can simply append another treatment after the first one for the second property … like this:

Import-Csv -Path ‘Your csv file’ |

Group-Object -Property Name |

Where-Object -Property Count -eq -Value 1 |

Select-Object -ExpandProperty Group |

Group-Object -Property ‘object id’ |

Where-Object -Property Count -eq -Value 1 |

Select-Object -ExpandProperty Group