# Help cleaning an array of 'duplicates'

**URL:** <https://forums.powershell.org/t/help-cleaning-an-array-of-duplicates/15089>\
**Category:** PowerShell Help\
**Created:** [October 14, 2020, 9:48am UTC](https://forums.powershell.org/t/help-cleaning-an-array-of-duplicates/15089 "2020-10-14T09:48:35Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ertuu85](https://avatars.discourse-cdn.com/v4/letter/e/bb73d2/32.png) [@ertuu85](https://forums.powershell.org/u/ertuu85)\
**Post date:** [October 14, 2020, 9:48am UTC](https://forums.powershell.org/t/help-cleaning-an-array-of-duplicates/15089/1 "2020-10-14T09:48:35Z")

</div>

I have a script that generates an array that has output as shown below:

[test group2] contains [test group1]  
[test group3] contains [test group2]  
[test group1] contains [test group2]  
[test group2] contains [test group3]

&nbsp;

The lines such as [test group2] contains [test group1] and [test group1] contains [test group2] are essentially duplicates, just in a different order. I really only need one or the other, not important which I keep.

How could I go about clearing out any duplicates from the array?

---

<div class="post-metadata">

**Author:** ![grokkit](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.powershell.org/grokkit/32/970_2.png) [@grokkit](https://forums.powershell.org/u/grokkit)\
**Post date:** [October 14, 2020, 10:15am UTC](https://forums.powershell.org/t/help-cleaning-an-array-of-duplicates/15089/2 "2020-10-14T10:15:52Z")

</div>

Hmmm… if [test group1] = [test group2] then  
[test group1] contains [test group2] = [test group2] contains [test group1]

but if [test group1] **!=** [test group2] then  
[test group1] contains [test group2] **!=** [test group2] contains [test group1]

I’m not really following the logic here…

---

<div class="post-metadata">

**Author:** ![krzydoug](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.powershell.org/krzydoug/32/1008_2.png) [@krzydoug](https://forums.powershell.org/u/krzydoug)\
**Post date:** [October 14, 2020, 10:39am UTC](https://forums.powershell.org/t/help-cleaning-an-array-of-duplicates/15089/3 "2020-10-14T10:39:47Z")

</div>

So the same would be true for [test group2] contains [test group3] and [test group3] contains [test group2], correct? If so you could just use a switch utilizing regex with a list and only add to the list if both don’t already exist in a line. It’s a lot easier to show than express with words so please see the example.

To simulate a Get-Content $somefile we’ll use a here-string split at new lines. Then we’ll prepare an empty list and finally process each line through the switch.

```
$output = @'
[test group2] contains [test group1]
[test group3] contains [test group2]
[test group1] contains [test group2]
[test group2] contains [test group3]
'@ -split [environment]::NewLine

$result = [System.Collections.Generic.List[string]]::new()

switch -Regex ($output){
    '\[(.+)\].+\[(.+)\]'{
        if( -not ($result | Where {$_.contains($Matches.1) -and $_.contains($Matches.2)}))
        {
            $result.Add($_)
        }
    }
}
```

And what ends up in $result are the first two that weren’t duplicated.

```
$result

[test group2] contains [test group1]
[test group3] contains [test group2]
```

The switch statement can also read directly from files. See the following expanded example.

```
$tempfile = New-TemporaryFile

@'
[test group2] contains [test group1]
[test group3] contains [test group2]
[test group1] contains [test group2]
[test group2] contains [test group3]
[test group2] contains [test group8]
[test group3] contains [test group7]
[test group1] contains [test group8]
[test group4] contains [test group7]
[test group7] contains [test group4]
[test group7] contains [test group3]
[test group7] contains [test group1]
'@ | Set-Content $tempfile -Encoding UTF8

$result = [System.Collections.Generic.List[string]]::new()

switch -Regex -File ($tempfile){
    '\[(.+)\].+\[(.+)\]'{
        if( -not ($result | Where {$_.contains($Matches.1) -and $_.contains($Matches.2)}))
        {
            $result.Add($_)
        }
    }
}

$result

[test group2] contains [test group1]
[test group3] contains [test group2]
[test group2] contains [test group8]
[test group3] contains [test group7]
[test group1] contains [test group8]
[test group4] contains [test group7]
[test group7] contains [test group1]
```

---

<div class="post-metadata">

**Author:** ![ertuu85](https://avatars.discourse-cdn.com/v4/letter/e/bb73d2/32.png) [@ertuu85](https://forums.powershell.org/u/ertuu85)\
**Post date:** [October 14, 2020, 11:00am UTC](https://forums.powershell.org/t/help-cleaning-an-array-of-duplicates/15089/4 "2020-10-14T11:00:04Z")

</div>

This works perfectly!

&nbsp;

Thank you!

---

<div class="post-metadata">

**Author:** ![dotnVo](https://avatars.discourse-cdn.com/v4/letter/d/4af34b/32.png) [@dotnVo](https://forums.powershell.org/u/dotnVo)\
**Post date:** [May 16, 2024, 8:28pm UTC](https://forums.powershell.org/t/help-cleaning-an-array-of-duplicates/15089/5 "2024-05-16T20:28:42Z")

</div>

Closed on May 16, 2024, 8:28pm UTC


