Get-unique items from an array

PowerShell Help
1

Hi,

I am trying to union two sets of data and get the distinct values back. The below code gets the top 5 processes by cpu and working set memory and unions the results together. I was expecting a distinct list of processes with the following id’s ( 4, 900, 2536, 3196, 4452, 6924, 7404, 1524) as you can see the processes with the id’s 7404 & 1524 are missing from the last result set that gets a unique set. I can not explain this. Any help would be appreciated.

$process = Get-Process
$cpu = $process | Sort-Object CPU -Descending | Select-Object -First 5
$ws  = $process | Sort-Object WorkingSet -Descending | Select-Object -First 5

$a = @()
$a += $cpu
$a += $ws

 
$cpu

  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName
    629       0      108        308     3 2,802.36      4 System     
    171      15    15024      21636   106 2,334.49   4452 vmtoolsd   
   3306     321   615960     772640  1349 1,415.93    900 devenv     
    313      24    17740      25096   105 1,393.74   2536 vmtoolsd   
   3558     328   606116     766860  1368 1,356.66   6924 devenv     

 
$ws

Handles  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName
   2969     393   694852     883436  1581   652.15   3196 devenv     
   2621     341   628636     778132  1430   343.76   7404 devenv     
   3306     321   615960     772640  1349 1,415.93    900 devenv     
   3558     328   606116     766860  1368 1,356.66   6924 devenv     
   2732     344   590188     761144  1390   598.62   1524 de

$a | Sort-Object id | Get-Unique

Handles  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName
    629       0      108        308     3 2,802.36      4 System     
    171      15    15024      21636   106 2,334.49   4452 vmtoolsd   
   3306     321   615960     772640  1349 1,415.93    900 devenv     
    313      24    17740      25096   105 1,393.74   2536 vmtoolsd   
   3558     328   606116     766860  1368 1,356.66   6924 devenv
2

Get-Unique compares each item in a list with the next item in the list and will only work against sorted lists.

If you want to user Get-Unique you have to sort the list first:

$a | Sort-Object -Property Id | Get-Unique

Or you could use Select-Object to select unique entries from the list:

$a | Select-Object -Unique
3

Hi Simon,

I have tried both approaches and they both yield the same results. Also I am already sorting the array am I not 

$a | Sort-Object id | Get-Unique
. Do you think that I need to sort the collection by something other than Id?

Thanks

4

You are right, Select-Object -Unique and Get-Unique seems to both select unique entries based on ProcessName.

To select unique entries based on only Id you could use this:

$a | Sort-Object -Property Id -Unique

If you want to select unique entries based on multiple values you would have to select all properties you want to use like this:

$a | Select-Object 'Handles', 'NPM', 'PM', 'WS', 'VM', 'CPU', 'Id', 'ProcessName' -Unique
5

Try Sort-Object -Unique since it lets you indicate the property that you want to sort\be unique:

$process = Get-Process
$cpu = $process | Sort-Object CPU -Descending | Select-Object -First 5
$ws  = $process | Sort-Object WorkingSet -Descending | Select-Object -First 5
 
$a = @()
$a += $cpu
$a += $ws

"Total object Count: {0}" -f $a.Count
$a

"Total unique ID's: {0}" -f ($a | Select ID -Unique).Count
$a | Select ID -Unique | Format-Table -AutoSize

$unique = $a | sort-object id -Unique
"Total with Sort-Object -Unique:  {0}" -f $unique.Count 
$unique

Output:

Total object Count: 10

Handles  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName                                                       
-------  ------    -----      ----- -----   ------     -- -----------                                                       
   3990     116   155344     183300   867 2,941.16   4124 OUTLOOK                                                           
   1529     135   152700     150592   594 2,420.08   3164 chrome                                                            
    221      59   180724     109000   358 2,208.63   2060 chrome                                                            
    564      15     4820       3520   107 1,258.55   2732 SynTPEnh                                                          
    428      32   143480     123344   460 1,190.34   3240 chrome                                                            
   1232     126   364880     313972  1147   955.64   6024 powershell_ise                                                    
   3990     116   155344     183300   867 2,941.16   4124 OUTLOOK                                                           
   3243     252   118944     162424   955   976.08   2692 explorer                                                          
   1529     135   152700     150592   594 2,420.08   3164 chrome                                                            
   2508      75   120028     130288   727   162.45   3368 lync                                                              
Total unique ID's: 8



  Id
  --
4124
3164
2060
2732
3240
6024
2692
3368


Total with Sort-Object -Unique:  8

Handles NPM(K)  PM(K)  WS(K) VM(M)   CPU(s)   Id ProcessName   
------- ------  -----  ----- -----   ------   -- -----------   
    221     59 180724 109000   358 2,208.63 2060 chrome        
   3243    252 118944 162424   955   976.08 2692 explorer      
    564     15   4820   3520   107 1,258.55 2732 SynTPEnh      
   1529    135 152700 150592   594 2,420.08 3164 chrome        
    428     32 143480 123344   460 1,190.34 3240 chrome        
   2508     75 120028 130288   727   162.45 3368 lync          
   3990    116 155344 183300   867 2,941.16 4124 OUTLOOK       
   1232    126 364880 313972  1147   955.69 6024 powershell_ise

Reference: http://blogs.technet.com/b/heyscriptingguy/archive/2012/01/15/use-powershell-to-choose-unique-objects-from-a-sorted-list.aspx

6

Thank you both Simon and Rob,

Both solutions worked for me. Simon I cant believe that I didn’t try:

$a | Select-Object 'Handles', 'NPM', 'PM', 'WS', 'VM', 'CPU', 'Id', 'ProcessName' -Unique

I had previously tried :

$a | Select-Object  -Unique 'Handles', 'NPM', 'PM', 'WS', 'VM', 'CPU', 'Id', 'ProcessName'

which did not work for me.

Rob I didn’t even know that I could use -unique on a sort. That worked really well as using the select-object method did what I wanted but messed up the formatting which required me to format the table using expressions to tidy things back up.

$a | Select-Object 'Handles', 'NPM', 'PM', 'WS', 'VM', 'CPU', 'Id', 'ProcessName' -Unique | 
        Format-Table @{n='Handles'; ex={$_.Handles}; align='right'; width=10} ,
                     @{n='NPM'; ex={$_.NPM}; align='right'; width=7}, 
                     @{n='PM'; ex={$_.PM}; align='right'; width=15}, 
                     @{n='WS'; ex={"{0:N2}" -f $_.WS}; align='right'; width=17},
                     @{n='VM'; ex={$_.VM}; align='right'; width=15}, 
                     @{n='CPU'; ex={"{0:N2}" -f $_.CPU}; align='right'; width=12},
                     @{n='Id'; ex={$_.Id}; align='right'; width=7}, 
                     @{n='ProcessName'; ex={$_.ProcessName}; align='right'; width=20} -wrap

vs

$a | sort-object id -Unique

Once again thanks both of you. I learned something new today.

7

Thanks for your help. i know this is post is old but I found it helpful and just wanted to mention you can also use the below code to match all the property values of the object of any array. I have found this very useful importing CSV files that have overlapping lines (like application logs from different days) and no single key column.

$LineUnique = $a | Select-Object * -Unique