Deleting duplicate entries

PowerShell Help
1

Hi,
I have a text file withe below data & I am trying to delete duplicate content.
Aggregation Start Time - 2016-07-23 22.30.01
Aggregation End Time - 2016-07-24 00.07.15
Time taken for whole process - 5833684
Aggregation Start Time - 2016-07-23 22.30.01
Aggregation End Time - 2016-07-24 00.07.15
Time taken for whole process - 5833684
Aggregation Start Time - 2016-07-22 22.30.00
Aggregation End Time - 2016-07-23 00.06.30
Time taken for whole process - 5790047
Aggregation Start Time - 2016-07-22 22.30.00
Aggregation End Time - 2016-07-23 00.06.30
Time taken for whole process - 5790047
Aggregation Start Time - 2016-07-21 22.30.00
Aggregation End Time - 2016-07-22 01.10.59
Time taken for whole process - 9659321
Aggregation Start Time - 2016-07-21 22.30.00
Aggregation End Time - 2016-07-22 01.10.59
Time taken for whole process - 9659337
Aggregation Start Time - 2016-07-20 22.30.00
Aggregation End Time - 2016-07-20 23.35.56
Time taken for whole process - 3956665
Aggregation Start Time - 2016-07-20 22.30.00
Aggregation End Time - 2016-07-20 23.35.56
Time taken for whole process - 3956665
Aggregation Start Time - 2016-07-19 22.30.00
Aggregation End Time - 2016-07-19 23.30.55
Time taken for whole process - 3655256
Aggregation Start Time - 2016-07-19 22.30.00
Aggregation End Time - 2016-07-19 23.30.55
Time taken for whole process - 3655256
Aggregation Start Time - 2016-07-17 22.30.00
Aggregation End Time - 2016-07-17 23.32.02
Time taken for whole process - 3722827
Aggregation Start Time - 2016-07-17 22.30.00
Aggregation End Time - 2016-07-17 23.32.02
Time taken for whole process - 3722827
Aggregation Start Time - 2016-07-18 22.30.00
Aggregation End Time - 2016-07-18 23.30.56
Time taken for whole process - 3656366
Aggregation Start Time - 2016-07-18 22.30.00
Aggregation End Time - 2016-07-18 23.30.56
Time taken for whole process - 3656366

Code:
$hash = @{} # define a new empty hash table
gc C:\Opscripts\Kalyan\Vision_Data_Aggregation\filter.txt | %{if($hash.$_ -eq $null) { $_ }; $hash.$_ = 1} > C:\Opscripts\Kalyan\Vision_Data_Aggregation\final.txt

But output is coming as below:
Aggregation Start Time - 2016-07-21 22.30.00
Aggregation End Time - 2016-07-22 01.10.59
Time taken for whole process - 9432
Time taken for whole process - 9432
Aggregation Start Time - 2016-07-20 22.30.00
Aggregation End Time - 2016-07-20 23.35.56
Time taken for whole process - 3863
Aggregation Start Time - 2016-07-19 22.30.00
Aggregation End Time - 2016-07-19 23.30.55
Time taken for whole process - 3569
Aggregation Start Time - 2016-07-17 22.30.00
Aggregation End Time - 2016-07-17 23.32.02
Time taken for whole process - 3635
Aggregation Start Time - 2016-07-18 22.30.00
Aggregation End Time - 2016-07-18 23.30.56
Time taken for whole process - 3570

Can someone please help me, where I am missing?

-Kalyan

2

(gc .\text.txt | group).name

3 
Get-Content "C:\Opscripts\Kalyan\Vision_Data_Aggregation\filter.txt" | 
Select-Object -Unique | Out-File "C:\Opscripts\Kalyan\Vision_Data_Aggregation\final.txt"
4

Hi,
Thanks. Tried the same & it worked fine.

Aggregation Start Time - 2016-07-24 22.30.00
Aggregation End Time - 2016-07-25 00.12.13
Time taken for whole process - 6133243
Aggregation Start Time - 2016-07-23 22.30.01
Aggregation End Time - 2016-07-24 00.07.15
Time taken for whole process - 5833684
Aggregation Start Time - 2016-07-22 22.30.00
Aggregation End Time - 2016-07-23 00.06.30
Time taken for whole process - 5790047
Aggregation Start Time - 2016-07-21 22.30.00
Aggregation End Time - 2016-07-22 01.10.59
Time taken for whole process - 9659321
Time taken for whole process - 9659337
Aggregation Start Time - 2016-07-20 22.30.00
Aggregation End Time - 2016-07-20 23.35.56
Time taken for whole process - 3956665
Aggregation Start Time - 2016-07-19 22.30.00
Aggregation End Time - 2016-07-19 23.30.55
Time taken for whole process - 3655256

Is there a way, I need a line gap for each and every ‘Time taken for whole process’?

my code:
sc VIS_PRD_Data_Aggregation_Log.txt (gc final1.txt | % {$l = 1} {if ($l++ % 3) {$} else {$,‘’}})

-Kalyan

5

Thanks all.
Modified the script a bit & all seems to be working.

-Kalyan