Fill array quickly

PowerShell Help
1

Hi everyone,

I am trying to create a massive array (5 Million items) and fill them with random items. I have tried the following

$Data = 1..5000000 | ForEach-Object {(Get-random -Maximum 5000000)}

The problem is that this method is far to slow. I was wondering if there is a way to use DotNET to quicken this up a bit.

Many thanks in advance!

2

TeeStar,
Welcome back to the forum. :wave:t4: ā€¦ long time no see. :wink:

Somewhat faster even without dot net would be this already ā€¦

$data =
foreach ($Item in (1..5000000)) {
    Get-random -Maximum 5000000
}

Why is it not fast enough? Whatfor do you need it?

3

This took about 2.5 seconds

$random = [System.Random]::new()

$data = foreach($loop in 1..5000000){
    $random.Next()
}

If you want to limit the size of the number you can pass in the max value using a different constructor.

$random = [System.Random]::new()

$data = foreach($loop in 1..5000000){
    $random.Next(50000)
}
1 Like
4

The method takes about 5 seconds to add 5 million random items out of 6 million available.

$mylist = New-Object -TypeName System.Collections.ArrayList
$mylist.AddRange((Get-Random -Count 5000000 -InputObject (1..6000000)))
# To measure the time it takes for command to execute, put your code in a scriptblock after 
# Measure-Command cmdlet.
# Example: 
Measure-Command {
$mylist.AddRange((Get-Random -Count 5000000 -InputObject (1..6000000)))
}
1 Like
5

Thanks everyone, I appreciate the help

This is an experiment to see the differences between native PowerShell CMDLets and DotNET especially when working with large data sets.

The first one I did was read a password list with Get-Content vs [System.IO.File]
PowerShell

Measure-Command -Expression {$PW_PS=Get-Content $Path}

DotNET

Measure-Command -Expression {$Pw_DN=[System.IO.File]::ReadAllLines( ( Resolve-Path $Path ) )}

The second one I did was to sort that list
PowerShell

Measure-Command -Expression {$SortedPw_PS= $Pw_PS | Sort-Object}

DotNET

Measure-Command -Expression {$SortedPw_DN= [array]::sort($Pw_DN)}

The third test was to create an array with 10 million items filled with random numbers between 1 and 10 million. For this one I used three different tests with PowerShell, DotNET and Hybrid.

PowerShell

$items=10000000
Measure-Command -Expression { [Array]$Array1_PS = Get-Random -Count $Items -InputObject (1..$Items)}

DotNET

$Items=10000000
$Random = [System.Random]::new()
Measure-Command -Expression   { 
  [Array]$Array1_DN = ForEach ($Loop in 1..$Items)
   {
    $Random.Next(1,$Items)
   }
  }

Hybrid

$Items=10000000
$Array1_HB= New-Object -TypeName System.Collections.ArrayList
Measure-Command -Expression   {$Array1_HB.AddRange((Get-Random -Count $Items -InputObject (1..$Items))) }

Here are the results

Time in seconds to read large data file with native Powershell 21.5216665
Time in seconds to read large data file with DotNET 0.0693576

Time in seconds to sort data with native Powershell 18.4273818
Time in seconds to sort data with DotNET 3.0236697

Time in seconds to create an array with 10 million items with native Powershell 21.1890782
Time in seconds to create an array with 10 million items with DotNET 141.3131941
Time in seconds to create an array with 10 million items with Hybrid 30.30894

Total time 236.0904982

Here the time to create an array with 10 million items is the outlier. The results are not what I expected at all. I think the Foreach is causing a major slow down.

These tests were run on a Lenovo T580 i7-8650U with Windows PowerShell

Anyone have have any ideas on how to speed these up? Any feedback would be appreciated!
Thanks again
Tim

6

Iā€™m curious ā€¦ whatfor? :thinking: Do you want to confirm that dot Net is faster than PowerShell? You should be aware of that this is nothing new. :man_shrugging:t4:
Do you need those kind of tasks for something particular?