Reading large text file

PowerShell Help
1

Hi All

I have a large text file,over 6 million lines (A list of primes) and I am trying to import it into a array. Get-Content is way to slow so I have tried to use the DotNet to help out. This is what I have tried so far.

$a = New-Object -ComObject Scripting.FileSystemObject
$f = $a.OpenTextFile("$ENV:USERPROFILE\Documents\Primes\PrimeList.Cfg", 1)
[Array]$Primes=$f.ReadAll()

What I end up with this every element of the array is a single character. Part of the array looks like this.

1

2

3

5

7

1
1

What I want is

1
2
3
5
7
11

Any help would be appreciated.
Thanks
Tim

2

Hey,
We would need to understand your input file.

I created a file that contains the following

1
2
3
5
7
11
13

I then ran you code against it

$a = New-Object -ComObject Scripting.FileSystemObject
$f = $a.OpenTextFile("D:\input.txt", 1)
[Array]$Primes=$f.ReadAll()
$Primes

Results:

PS C:\Windows\System32\WindowsPowerShell\v1.0> 
$a = New-Object -ComObject Scripting.FileSystemObject
$f = $a.OpenTextFile("D:\input.txt", 1)
[Array]$Primes=$f.ReadAll()
$Primes
1
2
3
5
7
11
13
3

I did the same thing, Curtis, and got the same results you did. TeeStar, what do the contents of your file look like? Can you show us maybe the first 10 lines or so? You might open your .cfg file in something like notepad++ and see if there are any special characters that could be causing issues.

4

There must be \r on each line of .cfg file you are trying to read.
I’m very new to Powershell myself but it would happen in Python.
Try to strip any \r and \n character.

5

Thanks everyone

My Notepad ++ output looks fine as well as Notepad. No special characters or anything.

I have tried this

$Primes=[System.IO.File]::ReadAllLines("$ENV:USERPROFILE\Documents\Primes\PrimeList.Cfg")

and it seems to work. I don’t know why this one and the other does not work, must be the way I saved it perhaps??

6

Use Format-Hex for looking into your file.
You can see any of “0D/0A” combinations, but only several of it correct :slight_smile: