Read CSV file in chunks

Hi All,

I have wrote a program that analyse the csv file. But csv file is very huge (500MB+) and my server hangs/stops while executing the script. I want to read csv file in chunks. But i can’t figure out how to make it possible. Help please…!!

Have you tried to search for that problem? What have you tried so far? Show your code please.

I have this code but it takes forever on slow machine

$PSScriptRoot

write-host $PSScriptRoot

$loc = $PSScriptRoot

$reader1 = [System.IO.File]::OpenText($loc + “C:\Users\hemant.bagade\Downloads\BeforeInstallation_10.43.130.61_large.csv”)
$reader2 = [System.IO.File]::OpenText($loc + “C:\Users\hemant.bagade\Downloads\BeforeInstallation_10.43.130.61.csv”)
try {
for(;:wink: {
$line1 = $reader1.ReadLine()
$line2 = $reader2.ReadLine()
if (($line1 -eq $null) -and ($line2 -eq $null))
{ break }

else {
Compare-Object -ReferenceObject $line1 -DifferenceObject $line2 -Property “filename”
}
}
}
finally {

}
$reader1.Close()
$reader2.close()

Here is a whole article on the topic…

Reading large text files with Powershell The thing to understand is that the right tool to use in Powershell is determined from the kind of file you are dealing with. Here's the bunch of options you have for reading text files: http://www.happysysadm.com/2014/10/reading-large-text-files-with-powershell.html
Reading Text Files Fast There are plenty of ways how PowerShell can read in text files, and they can differ considerably in time. Check for yourself. The examples below illustrate different approaches and measure the execution times. Just make sure the path in the example exists, and if not, choose a large text file to play with. https://community.idera.com/database-tools/powershell/powertips/b/tips/posts/reading-text-files-fast-1152612330
Break Text File into Chunks with PowerShell https://blogs.technet.microsoft.com/heyscriptingguy/2015/09/22/break-text-file-into-chunks-with-powershell
PowerShell – Splitting large log files https://discoposse.com/2011/09/29/powershell-splitting-log-files