Read huge binary file

hi, I not a native English speaker, sorry if there any mistake,

some month ago i do this :
https://forums.powershell.org/t/find-and-replace-2-byte-binary-file/15308/5
it works, but when i tried to open a 250mb file, the program cannot handle it,
i forgot the error log text, but it about “Get-Content”, and ram consume about 8GB,
my question is : how to solve this?
alternative : i tried to split the 250mb into 1.44mb using certain software, but when powershell read second data using same syntax but different file name (I guess it will be overwrite, turn out its not) the memory keep rising,
my second question is : what syntax to “flush”(it correct term right?) that ram data without closing powershell “window”?

thank you for reading, have a nice day

Using a switch statement to read each line of my 225MB text file took 6 seconds compared to Get-Content (26 seconds). I didn’t notice any increase in system memory usage.

Measure-Command {
    $file = Get-ChildItem -Path .\test.txt
    switch -Regex -File $file
    {
        'TextToSearchFor' {$_}
        Default {}
    }
}

thank you for the answer,
I a newbie in this field, I just need some clarification:
A. the “switch -Regex -File $file” is a loop command that reads each line right?
B. the “$_” is the variable name of “the current line” right?
C. what “Default {}” command do? I google it and found nothing!

sorry if I bring more question, thank you again for the reply

Have you tried the StreamReader .NET class? I have never worked with files this large but it is the recommended approach when dealing with large amounts of text.

$Myfile = New-Object System.IO.StreamReader{C:\CSLS_Testing_SG_276.txt}

$Myfile.ReadToEnd()
1 Like

thanks for the reply,
the syntax you provided is work, the memory/ram only 400mb,
but i have 1 problem:

#$bytes = Get-Content “D:\iso\iso.iso” -Encoding Byte -ReadCount 0

$bytes = New-Object System.IO.StreamReader{D:\iso\iso.iso}
Read-Host -Prompt “1.Press any key to continue”

#Convert the bytes to hex
$hexString = [System.BitConverter]::ToString($bytes)
Read-Host -Prompt “2.Press any key to continue”

and the result:

Cannot find an overload for “ToString” and the argument count: “1”.
At D:\iso\coba.ps1:7 char:1

  • $hexString = [System.BitConverter]::ToString($bytes)
  • + CategoryInfo          : NotSpecified: (:) [], MethodException
    
  • FullyQualifiedErrorId : MethodCountCouldNotFindBest

can you help me, please?

I am not sure I fully understand what you are trying to accomplish but try this:

$bytes = New-Object System.IO.StreamReader{D:\iso\iso.iso}
$bytes = $bytes.ReadToEnd()
Read-Host -Prompt “1.Press any key to continue”

#Convert the bytes to hex
$hexString = [System.BitConverter]::ToString($bytes)
Read-Host -Prompt “2.Press any key to continue”

thanks again for the reply,
i tried to find and replace byte data type in the binary file as I show in this link
that link works if file under 10mb, but i tried to edit 250mb of file,
please note, I am a newbie, and not a native English speaker, sorry if I rude

Have you tried what I suggested in my previous post? I don’t think $bytes was a string in your last attempt. Try adding a line with ‘bytes = $bytes.ReadToEnd()’.

i tried the “$bytes = $bytes.ReadToEnd()”
then in task manager: the ram / rising over 8gb, then the program crash to window

Then you should probably stick to Get-Content. Is that working for you? If not, what is the problem you are facing?

the Get-Content is working, but when data under 10mb, if the data over 10mb, the powershell will comsume ram data over 8gb then crash,
this is the problem i face (i copy paste from my first post):

hi, I not a native English speaker, sorry if there any mistake,

some month ago i do this :
https://forums.powershell.org/t/find-and-replace-2-byte-binary-file/15308/5
it works, but when i tried to open a 250mb file, the program cannot handle it,
i forgot the error log text, but it about “Get-Content”, and ram consume about 8GB,
my question is : how to solve this?
alternative : i tried to split the 250mb into 1.44mb using certain software, but when powershell read second data using same syntax but different file name (I guess it will be overwrite, turn out its not) the memory keep rising,
my second question is : what syntax to “flush”(it correct term right?) that ram data without closing powershell “window”?

thank you for reading, have a nice day

If you split the file, you can use a foreach loop to read all the fragments. You can also assign the value to a variable which will be called after the file name. Then you can either set the variable to $null or remove it altogether which should free up RAM.

For instance, assume you have a folder with 100+ files, which contain each part of the original large file. The path is C:\Files in this example.

foreach ($Item in Get-ChildItem -Path C:\Files) {
    New-Variable -Name $Item.Name -Value (Get-Content -Path $Item.FullName -Encoding Byte -ReadCount 0)

    $bytes = Get-Variable -Name $Item.Name | Select-Object -ExpandProperty Value

    $hexString = [System.BitConverter]::ToString($bytes)

    # Here you would have to insert the code where you do something with variable $hexString

    Remove-Variable -Name $Item.Name -Confirm: $false -ErrorAction SilentlyContinue

    Remove-Variable -Name $hexString -Confirm: $false -ErrorAction SilentlyContinue
}
1 Like

sorry for the really late reply, I am too focus because your syntax works,
thank you very very much

Great! Glad that it finally worked.