Hello, can someone please help me with the following (I had a similar thread but I started a new one for clarify) I want to find an alternative to using Where-Object in a pipeline. I have two large CSV files (where some of the fields are the same and some are different). I have a forach loop at the…

[image] cxmelga: I had a similar thread but I started a new one for clarify If it’s the same issue that doesn’t make any sense. Didn’t you want to compare two CSV files? [image] cxmelga: I want to find a record ‘directly’ for example where the property sAMAccountName is equal to Fred I …

Hello Thanks for the reply much appreciated At the moment my script takes days to run, I am trying to speed it up. I basically have two CSVs of data so I import them both and save as a variable $A = import-csv C:\Data\file1.csv $B = import-csv C:\Data\file2.csv foreach ($item in $A) { $B | w…

You can make a hash table then your lookups would be quicker. There are several ways to accomplish this, one is using Group-Object $A = Import-Csv C:\Data\file1.csv $B = Import-Csv C:\Data\file2.csv | Group-Object -Property SamAccountName -AsHashTable foreach($item in $A){ $B[$item.uniqueID] }…

[image] cxmelga: The above is slow, I want to remove the ‘where’ and find the item in $B directly where the sAMAccountName matches a string (e.g. $item.UniqueID) without going through all the records each time in the loop Did you try the code suggestion I gave in the other thread? …

Thanks very much for your replies, I like to idea of Group-Object I tried this but unfortunately the data takes too long to load (just sits there), there are about 63,000 rows. I guess I could split the file up into smaller files first, but then I will have to add the CSV header back to each file, …

Hmm are you running this on a 1 cpu/2GB ram virtual machine? 63k is a drop in the bucket. Let us set up a ~63K line CSV with 3 properties. $date = Get-Date $tempfile = New-TemporaryFile 1..62KB | % { [PSCustomObject]@{ String = "$([char]$_)" * 10 Number = $_ * 1KB Dat…

[image] krzydoug: Perhaps the issue isn’t with the size of the CSV? I was thinking almost the same. At least it’s not just the amount of lines I suspect. It’s one file with about 63000 rows and one with about 21000, right? How many columns do the files have? [image] cxmelga: I like to i…

Hello everyone, thank you very much for your help so far, much appreciated I have placed a copy of the script and two sample CSV files are the following location on my onedrive [image] Script Folder If you try it with the files (see parameters section at th…

$Start = Get-Date $ADData = Import-Csv -Path D:\sample\ADData.csv $Data = Import-Csv -Path D:\sample\Data.csv | Select-Object -Property *,@{Name = 'sAMAccountName'; Expression = {$_.UniqueID}} $Result = Compare-Object -ReferenceObject $Data -DifferenceObject $ADData -Property sAMAccountName -Includ…

It’s probably the rest of your script that’s slow, because neither the Compare-Object nor the Group-Object portion is. With Write-Host (for still seeing output when measuring expression) Measure-Command -Expression { $A = Import-Csv .\Data.csv $grouped = Import-Csv .\ADData.csv | Group-Ob…

Picking out a record in a DataTable

PowerShell Help

Olaf April 29, 2021, 4:14pm 5

Did you try the code suggestion I gave in the other thread?

Topic		Replies	Views
Speeding up comparision of two CSV file imports PowerShell Help	12	1051	May 4, 2021
ForEach in an Foreach / Comparing multiple Arrays PowerShell Help	8	260	April 25, 2017
Need to match column data across CSV's. PowerShell Help	13	1442	August 27, 2019
Remove duplicates from csv taking too long PowerShell Help	20	535	April 17, 2020
Comparing CSV to Active Directory, then output fields to a table PowerShell Help	31	982	April 8, 2021

Picking out a record in a DataTable

Related topics