Export slowness for data ALREADY in memory

Hello,

I’ve been using PowerShell for several years, but I’ve noticed a very strange behavior and I’d like to get your opinion on whether this is normal or if it can be fixed/worked around.

Here’s my test: I retrieve the list of users, once by specifying a domain controller and once without.

$UserA = Get-ADUser -Filter * -Properties *
$UserB = Get-ADUser -Filter * -Properties * -Server $DC

Then I export them to CSV, for example.

The problem is as follows: the export of UserB takes MUCH longer.
I’m not referring to the time it takes to execute the Get-ADUser command, just the CSV export.
Export A takes 0.3s
Export B takes 46s

If I disconnect my network connection befoire the CSV export, the export of UserB is incomplete, with some properties being empty (such as the email field).
So, it seems that PowerShell is not storing everything in memory, and some properties appear to be fetched DURING the export :frowning: .

So, is this normal?
How can I make this process faster?

Let me know if you want my test script to reproduce the behavior:

$UserA = Get-ADUser -Filter * -Properties * -SearchBase $OU
$UserB = Get-ADUser -Filter * -Properties * -SearchBase $OU -Server $DC
(Measure-Command {$UserA | Export-Csv -Path “.\UserA.csv”}).TotalSeconds
“Export with DC”
(Measure-Command {$UserB | Export-Csv -Path “.\UserB.csv”}).TotalSeconds

You are utilizing the pipeline for the entire command. Try storing the data before exporting

$UserA = Get-ADUser -Filter * -Properties *
$UserA | Export-Csv -Path ".\UserA.csv"

and

$UserB = Get-ADUser -Filter * -Properties * -Server $DC
$UserB | Export-Csv -Path ".\UserB.csv"

That’s what I’m doing, the data is stored in a variable before the export.
You have my script at the end of the message.

that’s pretty interesting. When I’m at work I’ll try to reproduce this and see if i get the same experience.

Oh you’re right, I didn’t see that. Since the only difference is the server being specified, can you confirm the speed is the same for all the DCs?

Hey there Durand,

While I don’t have an answer, I wanted to simply share that I was able to replicate in my environment, at least for the ‘first’ run . If I ‘up arrow’ and rerun, its fine, and fast. Very strange. I did try a few servers and they all did the same, but not specifying the server at all, seems to ‘work around’ whatever the issue is. It’s weird because, $UserB should be the thing in ‘memory’. I confirmed in my environment that $UserA and $UserB were also identical.

Do you have to specify a DC in your process?

Was messing around with this more… and I found a strange workaround as well, give it a shot:

$UserB = Get-ADUser -Filter * -Properties * -SearchBase $OU -Server $DC | Select-Object -Property *
$userb | Export-Csv -Path ".\UserB.csv" -Verbose

By adding the Select-Object -Property * , which would seem redundant, it’s doing something that allows that export to go faster when specifying a DC ( for the first export, which is of course probably what matters for you).

If you plan on using that as a workaround, I’d suggest adding a comment to explain why, as that would appear to be very strange to anyone reviewing your code. I’d still love to know why this is behaving like this, as it seems like a psuedo bug to me.

I specify a DC to ensure I’m on the primary and have recent information (our replication times are about 1 hour).

@dotnVo: If the export is run several times in a row, it becomes very fast indeed.
I had noticed that too. For me, your example is functional.

I finally found the “why”:

This also explains why your example works because the properties are initialized from the first request.

1 Like

Thanks for the share! Really helpful to know that, though I wish it was more clear from a PS perspective (many folks in PS aren’t going to be aware or even know how to ‘google’ this), that by specifying a domain, it does a ‘lazy-initiationalization’ .

Thanks for the followup @Durand , that’s the first I had heard of lazy initialization.
I wanted to chime in just to say I played along too (finally).
Following your example pulling $UserA took a minute 51 seconds, and $UserB took a minute 48 seconds and they both ended up being 3806 users.
I didn’t see anything different between the objects themselves manually indexing through a few of them, so I did the Export test and it took 1.14 seconds for $UserA and 263.9 seconds for $UserB. Immediately re-running the $UserB export only took .99 seconds.
Definitely seems like the lazy initialization explains it.

Cool find.

BTW, not sure if it was just for your example or how you’re normally doing it but requesting all of the properties with -Properties * really slows things down too, depending on number of users.
I will often write an array of the properties I really want to see and pass that as an argument.

getting all properties using Get-ADUser definitely slows things down, though in this particular case the slowness being perceived is more about the lazy init stuff. However, it would speed things up more to only get the properties you want on the get call. Typically you just won’t notice it for a few number of users as its usually very small, but for large runs, it makes a significant difference.

I’m not requesting all properties, it was just an example and to make the script as short as possible for easier readability.

Yea no worries, think it was just an observation zero made