Memory optimization

Hey all,

So a new issue i am encountering when working with large data, is my servers running out of memory and then the script grinding to a slow pace.

The script i am running is as follows:

$SourceFile = Import-Csv -Delimiter “;” -Path

foreach($User in $SourceFile)
{

$HomeServer = Get-QADObject -Identity $User.DistinguishedName -IncludeAllProperties

If($HomeServer.msExchHomeServerName.Length -eq 0)
    {

    $User |select DistinguishedName, userPrincipalName, mail, ObjectClass | 
    Export-Csv -NoTypeInformation -Delimiter ";" -Path Forest_msExchHomeServerName_Empty.csv' -Append

    }

}

Now I am currently scanning around 40000 users and my server (which has 16gb memory) has maxed out and the script is now running extremely slowly. Now i suspect it has to do with my variable management and maybe i need to clear the variable, but i am not to sure.

How can i fine tune this script so that it doesn’t consume so much memory?

And does anyone have any good articles around performance tweaks that can be used in PowerShell scripting?

Hi Recco,

here are some suggestions you could try (see comments)

https://gist.github.com/anonymous/172a9d3aad4cdc323e6f

Since this script generates a PSObject, you are able to output the content however you want, for example:
Script.ps1 | Export-CSV -Path file.csv -Delimiter “;”
or
Script.ps1 | Out-GridView

You could also measure the runtime duration with “Measure-Command”. Would be interesting to see if it really improves and if so, how much.

Regards,
Michael

Few things here:

  1. You’re importing your entire CSV into memory and holding it there. If it’s a very large file, that can be a problem.

  2. You’re using the Quest AD module. While I don’t have personal experience with this, I have found several posts online stating that this module seems to cause huge memory usage problems. Even when your script doesn’t look like it should be causing this, the module seems to hold onto object references so they don’t get properly garbage collected.

  3. In conjunction with #2, you’re using that -IncludeAllProperties switch. So every object that the QAD module is holding in memory will be that much bigger.

The best way I’ve found to save on memory usage is to stream everything through one long pipeline. It’s not too pretty, probably takes longer to run, but the memory usage is capped quite a bit, especially for those big queries. So something like this

import-csv -delimiter ';' -path  | 
    get-qadobject -identity $_.distinguishedname -properties mail, objectclass, homeserver |
        ?{$_.msexchhomeservername.length -eq 0} |
            select distinguishedname, userprincipalname, mail, objectclass |
                export-csv -NoTypeInformation -Delimiter ';' -path 'Forest_msExchHomeServerName_Empty.csv'
    

Thanks for the responses!

I am working in an environment that has 26 domains in the forest. So the problem i am having, is if i use “get-adobject” it wont search cross domains even if i reference to it through the DN. I need to specify -server contoso1.directory.com and only then will it search for the object in that domain.

Using the DN, how can i change the -server to match the domain.

So for example if the DN is “OU=Users,DC=Contoso1,DC=Directory,DC=com”, then the switch would be contoso1.directory.com?

My 1st thought is an “If” statement, followed by “Else”, but to do that 26 times seems a little cluttered. I part of me is saying i can use regex somehow, but i am still very new to regex and would need to do more research.

How best can i match the “get-abobject -server” to domain, based on the info in the DN text?

$server = $user.DistinguishedName -split ',' -match '^DC=' -replace '^DC=' -join '.'
Get-ADUser $user.DistinguishedName -Server $server

Is your domain name always 3-deep, e.g., contoso1.directory.com?

@Dave
Cool let me mess around with that.

@Bob
Always

If it’s always the same depth, then you could do a RegEx match and then replace.

# sample data
$dnList = @"
CN=USER_TEMPLATE,OU=sierra,DC=foo,DC=tango,DC=local
CN=DOC-VIEWER,OU=Users,OU=sierra,DC=foo,DC=tango,DC=local
CN=Ricoh Copier,OU=Users,OU=sierra,DC=bar,DC=tango,DC=local
CN=Scan User,OU=Users,OU=sierra,DC=hq,DC=tango,DC=local
CN=Sonic Wall,CN=Managed Service Accounts,DC=foo,DC=tango,DC=local
CN=Orderdesk,OU=Web contacts,OU=sierra,DC=foo,DC=tango,DC=local
CN=Technicalsupport,OU=Web contacts,OU=sierra,DC=bar,DC=tango,DC=local
CN=tango SALES,OU=Web contacts,OU=sierra,DC=foo,DC=tango,DC=local
CN=Amanda Bines,OU=Users,OU=sierra,DC=foo,DC=tango,DC=local
"@ -split "`r`n"

$pattern = ".+,DC=(\w+),DC=(\w+),DC=(\w+)"
foreach ($dn in $dnList)
{
    if ($dn -match $pattern)
    {
        $dn -replace $pattern, '$1.$2.$3'
    }
}

So this is the final script i am now testing:

Get-PSSnapin -Registered | Add-PSSnapin

$SourceFile = Import-Csv -Delimiter “;” -Path Import_file.csv

$SourceFile | ForEach-Object{

$server = $_.DistinguishedName -split ',' -match '^DC=' -replace '^DC=' -join '.'

$HomeServer = Get-ADObject -Identity $_.DistinguishedName -Server $server -Properties msExchHomeServername, DistinguishedName, userPrincipalName, mail, ObjectClass

If($HomeServer.msExchHomeServerName.Length -eq 0)
    {

       $Report = @()
                ForEach-Object{
                $Report += New-Object psobject -Property @{
                'DistinguishedName' = $HomeServer.DistinguishedName
                'userPrincipalName' = $HomeServer.userPrincipalName
                'mail' = $HomeServer.mail
                'ObjectClass' = $HomeServer.ObjectClass
                }

                              }

    }

}

My servers memory (16gb) was constantly getting maxed out. So I did some research about when to use Foreach vs Foreach-object, and it seems that in my case using Foreach-object is better. It wont be as fast as Foreach, but will be better on memory usage.

Is there a way to only search a X number of lines at a time from the $Sourcefile, clear all variables and the process X+1 and so forth?

You’re still storing everything in memory for the duration of your script. every ad object, every output property, every list. Try this code:

Import-csv -Delimiter ";" -Path Import_file.csv | 
    ForEach-Object { Get-adobject -Identity $_.distinguishedname -server $($_.DistinguishedName -split ',' -match '^DC=' -replace '^DC=' -join '.') -Properties msExchHomeServername, DistinguishedName, userPrincipalName, mail, ObjectClass} | 
    where-object { $_.msExchHomeServerName.Length -eq 0} |
    Select-Object distinguishedname, userprincipalname, mail, objectclass

Note there are no equal signs, everything just gets sent down the pipeline. The select-object will create your pscustom object for you, and the where-object is the pipeline equivalent of your if statement, it just filters. Should have much better memory utilization. It will output to the screen, but if you want it to a file just add a pipe to export-csv at the end.

niiiiiiice!!!

Thanks Jeremy, ill test that!

Again, thanks for all the help all!

@Dave
In the following “-match ‘^DC=’ -replace ‘^DC=’”

What does the “^” actually stand for? I have been doing some syntax searches and haven’t found anything yet.

-match and -replace in PowerShell use regular expressions, and ^ is an anchor in regex which matches the beginning of a string (or of a line, depending on how you’ve configured the options.) http://www.regular-expressions.info/anchors.html for more info. :slight_smile:

As always a massive help!

Thanks again!