Best way to build a collection out of MSMQ

Here is my challenge. I’ve taken a script that used to take a couple hours to process. Now, by leveraging jobs, I was able to reduce it to 2 minutes. Since this script does a large amount of output, (I previously wrote all of the output to flag csv files, then read them all in and the end and generated reports from them), I leveraged MSMQ to dump all of the out from the jobs. My intent is to come back at the end after all the jobs complete, read in each of the queues and create my report.

I have read that using += to add a large number of objects to an array is a bad idea that gets exponentially slower with a large amounts of data.

My questions is, what would be the best way to read in all of the objects from the queue so I can build a report? Before I read in a csv, used ConvertTo-HTML, and away I went.
I know I would have to build a loop to pull out each message to add it to a table or some such.

Any thoughts?

There are a couple of options. You can use the .NET List or ArrayList classes, or you can create a PowerShell function that just ouputs the objects, and let the engine take care of turning it into a collection. Here are examples of each:

$arrayList = New-Object System.Collections.ArrayList
for ($i = 0; $i -lt 10000; $i++)
{
    $null = $arrayList.Add($i)
}

$array1 = $arrayList.ToArray()

function Test
{
    for ($i = 0; $i -lt 10000; $i++)
    {
        $i
    }
}

$array2 = Test

In general, the second option will give better performance, because most of the work is performed by compiled .NET code, but both are much better than using += on arrays.

Sounds like the second method is what I want to use, just not sure how to implement it with what I have.

For example, here is out the object is created:

[xml]$UpdatedUsers = “<user>
<username>$Username</username>
<firstname>$FirstName</firstname>
<lastname>$LastName</lastname>
<title>$Title</title>
<email>$yEmail</email>
   </user>”

Now, to get the few thousand objects back from MSMQ, I can do something like:

$UserInfo= (Receive-MSMQueue -Name "UpdatedUsers" -Private -TargetType XML)

That gives me 1 xml object back from the queue into an object. I’m stuck on how to repeat that till the queue is empty and have one object I can plug into my report I’m creating. Before I just used += (which gets ugly when it gets big).

Any help is appreciated.

What happens when the queue is empty and you call Receive-MSMQueue? Does it just return $null, or does it produce an error?

In this case, you don’t even need to have a function, if you don’t want to. The results of a loop can be assigned directly to a variable, like this (assuming, for this example, that a return value of $null is what happens when there’s nothing left in the queue):

$UserInfo = while ($true)
{
    $object = Receive-MSMQueue -Name "UpdatedUsers" -Private -TargetType XML
    if ($null -eq $object)
    {
        break
    }
    else
    {
        $object
    }
}

Yes, it returns $Null when it’s empty. I think this is exactly what I’m looking for. I’ll plug this in on Monday and let you know how it goes. Thanks so much, you just made my weekend! :slight_smile:

No problem! :slight_smile: Here’s a shorter way of writing that, if you don’t mind assigning variables inside your loop condition:

$UserInfo = while ($null -ne ($object = Receive-MSMQueue -Name "UpdatedUsers" -Private -TargetType XML))
{
    $object
}

The end result is the same.

I admit, I couldn’t wait till Monday. I got excited about it and worked on it yesterday and have it up and running. Still need to handle a lot of special error handling/checks, but overall, the script looks like it will be a success! And it’s SO much faster!
Previously it took 30+ minutes to process 500 users with no data changes (and considerably longer when it actually had to update data). In my test runs, it’s processing 1500 users, updating over 700 of them, and takes < 3 minutes. That’s HUGE.

I have encountered a couple anomalies with Microsoft’s Active Directory Module though. Seems to give errors when too many jobs are calling Get-ADUser at the same time. I worked around it, but it’s an annoying little bug for sure. I thought about seeing if I could do what I need in .Net vs using the module to not have to deal with it at all.

Thanks again for your help! You have made my Monday!