some trouble with invoke-command and error catching.

by teezee at 2013-02-27 10:02:08

this might be a long post. I won’t paste all the code as its over 500 lines, but ill post some on request. maybe I will just do a quick ‘proof of concept’ piece of code to show my problem.
I’m basically trying to get some information about file cound, and other stuff into an array using invoke command on many server at the same time. (asynchronoiusly as much as possible).

for now the part that give me trouble is to get the ‘good’ error message back into an array. it seems that if I use invoke command I need to pass along a unc path to get the good error message back, else it gives me .ctor whatever weird errors… let me try to explain this with some ‘visual’ approach…

function receivecommand
{
$runnumber = $args[0]
#$runnumber

$stop = 0
$result = @()
do
{
Foreach ($job in Get-Job)
{

if($job.state -eq ‘Completed’)
{
Write-Debug "boom..."
write-verbose "Receiving job id $($job.id) with name $($job.Name) and location $($job.Location)rn"
if ($runnumber -eq 1){$result += (Receive-Job -Id $job.id)}
if ($runnumber -eq 2){$result += (receive-job -id $job.id)}
if ($runnumber -eq 3){$result += (receive-job -id $job.id)}
Remove-Job -id $job.Id -Force

}
#else
#{
#Write-Debug "This shouldn’t appear!!!"
#write-verbose "$($job.id ) job with name $($job.Name) and location $($job.Location) is still runningrn" #$($job.command)
#sleep -Milliseconds 500
#}
$check = get-job

}
if($check -eq $null){$stop = 1}

}
until($stop -eq 1)

return $result

}


function pathtoolongtest
{
[scriptblock]$test=
{

$path = $args[0]
$servername = $args[1]

function FolderCountlight{
$path = $args[0]

$ErrorActionPreference = "continue"

function Recursive{
#$ErrorActionPreference = "silentlycontinue"
$dir = $args[0]
$i = $args[1]
try{
foreach ($d in $dir.GetDirectories()){

foreach ($f in $d.GetFiles()){
#$f #uncomment $f to get objects
$i.Count++ #comment this if you uncomment the $f above...
}

Recursive $d $i
}
}
catch{
if ($error[0].FullyQualifiedErrorId -match "PathTooLongException")
{$test += $error[0].FullyQualifiedErrorId;return $test;continue}
}

}


$leftover = New-Object System.IO.directoryinfo -ArgumentList $path
$addleftover = $leftover.GetFiles()
$statistics = New-Object PsObject -Property @{Count = 0}
$check = Recursive (new-object System.IO.DirectoryInfo $path) $statistics
$statistics = ($addleftover.count + $statistics.count)
$testing = ,($statistics, $servername, $check)
return ,$testing
}

$fullname = ("\"+$servername+""+$path)

$check = FolderCountlight $fullname
#$check = FolderCountlight $path
return $check
#return $statistics, $check
}
#$servername = "192.168.2.210"
$computername = @(("192.168.2.210"),("192.168.2.230"))
$pathz = "z$"

foreach ($servername in $computername)
{


$that = Invoke-Command -computername $servername -ScriptBlock $test -ArgumentList $pathz,$servername -AsJob -JobName 1

$that = @()
$that = receivecommand 1
$thatresult += ,($that)


}
return ,$thatresult


}

$output = pathtoolongtest
$output


This will output:
[quote]
VERBOSE: Receiving job id 95 with name 1 and location 192.168.2.210

VERBOSE: Receiving job id 97 with name 1 and location 192.168.2.230

4582
192.168.2.210
PathTooLongException
4582
192.168.2.230
PathTooLongException[/quote]
wish is exactly what I want. well somewhat…

So basically if you change the ips, and the path (USING $ sign) this works. it uses unc path to gather the information and return me everything I want the way I want it. this is what I used in the script to ‘fix’ my error catching stuff… well it is different but the patern is the same as above.

What happen in this situation is that it takes 3 times as much time to gather all the information on a folder that contain around 100k files for example. and the more there is the more times it takes. basically in the test i’ve been doing, without the unc path method I get around 35 seconds to count what I need and with the other way around it takes almost 200 seconds ! and this is not even half of the system I need to gather information from.

So to explain a little, if I change the line that call the function inside the scriptblock " $check = FolderCountlight $fullname " to for example " $check = FolderCountlight $path " and change " $pathz, to "z:" it will take ALOT less time. but I lose the ability to get the ‘path too long exeption’ in the final result… :frowning:

anyone have an idea of how I could possibly get to catch the error I want without having to rely on '\localhost\share$$path" method while using an invoke command. I’ve been trying alot of different things. but in this particular loop setup I couldn’t find any other way to do it. and like I said…
it drastically slow down the process.

Jon Go.

Thanks in advance for any tips :), I hope this was clear :X
by poshoholic at 2013-02-27 12:13:31
I’m starting to follow what you’re trying to do. Without digging in to find you a specific answer to your question, I find myself asking questions of my own. :slight_smile:

Are you not using Get-ChildItem to get the file/folder count because you want it done more quickly than PowerShell native cmdlets allow?
You seem to be doing this across multiple machines with large numbers of files and folders. Are you processing each machine one at a time? Because you could consider using jobs with remoting to do the work on multiple machines at once instead, which may make your performance concerns go away by scaling the work out across multiple machines.

Those questions aside, for the path too long exception, have you determined why you don’t get that exception when you use absolute non-UNC paths? Is it simply because the path is that much shorter?

I’d love to see this peeled back to a very simple example that reproduces the specific problem you’re facing if possible.
by teezee at 2013-02-27 17:10:21
Well, I must admit that this started off from a friend project and I kick it off to simply learn more about powershell but also, at the same time. some more "programmation way of doing script". I’m fairly new to anything related to programmation so some the choice I made might have better solutions :slight_smile:
last month I almost had to get-help every single line of code and google anything I would want to do… I think the letters of some keys wared off in the past few days… what was it, ctrl-c v not sure… :smiley:

I first chose to not use the cmdlet just to give me more challenge and find different ways to do the same thing with another piece of code of my own, then I figured that it greatly speed up the process. so at this point, I must admit that it is mostly for ‘speed’.

In my last (current) script, i’m currently using invoke command as jobs, and I changed the way it gathers all the job to be sure it run asynchronously on all machines, but this complicated things a little bit since I need the array to come back in a certain way, so I had to make up ‘fake path id’ (a friend idea) to be able to sort them back in order later. but all in all I cut the precious running time by alot. :slight_smile:

as for the problem itself, I have 2 version of the script, one wich use unc path and also gathers all the error in a single slot inside the multidimensional array. and one wich doesn’t.

The problem is that the one without the unc path return me the same error whatever the error is. so it doesn’t permit me to ‘filters’ the kind of error I want.
And well, the real problem is that if I use the one with the unc path, wich clearly work in itself… it takes twice as much time to run then the other version without unc path. it might not be a big difference, or maybe I’m just a little bit too touchy to let it go :slight_smile:

speedwise, here is some output without unc (so no error catch)
[quote]there is a total of: 43558 Folders
there is a total of: 263527 Files
it took 21.87 seconds to complete the operation…[/quote]

with unc path (with error catch):
[quote]there is a total of: 43558 Folders
there is a total of: 263534 Files
it took 88.04 seconds to complete the operation…[/quote]

and this is the only error I get when I don’t use unc path, only error meaning that any error change to this kind of error wich kind of mask the real name and disable my ability to filter the error by name, number or whatever…
[quote]Exception calling ".ctor" with "1" argument(s): "The given path’s format is not supported."
+ CategoryInfo : InvalidOperation: (:slight_smile: [New-Object], MethodInvocationExceptio
n
+ FullyQualifiedErrorId : ConstructorInvokedThrowException,Microsoft.PowerShell.Comman
ds.NewObjectCommand
+ PSComputerName : xxx.xxx.xxx.xxx[/quote]

(and this is less then 1/10 of the servers…)
At this point I wonder if it is not faster to use get-childitem :X, I doubt it but well I guess I could give it a try.

Jon

Edit:
I made a version using get child and it works, but its even slower then my slower version ! :frowning:

Edit 2:
oh lol, I think that I just figured that I don’t get any ‘path too long’ error simply because the function actually goes thru the sequence without giving me any error at all… I can get the ‘path too long’ file name without any problem… I’m very new to this as I said, but my brain told me that this must be because of the recursive function :frowning: I don’t know how all this work, but going recursively into subfolder seems to eliminate the long path error all in all… well that is what I think at least… I guess the only other work around I could find is to count the lenght and create an array with that and if it exeed some number it will basically point it out… no clue.
by teezee at 2013-02-27 19:05:56
This is the most ‘peeled’ simple example I could do to show the work around I found to make it ‘work’. but basically, I couldn’t find any other way with that piece of code to have it throw me a ‘toolongpath’ exeption :frowning:
function pathtoolongtest
{
[scriptblock]$test=
{

$path = $args[0]
$servername = $args[1]

function Recursive{

$dir = $args[0]
$i = $args[1]

foreach ($d in $dir.GetDirectories()){

if ($d.fullname -eq $null){$dir.fullname;return $test ;continue}

foreach ($f in $d.GetFiles()){

if ($f.fullname -eq $null){$d.fullname;return $test ;continue}
$i.Count++ #comment this if you uncomment the $f above...

}
Recursive $d $i
}

}

$leftover = New-Object System.IO.directoryinfo -ArgumentList $path
$addleftover = $leftover.GetFiles()
$statistics = New-Object PsObject -Property @{Count = 0}
$test = Recursive (new-object System.IO.DirectoryInfo $path) $statistics
$statistics = ($addleftover.count + $statistics.count)
$testing = ,($statistics, $servername, $test)
return ,$testing
}

$servername = "192.168.2.210"
$pathz = "z:"
$that = @()
$that = Invoke-Command -computername $servername -ScriptBlock $test -ArgumentList $pathz,$servername

$thatresult += ,($that)

return ,$thatresult
}

$output = pathtoolongtest
$output[0][0][2]

The array number is normal and needed.

the output basically give me this.

[quote]z]

simply said, this output the ‘parent’.fullname of the ‘folder’ or ‘file’ that can’t be ‘read/reach’

thanks, you made me think about something with your question that lead me to this :slight_smile:

Jon.

ps: I won’t lie that I still need to test it out see if it is faster then using the unc path :slight_smile: tomorrow I guess.
Edit:

there we go ! Happy with the new result so far :slight_smile:

version 1.2, that catch toolongpath exeption:
[quote]VERBOSE: --------------------------------------------------
there is a total of: 43781 Folders
there is a total of: 265977 Files
it took 136.89 seconds to complete the operation…
VERBOSE: --------------------------------------------------[/quote]

version 1.3, that catch toolongpath exeption and run asynchronously:
[quote]VERBOSE: --------------------------------------------------
there is a total of: 43775 Folders
there is a total of: 265981 Files
it took 79.61 seconds to complete the operation…
VERBOSE: -------------------------------------------------- [/quote]

version 1.4 that catch toolongpath with some ‘homemade workaround’ asynchronously:
[quote]VERBOSE: --------------------------------------------------
there is a total of: 43775 Folders
there is a total of: 265973 Files
it took 23.22 seconds to complete the operation…
VERBOSE: -------------------------------------------------- [/quote]

This is slowly getting alot faster :slight_smile:
Thanks for the reflexion :slight_smile:

Jon

edit:

I think im done optimising… :slight_smile:
version 1.5 :: thanks powershell jobs! ::
[quote]there is a total of: 74672 Folders
there is a total of: 671424 Files
it took 18.21 seconds to complete the operation…[/quote]