runspacepool hangs with more 30 WMI connectserver

PowerShell Help
1

Hello,

I use this script :

I modify the scriptblock because my goal is to connect to more 30,000 remote machines via WMI and create a process on them also via WMI.
I can do in a different way, and i do in a different way with a professionnal tool. But i want to explore the way with powershell and WMI.

Before, i must describe the contexte : 30,000 computers in workgroup, not domain, with different “admin” accounts and different “user” “pwd”.
A file gives : "IP,“user”, “pwd”
The computer “sender” : W2K8R2SP1 with PS V3, 4 Go Ram
And i test on a few computers (1000) .

So, in the scriptblock, for each computer, i do :

  • connectserveur via WMI with credentials
  • create contexte (command, arg, current directory…) for the “createprocess” wmi request
  • call the remote “createprocess” wmi request
    note : the remote process for my test is “powershell” and the script lauched is present on all remote machine ( write the date/time in a log file )
    If necessary I can put my scriptblock code tomorrow because i don’t have it at this time.

My poor investigation :
All things is good if $throttleLimit in egal to 20 ( less than 25 in fact ) for 1,000 remote computers ( but too many hours! ).
Powershell “Sender” hangs along 1 or 2 minutes if $throttleLimit is greater than or egal 30. With WireShark, i see the all connectserver but not one createprocess in the network trames when it fail.
I modify the scriptblock and replace wmi requests connectserver and createprocess with simple commands “net use IP USER PWD and a copy from sender to remote via SMB and copy-item \ip…”.
All things is good is $throttleLimit if egal to 200 !

My questions :
Do you know a limite in “fan out” WMI ? or in Powershell V3 ?
Someone can do or want to do the test?
How can i investigate ? i try to trace WMI with activated ETW-debug but it seem to me that the “sender” don’t log the “out” activity.

Please, help me !
Respectfully

Ps : i’m a poor french guy and i don’t write/read not very well english.

2

To be of any help, we’d probably need to see the script block that you’re running. Using runspace pools like this is a neat trick, but there’s basically zero support or documentation on it from Microsoft. From what I’ve read on the subject, there is no thread safety or locking mechanism available in this scenario. If that’s true, you have to be very careful about how you write the script blocks.

You’d get better support (though not necessarily better performance) if you used workflows and a “ForEach -Parallel” loop. Have you tried that, by chance?

3

Keep in mind that the WMI commands are, technically, deprecated. They’re not receiving further investment. They rely on an older, less-efficient protocol (RPCs/DCOM). So in the kind of scale you’re looking for, WMI itself just might not be up to the task. There isn’t a hard limit, it’s just a matter of your environment and what, specifically, you’re doing.

Microsoft would probably suggest that this is exactly what Remoting was designed to solve. I realize that might not be an option for you, but that doesn’t mean WMI is necessarily a good option, either. It just wasn’t designed with this kind of scenario in mind.

You also didn’t mention HOW you’re running this. Are you using Get-WmiObject, I assume? I agree with Dave in that I don’t think I’d personally attack this using runspace pools. Not that they’re a bad technology, but they aren’t as well-understood as some other options, such as jobs.

4

Hello Sirs,
Thank you very much for your fast reply.
The remote computers run under XPSP3. I choosed WMI because connections seem to be faster than winrm and invoke-command.
I use runspacepool with “net use … copy … net use /d” each week on 2,000 xp remote computers because we are in a scenario of migration and we need to collecte many informations about “users activity”, different informations each week.
Never the “sender” computer hangs.
And it’s difficult to change to an other technologie ( i’m old ).
I’ve install PS V3 but no time to learn. I stay with my poor knowledge “PsV2”.
t’s true : i use scripts writed by other without understand all the things. I use the “modify/test” method to work and i don’t go “under the hood”. For sample, i don’t understand raison/consequence of this : $pool.ApartmentState = “STA”

I find : http://us.generation-nt.com/answer/advanced-wmi-multiple-wmi-connections-network-bottleneck-help-25992852.html
May i think there is a bug in WMI, not in powershell, not in dot.net ?

About job, it seem to me that it’s not fast :

I’ll try “foreach -parallele” next week. It seems to looks like tread of “dot.net”. But what is under the hood?

OK, WMI is old. But “Remoting” with scripting, even PS, is too long in large scale without domain. We work actually for tomorrow and W8…sorry W7, for 2015. Oh, yes, we are always in late.

Respectfully

My scriptbock ( in my memory because i forget the code at my office ) :
The “in” parameters are $computer, $user, $mdp

$cmd=‘c:\windows\system32\cmd.exe /c c:\test.cmd’
$locator = New-Object -ComObject “WbemScripting.SWbemLocator”
$oRemoteWmi = $locator.ConnectServer($computer,“root\cimv2”,$user,$mdp)
$oRemoteClassProcess=$oRemoteWmi.Get(“Win32_Process”)
$oPS=$oRemoteWmi.Get(“Win32_ProcessStartup”)
$oMethodeCreate=$oRemoteClassProcess.Methods_|?{$.name -eq ‘create’}
$oInParam=$oMethodeCreate.inParameters.SpawnInstance()
($oInParam.Properties_ | where {$.name -eq “CommandLine”}).Value = $cmd
($oInParam.Properties | where {$.name -eq “CurrentDirectory”}).Value = "c:"
($oInParam.Properties | where {$.name -eq “ProcessStartupInformation”}).Value = $oPS
$outParam = $oRemoteClassProcess.ExecMethod(“Create”, $oInParam)
#lecture code du pid
$remotePid = ($outParam.Properties_ | ?{$_.name -eq “ProcessId”}).Value
Write-Host “remotePid = $remotePid”

5

Hello Sir,
Today i can test with 7,800 remote computers XPSP3. I launch script from a notebook W2K8R2SP1 with PS v3 end i do it with ps console not ps_ise. Someone says that one is sta and other is mta but i don’t kown which is which.

First, i constate two things :

  • for a runspacepool new created
    PS C:\ $throttleLimit=200
    PS C:\ $sessionState = [system.management.automation.
    runspaces.initialsessionstate]::CreateDefault()
    PS C:\ $runspacePool = [runspacefactory]::CreateRunsp
    acePool(1, $throttleLimit, $sessionState, $host)
    PS C:\ $runspacePool.ApartmentState
    Unknown <<<< ???
  • and for the console :
    PS C:$Host.Runspace.ApartmentState
    Unknown <<<< ???
    So, i don’t know how to be sure of ApartmentState for the two object.

Second, the scriptblock used for the test ( like on precedent post ) :
net use \ip pwd /u:user
copy local \ip
net use \ip /d
connectserver WMI resquest
createprocess wmi resquet

Third, my 4 tests :

a - without $runspacePool.ApartmentState = “STA” before $runspacePool.Open() :

  • with the same script and trottlelimite = 30 for 7,800 remote computers XPSP3 : NOK (PS no respond, no progress, visible with wireshark on the network :only one a serie of 50 connectserver, and no createprocess on the net )

b - without $runspacePool.ApartmentState = “STA” before $runspacePool.Open() :

  • with the same script and trottlelimite = 20 for 7,800 remote computers XPSP3 : OK
    it takes more 14 hours with trottlelimite = 20 without sta

c - with $runspacePool.ApartmentState = “STA” before $runspacePool.Open() :

  • with the same script and trottlelimite = 500 for 7,800 remote computers XPSP3 : OK
    it takes 5H30 ( more 14 hours with trottlelimite = 20 without sta )
    all the time ps turns : ( a little more ) 500 process 2000 threads 90%RAM 90%CPU, 100,000 handles ( values evolue a little, positive/negative )

d - with $runspacePool.ApartmentState = “STA” before $runspacePool.Open() :

  • new scriptblock :
    connectserver, provider remote registry
    GetWordValue/SetWordValue WMI request
    no net use/copy
  • i must leave office before it finish but i see it during 1 hour
  • with a new script and trottlelimite = 500 for 7,800 remote computers XPSP3 : OK
    before to leave my office, i can see in taskmgr :
    all the time ps turns : 52 process 800 threads 90%CPU (maxi green 1/3 red) no more RAM used 1,5G ( 4Go maxi)
    It’s cool !

During many weeks i use the script without sta and with trottlelimit=200 to make “net use/copy” without error. PS hangs only when i introduced connectserver wmi request and use trottlelimit greater than 30.

So, it seems to works well for me. And fast. Ok XPSP3 on the remote computers. But i do work to eat. And i do work fast.

I buy 2 books ( LEARN WINDOWS POWERSHELL and POWERSHELL IN DEPTH ) and i hope i’ll work more faster. ;-))
Is futur “remoting” so faster without runspacepool ? See you in 2015.
Respectfully.

6

In PowerShell v2, the console defaulted to MTA, but that was changed to STA by default in v3. The ISE is always STA.

You can see this by running the following command in whichever host you’re using:

7

Also, the WbemScripting.SWbemLocator COM object requires STA (its ThreadingModel is “Apartment”; see http://msdn.microsoft.com/en-us/library/windows/desktop/ms682390(v=vs.85).aspx ). That may have been the cause of your hanging problems. If you had used the .NET classes in System.Management (or even better, the Invoke-WmiMethod PowerShell cmdlet), you probably would have been better off.

Edit: Here’s what I’m talking about with Invoke-WmiMethod. This entire block of code:

$cmd=’c:\windows\system32\cmd.exe /c c:\test.cmd’
$locator = New-Object -ComObject “WbemScripting.SWbemLocator”
$oRemoteWmi = $locator.ConnectServer($computer,”root\cimv2″,$user,$mdp)
 $oRemoteClassProcess=$oRemoteWmi.Get(“Win32_Process”)
 $oPS=$oRemoteWmi.Get(“Win32_ProcessStartup”)
 $oMethodeCreate=$oRemoteClassProcess.Methods_|?{$_.name -eq ‘create’}
 $oInParam=$oMethodeCreate.inParameters.SpawnInstance_()
 ($oInParam.Properties_ | where {$_.name -eq “CommandLine”}).Value = $cmd
 ($oInParam.Properties_ | where {$_.name -eq “CurrentDirectory”}).Value = “c:\”
($oInParam.Properties_ | where {$_.name -eq “ProcessStartupInformation”}).Value = $oPS
 $outParam = $oRemoteClassProcess.ExecMethod_(“Create”, $oInParam)
 #lecture code du pid
 $remotePid = ($outParam.Properties_ | ?{$_.name -eq “ProcessId”}).Value
 Write-Host “remotePid = $remotePid”

Can be shrunk down to this:

$cmd=’c:\windows\system32\cmd.exe /c c:\test.cmd’
$result = Invoke-WmiMethod -Class Win32_Process -Name Create -ArgumentList $cmd,'c:\'
Write-Host “remotePid = $($result.ProcessId)”
8

Hello Sir,

I am really impressed by your answers.
It is a great lesson for me. Thank you.
I understand better the need to adapt the same ThreadingModel between WMI and PS.
Also,I naively transposed a vbs in PS code forgetting to think about. But even thinking, I would never think of invke-wmimethod.
Thank you really much for all these explanations.
Respectfully.
Noel

9

No problem at all. :slight_smile: Bonne chance!