Issue installing software using Jobs.

I’ve come across a very unusual problem. I’ve used jobs for parallel tasks against lots of devices, and have had lots of success with it. Right now though I’ve got an issue where my script runs fine but the line that kicks off the software install fails. This is where it gets really odd. I can take that same line of code that failed in the jobs script, run just that line and it will not fail.

The software I’m trying to install is VMWare View. Its an upgrade for thin client devices with file based write filters. It generates some logs that don’t help much. In the event viewer i see an error with the following entry “Product: VMware Horizon View Client – Error 1306.Another application has exclusive access to the file C:\ProgramData\VMware\VDM\logs\debug-2016-10-09-141938.txt. Please shut down all other applications, then click Retry.”

Don’t know why access to a txt file would stop the install from succeeding but that’t the only error in event viewer. The other verbose logs don’t help much. I could post my code, but i’m hoping someone else that uses jobs may have run into this problem before. Well I guess i’ll post the code, as to avoid someone asking for it.

#installing view 3.5.2
#10/7/2016

#Sam Kachar II

[string]$computers = cat ‘C:\352\testinstall.txt’
$total = $computers.Count
$maxjobs=70 #Number of Jobs to Run at one time
#$chunksize = 2 #Number of machines to run per job; thought i would use this but decided it better to just run one machine per job.
$jobs=@() #Initialize $job variable

#script block that will be called by start-job
$sb = {
param($computers,$i)

$comp = $computers[$i]

if (Test-Connection -computername $comp -Count 1 -Quiet){ #is the machine online??? If online log as online, failure will be logged

   
  
  try{
    
    Invoke-WMIMethod -Class Win32_Process -Name Create -Computername $comp -ArgumentList "cmd /c fbwfmgr.exe /disable" | Out-Null
   
    Restart-Computer -ComputerName $comp -Force

    sleep -Seconds 90
        try{
            Copy-Item -Path 'C:\352\x86\' -Destination "\\$comp\c$\install\352\" -Force -Recurse | Out-Null
            
            }
            
            catch{
            sleep -Seconds 30
            Copy-Item -Path 'C:\352\x86\' -Destination "\\$comp\c$\install\352\" -Force -Recurse | Out-Null
            }

       finally{
      write-host "running finally block now"  
    #Invoke-WMIMethod -Class Win32_Process -Name Create -Computername $comp -ArgumentList "cmd /c c:\install\352\install352x86.cmd"
      sleep -Seconds 20
    #This line calls the actual install. It works outside of the job, but not when the script runs in its entirety. The above cmd had this line in it.
    #I pulled the line out to narraw down the issue. If i run just this line the software installs...
    Invoke-WMIMethod -Class Win32_Process -Name Create -Computername $comp -ArgumentList "cmd /c c:\install\352\352x86.exe /S /V`" /qn`""
      sleep -Seconds 180
         } 
    [bool[]]$x 

   $p = 0
   $x = "true"
   while($x){
   sleep -Seconds 1
   $p++
   #write-host $p
   $viewversion = (dir "\\$comp\c$\install\vmwareview\ThinClientShell.hta").LastWriteTime
   if($viewversion.Year -eq 2016)
   {
   break
           }
           elseif ($p -gt 120){
           
           break
           }
    
          
   }
   #Invoke-WMIMethod -Class Win32_Process -Name Create -Computername $comp -ArgumentList "cmd /c fbwfmgr.exe /enable"
    "$comp" | Out-File -LiteralPath "c:\352\online\installed.txt" -force -Append  
    }
 catch
 {
      "$comp" | Out-File -LiteralPath "c:\352\failed\failed.txt" -force -Append
       }
         
    }
    

    #If the machine was offline then this will log the machine name for later processing

else {

    "$comp" | Out-File -literalpath "c:\352\offline\offline.txt" -force -Append
   }

}
#This for loop is what controls the number of jobs and fire’s them off
for ($i=0;$i -lt $total; $i++){
$jobs += Start-Job -ScriptBlock $sb -ArgumentList($computers,$i)

#creates a variable with job objects eq to running state
$running = @($jobs | ? {$_.State -eq 'Running'})

While ($running.Count -ge $maxjobs) { #if the count of currently running jobs is ge the set ceiling 
$finished = Wait-Job -Job $jobs -Any   #use the wait-job cmdlet to pause script execution from creating anymore jobs
$running = @($jobs | ? {$_.State -eq 'Running'}) #once a job finishes this line runs for re-evaluation
}

}
Wait-Job -Job $jobs > $null #this line pauses script execution until all remaing jobs finish, i tested this and it works

Jobs run under an odd security context - it’s possible that this installer needs a full user context in order to survive. It’s also possible you’re running into some kind of concurrency problem. That text file appears to be a debug log file for the installer, and if it can’t get exclusive write access for whatever reason, it’s blowing up. That could be another app, as the error implies, or lack of access due to the security context.

I appreciate the quick reply. I have run procmon on the device when the script hits the part where it calls the wmi process instantiation and wasn’t able to find anything odd or obvious. I too thought about security context. I did validate on the device when it starts the process for the view install(file name 352x86.exe) inside of task manager it shows up as my credentials and not some weird account or security context(e.g. system, network, etc etc).

I did read on a forum to try and modify the folder permissions containing the exe to read only, but idt thats the problem as the exe is mearly a wrapper for the msi. When i look in the logs i can see where its running the install out of the windows\installer directory. So it does create the msi file in the right spot in both situations.

Going off what you said though, is there something i could try for remediation. I’m willing to try anything that you think might help find a solution or workaround. Perhaps i should try the script without the jobs feature. Only downside is i have to run this against hundreds of devices. Not being able to run them in parallel would take, by my calculations conservatively 32 hours. That number will go up because these devices are on the other side of some really slow links. But it’d be a min of 32 hours, realistically i’d be in the ball park of 40 to 50 hours.

Since you’re doing all the crazy work of throttling jobs and starting processes, is it safe to assume Remoting is out of the question?

Yep that is correct. Security has that all locked down, and is why i have to go this route. It’d be great if i could just use invoke command or pssession but neither are an option unfortunately. I’ll be the first to admit i’m the farthest person from being a powershell advanced user, however this was the only way i could get this type of work done. I used SCCM to upgrade the lions share of devices. These are the devices that don’t have an sccm client which is another story for another day. What i will say on that topic is client push is out for these devices because of the write filter.:-(. Perhaps i’ll circle back and install the client after i get this immediate issue tackled.

I’m looking through the logs in WinMerge. One is a failed install (i.e. the entire script ran) and one is successful (single command ran byitself, but its the same cmd. I just highlighted the cmd in the script and hit run hightlighted code). I’m seeing several spots where its throwing that 1306 and all of it has to do with the removal of the old client. The log file is nearly 2 megs so i’ve got a ways to go yet…but if this is all i find it doesn’t help me move forward at all because i’m still in the same boat of not understanding what makes a cmd ran under the job context so different than run outside of the job by itself. Been googling but nothing helpful has come up, hece posting here. No where else to turn.

Found the solution i was looking for. Turned out if i just waited an extra 90 seconds after the copy of the new client version that did the trick. No clue why that worked buts its worked 6 out of 6 tries. I spent 2 days digging into this issue this past weekend. Tried procmon, no help. Tried parsing the event viewer but only found the errors i already had found. Had this not worked i would have went down the path of killing the process and releasing the lock on the file. First time i ran into a piece of software that wouldn’t install because it couldn’t write to a txt file. Thanks VMWare for equally sucking…but i can’t complain to much. If it wasn’t for these problems then i wouldn’t be able to have a job.

Issue closed. Hope this helps someone else.