Command exist and does not exist at the same time

So … I encountered something very weird:

I create a task that is running as a group managed service account:

$Trigger = New-ScheduledTaskTrigger -Once -At (get-date (Get-Date).AddSeconds(2) -Format o)
$Action = New-ScheduledTaskAction -Execute "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -Argument ("-File {0}" -f "c:\temp\do-stuff.ps1")

$Principal = New-ScheduledTaskPrincipal -UserID gmsa-test$ -LogonType Password
$Task = New-ScheduledTask -Action $Action -Trigger $Trigger -Principal $Principal

$Task | Register-ScheduledTask -TaskPath test -TaskName 123

The script that is run:

try {
    $ErrorActionPreference_old = $ErrorActionPreference
    $ErrorActionPreference = "stop"
    $env:USERNAME | Out-File "C:\temp\do-stuff.txt"
    Get-Date -Format "o" | Out-File -Append "C:\temp\do-stuff.txt"

} catch {
    $msg = $_
    $env:USERNAME | Out-File "C:\temp\do-stuff.txt"
    Get-Date -Format "o" | Out-File -Append "C:\temp\do-stuff.txt"
    $msg | Out-File -Append c:\temp\do-stuff.txt
    $ErrorActionPreference = $ErrorActionPreference_old
}

$ErrorActionPreference = $ErrorActionPreference_old

The content of do-stuff.txt after execution:

gmsa-test$
2016-11-17T15:29:43.7208152+01:00
Out-File : The term 'Out-File' is not recognized as the name of a cmdlet, function, script file, or operable program. C
heck the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\temp\do-stuff.ps1:4 char:21
+     $env:USERNAME | Out-File "C:\temp\do-stuff.txt"
+                     ~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Out-File:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

So: The Out-File is failing because it does not exist … and this error is written to a file by using Out-File(!).

What could possibly be going on here?

It’s not easy to reproduce this for all scripts. Just minor changes to the script and it might run as expected.

EDIT:

I just confirmed that the same script might fail and might not fail when the task executes. This must be some kind of race condition in powershell.exe? It’s almost as if the script is running before powershell is loaded…?

Does this go away if you’re not running it as a managed service account?

@donj

I haven’t been able to produce this error when using other accounts. I don’t know if you saw my update, but this looks like a race condition of some kind (so it might be possible to reproduce as non-GMSA).

Yeah, thats’ kind of what I was wondering. It’s almost like the execution environment is in restricted mode for a moment. I’m not sure that’s something we’re going to be able to help you resolve - except, I dunno, maybe adding a Start-Sleep into the beginning.

Well … sadly:

If I put start-sleep at the top it will fail with:

“Start-Sleep : The term ‘Start-Sleep’ is not recognized as the name of a cmdlet, function, script file, or operable prog
ram. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.”

Something weird is going on here :\

Yeah. Let me ask around a bit.

I had some trouble recently with powershell and scheduled tasks. No matter what I did, it would not honor the working directory parameter for the scheduled task. I even had to hard code the location of the script in the powershell.exe parameters. I didn’t want to hard code it into the script, but it needed to be set since I read and write files in the script directory. I don’t know why this fixed it, but I created a .bat file (in the same directory as the script) to call the powershell.exe command (without hard coding the the script path), used the .bat file in the scheduled task, and everything started working as expected.

One good thing to check - at the top of your script, see if you can output what’s in $env:PSModulePath. Some of us are wondering if the environment isn’t populated, which means PowerShell wouldn’t be able to find commands. If you’re not able to get a command to run to output that, you may need to do it with a raw .NET class, since those don’t depend on the environment existing.

What version of PowerShell are you running? I remember a similar problem from a thread over on the TechNet forums, but it was supposed to be fixed in PowerShell 4:

https://social.technet.microsoft.com/Forums/windows/en-US/edc4ae2d-b804-4361-94de-7482533a8084/intermittent-the-term-getdate-is-not-recognized-as-the-name-of-a-cmdlet-function-script-file?forum=winserverpowershell

This happens with V5. We ran into the same problem. We actually found it when restructuring a module used by the script the scheduled task calls. We think it’s related to task scheduler not loading profiles. If you explicitly load the core modules and out put $env they are all the guest values instead of the MSA profile values. Weirdly importing our module automagically makes autoloading work. We found it when we switched to trying to use required modules for dependencies. It break if you specify version. Just module name and it works. All in all, there is something seriously borked here.

Also, it is not intermittent for us. It happens every execution of the scheduledTask when using MSA

If you can read https://twitter.com/concentrateddon/status/799265924603752448, there’s a thread of useful replies. It may help to launch your script from a .BAT file.

I’ve only tested and experienced this with PSVersion 5.0.10586.117 on a Windows Server 2012 R2.

I did some modifications to the script:

try {
    $ErrorActionPreference_old = $ErrorActionPreference
    $ErrorActionPreference = "stop"
    "==TRY==" | Out-File "C:\temp\do-stuff.txt"
    $env:USERNAME | Out-File -Append "C:\temp\do-stuff.txt"
    $env:PSModulePath | Out-File -Append "C:\temp\do-stuff.txt"
    Get-ChildItem "env:" | Out-File -Append "C:\temp\do-stuff.txt"
    Get-Date -Format "o" | Out-File -Append "C:\temp\do-stuff.txt"

} catch {
    $msg = $_
    "==CATCH==" | Out-File -Append "C:\temp\do-stuff.txt"
    $msg | Out-File -Append c:\temp\do-stuff.txt
    $env:USERNAME | Out-File -Append "C:\temp\do-stuff.txt"
    $env:PSModulePath | Out-File -Append "C:\temp\do-stuff.txt"
    Get-ChildItem "env:" | Out-File -Append "C:\temp\do-stuff.txt"
    Get-Date -Format "o" | Out-File -Append "C:\temp\do-stuff.txt"
    $ErrorActionPreference = $ErrorActionPreference_old
}

$ErrorActionPreference = $ErrorActionPreference_old

The above script has both failed and run as expected when I’ve been running the task.

These results might be interesting?

Success:

==TRY==
gmsa-test$
C:\Users\gmsa-test$\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules\;C:\Program Files\Microsoft Monitoring Agent\Agent\PowerShell\

Name                           Value                                                                                   
----                           -----                                                                                   
ALLUSERSPROFILE                C:\ProgramData                                                                          
APPDATA                        C:\Users\gmsa-test$\AppData\Roaming                                                   
CommonProgramFiles             C:\Program Files\Common Files                                                           
CommonProgramFiles(x86)        C:\Program Files (x86)\Common Files                                                     
CommonProgramW6432             C:\Program Files\Common Files                                                           
COMPUTERNAME                   MY-SERVER01                                                                              
ComSpec                        C:\Windows\system32\cmd.exe                                                             
FP_NO_HOST_CHECK               NO                                                                                      
LOCALAPPDATA                   C:\Users\gmsa-test$\AppData\Local                                                     
NUMBER_OF_PROCESSORS           2                                                                                       
OS                             Windows_NT                                                                              
Path                           C:\Python27\;C:\Python27\Scripts;C:\Perl\site\bin;C:\Perl\bin;C:\Windows\system32;C:\...
PATHEXT                        .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.py;.pyw;.CPL                     
PROCESSOR_ARCHITECTURE         AMD64                                                                                   
PROCESSOR_IDENTIFIER           Intel64 Family 6 Model 42 Stepping 2, GenuineIntel                                      
PROCESSOR_LEVEL                6                                                                                       
PROCESSOR_REVISION             2a02                                                                                    
ProgramData                    C:\ProgramData                                                                          
ProgramFiles                   C:\Program Files                                                                        
ProgramFiles(x86)              C:\Program Files (x86)                                                                  
ProgramW6432                   C:\Program Files                                                                        
PSModulePath                   C:\Users\gmsa-test$\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPow...
PUBLIC                         C:\Users\Public                                                                         
SystemDrive                    C:                                                                                      
SystemRoot                     C:\Windows                                                                              
TEMP                           C:\Users\GMSA-T~1\AppData\Local\Temp                                                    
TMP                            C:\Users\GMSA-T~1\AppData\Local\Temp                                                    
USERDNSDOMAIN                  EXAMPLE.COM                                                                                  
USERDOMAIN                     EXAMPLE                                                                                     
USERNAME                       gmsa-test$                                                                            
USERPROFILE                    C:\Users\gmsa-test$                                                                   
windir                         C:\Windows                                                                              


2016-11-18T10:28:59.5750559+01:00

Fail:

==CATCH==
Out-File : The term 'Out-File' is not recognized as the name of a cmdlet, function, script file, or operable program. C
heck the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\temp\do-stuff.ps1:4 char:17
+     "==TRY==" | Out-File "C:\temp\do-stuff.txt"
+                 ~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Out-File:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
 
gmsa-test$
C:\Users\Default\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules\;C:\Program Files\Microsoft Monitoring Agent\Agent\PowerShell\

Name                           Value                                                                                   
----                           -----                                                                                   
ALLUSERSPROFILE                C:\ProgramData                                                                          
CommonProgramFiles             C:\Program Files\Common Files                                                           
CommonProgramFiles(x86)        C:\Program Files (x86)\Common Files                                                     
CommonProgramW6432             C:\Program Files\Common Files                                                           
COMPUTERNAME                   MY-SERVER01                                                                              
ComSpec                        C:\Windows\system32\cmd.exe                                                             
FP_NO_HOST_CHECK               NO                                                                                      
NUMBER_OF_PROCESSORS           2                                                                                       
OS                             Windows_NT                                                                              
Path                           C:\Python27\;C:\Python27\Scripts;C:\Perl\site\bin;C:\Perl\bin;C:\Windows\system32;C:\...
PATHEXT                        .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.py;.pyw;.CPL                     
PROCESSOR_ARCHITECTURE         AMD64                                                                                   
PROCESSOR_IDENTIFIER           Intel64 Family 6 Model 42 Stepping 2, GenuineIntel                                      
PROCESSOR_LEVEL                6                                                                                       
PROCESSOR_REVISION             2a02                                                                                    
ProgramData                    C:\ProgramData                                                                          
ProgramFiles                   C:\Program Files                                                                        
ProgramFiles(x86)              C:\Program Files (x86)                                                                  
ProgramW6432                   C:\Program Files                                                                        
PSModulePath                   C:\Users\Default\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShe...
PUBLIC                         C:\Users\Public                                                                         
SystemDrive                    C:                                                                                      
SystemRoot                     C:\Windows                                                                              
TEMP                           C:\Windows\TEMP                                                                         
TMP                            C:\Windows\TEMP                                                                         
USERDNSDOMAIN                  EXAMPLE.COM                                                                                  
USERDOMAIN                     EXAMPLE                                                                                     
USERNAME                       gmsa-test$                                                                            
USERPROFILE                    C:\Users\Default                                                                        
windir                         C:\Windows                                                                              


2016-11-18T10:29:12.7784602+01:00

I guess that i might use bat files as a workaround, but I’d prefer if this just worked as expected.

The fact that it sometimes works for you is the only part of that which differs from what I found in testing.

When the environment isn’t set right, all sorts of PowerShell things fail, but most importantly:

  • Add-Type doesn't work if $Env:Temp is set to C:\Windows\Temp
  • Module Autoloading doesn't work, so you have to explicitly import modules -- even the "default" Microsoft ones.

A few other oddities vanished when we started setting the Temp folder and importing modules explicitly, but I can’t begin to list all the weird ways things break :wink:

Edit: Got a little too excited and didn’t read all the other answers. This solution is already posted above.

I’m seeing the exact same issue. I am switching my scheduled tasks from traditional service accounts (users) to gMSA and the scripts fail with the same errors.

What I found is that it seems that the standard modules doesn’t always get loaded in time. My workaround is to put the following at the top of my scripts:

Import-Module Microsoft.PowerShell.Utility
Import-Module Microsoft.PowerShell.Management

You may have to load other modules as well. Microsoft.PowerShell.Utility covers the Out-* cmdlets and also Get-Date and others.