New-CimSession Suddenly Stopped Working

PowerShell Help
1

Working on a Server Info script that will first test the connection of a remote server, if successful it will run Test-WSMAN and pull the stack version and if 2012 + will use default WSMAN during New-CimSession. If it does not meet minimum then it goes to another Try where a New-CimSession is created using -SessionOption DCOM This is for the older servers still in use where PowerShell may not be loaded at all.

This has been working for the last 4 weeks and today around 11:30 CST using WSMAN on the 2012 + Servers all came to a halt.

Contacted Networking who said Nothing is going on and IF there was any change it would be after hours not during the work day.

Server Support said the same thing. Of course nothing (supposedly) has changed but regardless of where the scripts are run they will no longer connect using WSMAN. Then I started getting reports that PSSession has also stopped working causing more phone calls.

“The WinRM client cannot process the request it cannot determine the content type of the http”

Test-WSMAN -Computername Server works ok against target servers.

I CAN force DCOM on the same servers and that will connect…

I am going to start troubleshooting when I get back in the morning checking:

Target Server Open Ports& test ports from Local PC
WinRM Services Running (which when prev checked has always been ok)
test-wsman -computername server01 -authentication default etc…

The WIN Firewall is not used however a 3rd party one is and Im leaning toward it being the culprit.

If anyone has any other suggestions I will be more then happy to take each into consideration.

2

Based on the error message, it sounds like something’s modifying the content leaving the server. A proxying firewall, for example, can do that, and I’d specifically be looking at the http response headers. You’ve determined that this isn’t simply a port blockage.

3

We just discovered that my MGR who is hitting DC01 is able to run the New-CimSession against the 2012 R2 servers no problem. Myself and my co-workers are hitting DC09 - DC05 - DC03 etc. I am forcing my laptop to DC01 now. Trying PSDrive to DC01 did not work.

We are converting to 2012 R2 DCs (from 2003) and my $$ is on they did not open the Ports for Kerberos Authentication, for all of the new DCs, that the 2012 DCs will now use.

We are seeing connection issues with other applications now as well…

I will post what I find asap

4

Update -

  1. User was able to run New-CimSession using default WSMAN protocol to connect to 2012 R2 Server.

  2. User could also run Invoke-Command and New-PSSession against same server.

  3. User was added to AD Security Group (All of our team is in this Group) and now can NOT run any of above against server.

  4. Manually changed REG Setting for this registry key to 65534 and now WSMAN is working…! This obviously is not a fix but a temp workaround.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters]
“MaxFieldLength”=dword:00065534
“MaxRequestBytes”=dword:00065534

  1. Meanwhile my AD Account Token Size is “now” above 15,500

EstimatedTokenSize : 15840
GlobalGroups : 14
UniversalGroups : 1
DomainLocalGroups : 356
GroupsWithSidHistory : 7
AllGroups : 371

  1. We are going to remove user from the recently added AD Security Group and see if they can successfully use the WSMAN protocol & other methods for remoting again.

Is there anything that could have been set on the AD Security Group to prevent the members from using WSMAN or PowerShell remoting? I have never heard of anyone doing something like that but want to make sure before I lean towards it being a Token Size Issue…

This was mentioned to me as well -

> > Don’t overthink this. The new OS default has a tokensize of 48kb anyway. It would not be a bad idea to apply this to all machines under your control. That gives you room to grow group membership to about 1000, on average. This is close to the hard (SAM) limit of 1024. The main remaining problem is IIS and other applications that insist on decoding the token.

> > Kerberos itself doesn’t really understand the concept of a token size because what it transports is opaque to the protocol.

Applications, however, are different and can implement their own constraints such as buffer size. Applications ask SSPI (Kerberos) for the size of the authorization buffer of the authenticating user. If the size reported back is greater than the buffer allocated by the application, authentication fails. The size reported back is the actual size not the maximum size. Therefore, with MaxToken set to 65k and authorization data amounting to 12k; Windows will only report back 12k. MaxTokenSize simply limits the maximum value the SSPI can return to an application.

ME: ? I am thinking that the WSMAN HTTP protocol is falling back to the 12,000 size which I am over

REPLY: The problem is that an expanded token in HTTP is represented as a base64 string, which is roughly twice as much as the original size.

But to the point: assuming that your admin account is not the same as your end-user account, are you sure that your admin account should be member of so many groups? I see quite a bit tokensize problems, but the admin accounts are usually not part of it. Well, packagers maybe.

Any thoughts…? Anything that I can run locally on the 2012 R2 Servers to check any settings? Everything that I have found for WSMAN & WinRM has not shown any issues.

5

An error was encountered while processing an operation.
Error Code: 2150858999
Error String:The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid.

  • System

    • Provider

    [ Name] Microsoft-Windows-WinRM
    [ Guid] {Blah-Blah-Blah-87da-Blah}

    EventID 1840

    Version 0

    Level 2

    Task 13

    Opcode 0

    Keywords 0x2000000000000040

    • TimeCreated

    [ SystemTime] 2015-10-14T14:26:28.178833400Z

    EventRecordID 596

    • Correlation

    [ ActivityID] {GUID Numbers}

    • Execution

    [ ProcessID] 3612
    [ ThreadID] 3012
    [ ProcessorID] 0
    [ KernelTime] 0
    [ UserTime] 0

    Channel Microsoft-Windows-WinRM/Analytic

    Computer ServerName.BigDog.Com

    • Security

    [ UserID] S-1-5-21-Blah-Blah-Blah

  • EventData

    errorCode 2150858999
    errorString The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid.

More to come…!

6

Log Name: Microsoft-Windows-WinRM/Operational
Source: Microsoft-Windows-WinRM
Date: 10/14/2015 12:17:12 PM
Event ID: 142
Task Category: Response handling
Level: Error
Keywords: Client
User: BigDog\UserID
Computer: ServerName.BigDog.Com
Description:
WSMan operation CreateShell failed, error code 2150858770
Event Xml:

142
0
2
10
2
0x4000000000000002

143258


Microsoft-Windows-WinRM/Operational
 ServerName.BigDog.Com 



CreateShell
2150858770
7

What’s the authentication in play here? Two machines in the same domain, or no?

8

Also, any security software running on the target? Anti-malware, etc?

9

Same Domain. User Account has been added to a Domain Local Security Account, that Domain Local Security Account has been added to the Local Administrator Group of the WIN Servers.

I have worked with the AV Group and they insist nothing they are doing is blocking this. They have dumped several log files as well to show nothing is being triggered when we are forcing the issues.

I asked them to make sure that they are not using any Protocol Filtering and if they are to make sure - like another AV example - that Web and Email / Protocol Filtering / Excluded Applications includes PowerShell.exe & PowerShell_ISE.exe or anything to do with PowerShell…

I am leaning towards Token Bloat since my Admin Account TokenSize is hovering around 16,000

EstimatedTokenSize : 15840
GlobalGroups : 14
UniversalGroups : 1
DomainLocalGroups : 356
GroupsWithSidHistory : 7
AllGroups : 371

I used several of your WINRM Trouble shooting tips and Ed Wilson helped out with similar steps here - http://blogs.technet.com/b/heyscriptingguy/archive/2015/10/05/troubleshoot-winrm-with-powershell.aspx

Also -

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters]
“MaxFieldLength”=dword:00065534
“MaxRequestBytes”=dword:00065534

10

Also reminder This is affecting 2012 R2 Servers WSMAN only… 2008 R2 That have remoting enabled are able to connect Invoke-Command, New-PSSession and most importantly New-CImSession just fine. 2012 R2 I must use the DCOM protocol to connect, WSMAN will not connect…

$Opt = New-CimSessionOption -Protocol Dcom
New-CimSession -ComputerName 2012R2Server -SessionOption $Opt -ErrorAction Stop

11

The place I’ve seen that exact error before was an AV service preventing remote process creation, which is essentially what happens when Wsmprovhost spins up - that’s why the malware question ;).

And yeah, a ginormous token can do it. It ends up getting truncated. Easy enough to verify, though - create a dummy account that you can drop directly in the local Administrator group as a test, and see if it fixes it. And the fact that WS-MAN is affected on CIM at least means this is likely a WinRM-level thing.

So, can you sit ON one of those boxes, as an Administrator, and remote to that same box?

12

AV - AV Team said they are not blocking any remote processing. They have been very helpful even searching online themselves to ensure it was not an unforeseen issue that they are responsible for.

Created dummy account and added it to the Local Admin of a couple 2012 R2 Servers. All of them work just fine.

Removed dummy account from Local Admin, added dummy account to the AD Security Group that is used for official access. It no longer works. Same errors for CIM, Invoke-Command, New-PSSession etc…

Removed dummy account from AD Security Group, re-added it to the Local Admin account, and once again its working fine…

There is now a ticket opened to address the AD Security Group to have it cleaned up and multiple nested groups audited and removed as well as removing user accounts from the Domain Level AD Security Group and adding them to a Global Level group which will be added to the Domain Level Group.

Thank You For the assistance on this!