by Klaas at 2012-09-19 02:12:28
Remoting is enabled through GPO and yet there are some servers where it doesn’t work. All those servers are Windows Server 2008R2 SP1 machines in the same domain, in the same OU, with the same administrators. Firewall rules are also configured by GPO. I try to remote as domain admin with elevated rights. On most servers the enter-pssession works fine, but I found 3 machines that give the errorby DonJ at 2012-09-19 03:01:22
[quote]"Enter-PSSession : Connecting to remote server Myserver failed with the following error message : WinRM cannot process the request. The following error with errorcode 0x80090322 occurred while using Kerberos authentication: An unknown security error occurred."[/quote] and a list of possible causes.
In the event log I find "401 HTTP_STATUS_DENIED".
I found some troubleshooting tips in the help, Ravikanth’s ebook, Don’s ‘Secrets of Powershell remoting’, chapter 10 in ‘Powershell in Depth’ and 5 days of Googling, but still no succes.
When I try to examine listeners or configuration with winrm or WSMANcmdlets I get the same error:
[quote] "WSManFault
Message = WinRM cannot process the request. The following error occured while using Negotiate authentication: An unknown security error occurred."[/quote]
when I cd or ls to WSMAN: I get no further than WSMAN:\localhost, there are no plugin, shell,service or client ‘subfolders’, but again the same error.
WinRM is running and I restarted the service and rebooted the servers several times. Netstat shows that the machines are listening at 0.0.0.0:5985.
I checked SPN’s and they appear to be the same as for the other servers.
What am I missing?
On the surface it would appear that either you don’t have permission to the WinRM configuration (which doesn’t seem likely) or that it is corrupted (which I’ve never run into) somehow.by Klaas at 2012-09-19 05:01:46
Have you done a diagnostics trace, using the PSDiagnostics module, as described in "Secrets?" You would want to run one on both the client and the server whilst trying to connect.
I cannot get the messages, not even with construct-PSRemoteDataObject, so I’ll describe what I see in the event viewer:by DonJ at 2012-09-19 05:20:06
on the client:
session initialize and 8 API calls for setting options and Creating WSMan shell
254 Activity Transfer
161 User authentication: ‘WinRM cannot process …’ as in my first post
254 Activity Transfer
142 Response handling: ‘WSMan operation CreateShell failed, error code 2150858909’
16 WSMan API call: ‘Closing WSMan shell’
deintialize
on the server (I tried enter-pssession to it self):
session initialize and 8 API calls for setting options and Creating WSMan shell
166 User authentication: ‘The chosen authentication mechanism is Kerberos’
80 Request handling: ‘Sending the request for operation CreateShell to destination machine and port sqlprod03:5985’
166 User authentication: ‘The chosen authentication mechanism is Kerberos’
129 Response handling: ‘Received the response from Network layer; status: 401 (HTTP_STATUS_DENIED)’ 2 times
142 Response handling: ‘WSMan operation CreateShell failed, error code 2150858909’
closing shell and deinitialize
So, there’s definitely a permissions issue of some kind on the server. I’m just unsure where to start looking to correct it. I’ve seen other folks run into this - but without sitting right in front of the computer myself, it’s pretty tough to diagnose.by DonJ at 2012-09-19 05:21:30
I hate to say this, because I know what a pain it is, but this is something you really might want to take up with Microsoft Product Support. They’ve got access to more internal resources, and if you can find a resolution it’s something we could document, together, to help other folks out in the future.
I’m going to ask around a bit, and I’ll post again if I come up with anything new.
And - just so I’m clear - you’re able to log on to the server console as a Domain Admin. The server is, in fact, part of a domain. You’ve enabled remoting on the server, but are unable to remote from the server to the server (enter-pssession localhost). The only remoting setup is the default HTTP listener, and you’re not specifying any other parameters with enter-pssession. Do I have all that right?by Klaas at 2012-09-19 05:36:09
That’s all correct, the only thing I 'm not sure about is [quote="DonJ"]The only remoting setup is the default HTTP listener[/quote]by DonJ at 2012-09-19 05:50:20
As far as I know there has nothing been set up but enable-psremoting, but I’m not 100% sure. Can I check that while winRM and WSMan: are not available?
That’s all Enable-PSRemoting would have set up - but it can be a bit tricky to check if WSMan isn’t letting you in ;). It’d be useful though since we need to check the ACL on the default session config. Lets see.by Klaas at 2012-09-19 05:55:30
From Cmd.exe, not PowerShell, run
WinRM e WinRM/config/listener
You can also run
WinRM configsddl default
To check the security on the root. It should essentially be Administrators Full Control; don’t be tempted to mess with that if its already correct.
No succes.by DonJ at 2012-09-19 06:04:47
started cmd.exe as administrator,
both commands return the security error.
WSManfault WinRM cannot process…
Weird. OK. I’ll ask around - I still suggest taking this to Product Support. Short of reinstalling Windows (yuck) I’ve got no other immediate suggestions.by DonJ at 2012-09-19 06:10:38
A few other basic precautions:by Klaas at 2012-09-19 07:42:09
When opening PowerShell on the server console, the window title bar must say "Administrator." Ensure that’s the case.
Ensure the administrator password is set - a blank password isn’t allowed.
[quote]Domain Controllers Warning Event ID: 10154 | IT Core Blog
…
Since that WinRM runs under “Network Service†account, I was able to fix this warning by
granting the “Validated Write to Service Principal Name†permission to the NETWORK SERVICE [/quote]
Still assuming this server is in a domain; if not, review http://www.shirmanov.com/2011/04/winrm- … local.html
I’m not sure any of these will change anything for you, but they’re basic things to check before proceeding or contacting PSS b
It certainly wasn’t easy, but a support request is submitted. Expected reaction time is one working day.by jreinhold at 2012-09-24 23:54:39
to be continued …
Hello,by juneb_msft at 2012-09-25 09:59:49
I have the same problem.
Enventlog is showing the following entries:
WSMan operation CreateShell failed, error code 5, Event Id 142, Source Respone Handling
Received the response from Network layer; status: 401 (HTTP_STATUS_DENIED) , Event Id 129, Source Respone Handling (twice)
Did you recieve a response on your support request yet?
Regards
Jan
I’ve bubbled this up to the WSMan team. Stay tuned.by juneb_msft at 2012-09-25 13:10:24
Hi, Klaas,by DonJ at 2012-09-25 13:37:37
The WSMan team believes that the error messages suggest a name conflict in the layers below WSMan. WSMan is just surfacing this error message.
The Errorcode 0x80090322(SEC_E_WRONG_PRINCIPAL) indicates that the target principal name is not correct. This error code appear when the client tries to connect to server by providing a computer name that is different from actual computer name of the server. This happens when there is an IP address conflict, too.
The team suggests that you review the following KB article: http://support.microsoft.com/kb/263208:
CAUSE
The event 3034 error message with the specific status code of 80090322 (SEC_E_WRONG_PRINCIPAL) indicates that the client tried to connect to the server with a computer name that is different from the actual computer name of the server. You may also receive event 3034 event messages if two computers accidentally register the same IP address in the Domain Name System (DNS). In that case, the A record for the computer that hasthe incorrect IP address should be deleted in DNS.
RESOLUTION
This problem is typically the result of a host name resolution issue. If it is, on the client computer, use the ping command to verify that the IP address that the client is using for the server is correct. For information about the ping command, see Windows Help. If the IP address is not correct, you must determine where the incorrect address comes from, and then contact your network administrator. For additional information, click the following article number to view the article in the Microsoft Knowledge Base: 200525 (http://support.microsoft.com/kb/200525/EN-US/ )
Please let me know if this works for you.
Ah. Thanks, June - I didn’t connect that. Klaas, my "Secrets of PowerShell Remoting" freebook discusses how to configure WSMAN if you need to connect with a name other than the computer’s canonical Active Directory name (e.g., an IP address or DNS alias).by Klaas at 2012-09-25 23:02:02
I am not at work this week. I’ll check your tips next monday.by Klaas at 2012-10-12 08:07:36
Thank you both.
Just want to let you know I received a call from Microsoft Netherlands today, three weeks later than promised, and basically to answer the same questions about firewall and .Net framework versions and is remoting enabled and so on. The only new thing we discovered was that IP v6 was disabled, but after enabling the problem was exactly the same. He also made me try using Server Manager to connect to the server and that failed to, which led him to the conclusion that it was a security problem outside WinRM. Then the Microsoft employee had to attend a meeting and promised to call back later today but I haven’t heard him anymore. Hopefully more news next week.by DonJ at 2012-10-12 08:21:25
@June: I get the error about the wrong computername often, also from Kerberos, but I’ve checked this multiple times and nowhere I find those wrong or duplicate DNS records, IP addresses or SPNs.
He’s right about Server Manager given that this is pre-Win2012, and SM uses other protocols for the communications. So you’ve got a deeper auth problem.by Klaas at 2012-11-02 08:21:26
Still working on this.by Klaas at 2012-11-05 05:22:07
The last three weeks I’ve had a lot of calls from Microsoft and did a lot of tests, SPD, Network Monitor and so on. Lots of those .etl files were corrupt. After 4 tries they discovered they had given me wrong instructions: I had to end a scheduled task that ran a trace, but by ending this manually the frame wasn’t closed properly and the file was unreadable. The task was then set to end after 5 minutes and that resulted in a usefull file.
Now they analysed the files and come again to the conclusion that there’s something wrong with SPNs, kerberos and replication. I just ran DCDIAG, NLTEST, IPCONFIG, NSLOOKUP, REPADMIN with different parameters as instructed and uploaded everything to the Microsoft workspace. Thanks to remoting I could Invoke-command to all our DC’s at once and collect all information in txt files in a few minutes. I didn’t see anything wrong.
More news next week.
I just noticed I forgot to post some information we gathered along the way:by Klaas at 2012-12-27 00:55:45
- ‘Enter-pssession’ on IP works fine, but in that session I still get access denied on ‘ls wsman:\localhost\listener’ and ‘get-pssessionconfiguration’ a.o.
- Telnet to port 5985 works, also on computername. I get a blank page and after Ctrl+C I see the "http 1.1 400 bad request" page.
- After eliminating some disabled servers and a Win2008 with no Powershell installed, we now still have this issue on 3 servers, all with SQL2008R2 and multiple NICs. Doesn’t that sound suspicious? Although, we do have 14 more SQL servers that don’t have the issue.
Hallelujah
After 3,5 months of Microsoft assistance this issue is solved. We were contacted by several specialists who searched and examined untill they needed a vacation and passed the hot potatoe to the next team. About 4 times we were asked to perform the same tests (winrm qc, enable-psremoting, net trace,…) and everyone was sure they saw something and then they were sure it was a firewall issue, and then it had to be a kerberos fault, maybe double DNS records, something with SPNs, and back to start over and over again.
This week another attempt and the ‘new guys’ looked at a trace we made 6 weeks ago and mailed the solution 15 min later!
It seems that a SPN on http/servername was registered by the SQL Service account on 3 servers. This was confirmed bySetSPN -F -Q http/servernameOn the servers where remoting worked, the answer was "No such SPN found." On the problem servers this command returned a list of SPNs, 2 for each server, once servername and once servername.domain.be.
When I deleted those SPNs withSetSPN -D http/servername serviceaccountnameall issues were gone! Remoting works. Special thanks to Carlos Carrolo and Luc Talpe from Microsoft!
What remains is to find out why and how those SPNs were registered on those servers and not on the others, if SQL Server still works as it used to (there are no obvious signs yet), and if this problem could reoccur. Has anyone any idea?