Trying to figure out remote sessions issue but couldn’t find the answer so far, will appreciate any help.
I have a script used to perform user related actions in AD (new-aduser, get-aduser, remove-aduser). The script can be executed against domain controllers in 5 different forests and runs from a server in a different domain.
I’m using remote PowerShell sessions to execute the code on the remote DCs which works but is very slow. So, the script will create the session once starting then reuse the same sessions for all required actions however the session exits once the script is done.
After some testing I found that the new-psession was taking long time (up to a minute in some cases) then did some reading and understood this usually takes time (altough 1 minute might be too long). To speed things up, I changed the methodology and now having session to all servers kept up all the time and instead of doing new-psession I’m using connet-psession and disconnect-psession. I can see some improvement, times for connect-psession are relatively faster (20-30 secs) however this is still slow.
Once the session is connected, commands are executed fast enough so it seems like the initial connection is the issue and I’m trying to figure out if there is a way to farther improve this.
I was trying to keep the session connected i.e. not use disconnect-psession however when I do this, I get the following error when I try to reuse the session: “Invoke-Command : Specified RemoteRunspaceInfo objects have duplicates”.
To sum up I’m trying to figure out how to execute remote commands over sessions in a faster way. Seems like leaving the session active and using the connect/disconnect commands is still slow and when trying to tap into existing session without disconnecting, it will work on the first execution however for following ones I’ll receive the above error.
Can you try the same with a test script instead of the original script. Just to make sure whether the problem is general or something with the script running in a remote session.
Thanks for replying,
To eliminate other parts of the scripts I tested only the connect-psession command using measure-command. The time it takes to complete can vary so sometimes it’ll be 30 seconds and sometimes 8 seconds, but I did notice that when I disconnect from a session and reconnect immediately it always takes 1-2 seconds.
I’m trying to understand why when I immediately reconnect it is so fast and when I wait longer after disconnecting it’ll take longer to reconnect? Since I don’t create a new session but just reconnecting.
Something must be happening that causes this time change, and this is exactly what I’m trying to understand. If this is something that can be controlled so every reconnect will be that fast, that will significantly shorten the execution times.
How spread out are these servers, geographically? What kind of networks serve them, and what hops are along the routes? What are the normal ping return times to the various servers, and is there a difference in times before and after establishing the connection? Do all five remote DCs show the same behavior?
Is there a Kerberos server or other authentication system in place for validating your credentials when you connect remotely? If so, when you establish the connection you may be waiting for the authentication process to complete, and then once it’s complete you’re able to work at normal speeds. The Kerberos authentication process is a little involved and has several back-and-forth steps. It would also explain this:
If you reconnect quickly enough, your authentication is still valid (hasn’t timed out).
The servers are in different regions globally but to simplify the troubleshooting I’m focusing on a server which is in the same location and on the same LAN and subnet.
The server from which I’m running the script is not a member in the same domain as the servers that I’m connecting to and has no trust relationship with any of them so it isn’t Kerberos, the credentials for the connection are stored in an encrypted file and used when creating the session.
The connection is to the domain controller itself, so I don’t think any other server is involved in the authentication process.
Your suggestion about the authentication sound like a direction I should farther investigate, maybe there is a way to keep the authentication valid for longer time so I can reuse it. I don’t have the required knowledge for this, so I’ll try to find some info.
So, you have an admin server from which you are running scripts against domain controller servers which are in separate domains. In order to access those domains you have to authenticate, which you do by supplying an encrypted credential file from the admin server which is validated against the domain controller’s authentication database. This remote access is done (presumably) over public internet (not through a VPN tunnel).
It probably is Kerberos because:
When you supply your credential file, your admin server is the Kerberos client, and the remote domain controller is the Key Distribution Center for the remote domain and in this case it is also the Kerberos service to which access is being requested. Once your credential is verified, the KDC process on the remote domain controller grants your (remote domain) user account access to the requested service, which is the domain controller itself. Here is a more in-depth description of how Kerberos works on Active Directory.
There are (unfortunately) many possible reasons for this process to be slow:
- The ticket granting process requires several back-and-forth communications between the client (your admin server) and the KDC (the remote domain controller), which will be impacted by network traffic. (this might be unsolvable, but you should at least run
tracert between your admin server and the remote domain controller to see what it looks like)
- If this system serves a large work environment, the KDC service may be overloaded with authentication requests (especially during normal working hours for the remote site). (solvable with better server hardware or more servers & load balancing)
- The domain controller itself may be overloaded. (better hardware/more servers/load balancing)
- Depending on how the remote site is set up, if they have local redundancy for the domain controller then sync issues between redundant DCs may be causing problems. (remote site configuration/networking issues, possibly unreliable/failing hardware or VM instance)
- The domain controller probably uses a RAID array, possibly in a NAS and not actually in the DC server itself. The array may be serving too many requests, or it could be suffering drive failure resulting in slow response times (highly dependent on RAID level and other configuration details). (check the array for failing drives and remaining free space &etc)
For security reasons, this is a bad idea. Your authentication session should expire in a relatively short time, and you should definitely not have any long-term sessions. If you were to set such a thing up between your various sites, and your account were to be compromised at any one of the sites, you will have left a door open that exposes all of the sites. PowerShell is a popular attack vector. If you leave that door open you could bring down your entire company.
Look at solving the speed problem. If it can’t be done for infrastructure reasons, then you’ll probably have to live with it.
Grokikit, thanks again for taking time to reply.
As mentioned, the admin server is not a member in the domain of the target server and has no trust relationship with them thus the authentication is not Kerberos. For Kerberos to be used both servers must be members of the same domain or has some sort of trust relationship. I’m not sure about the authentication method used here, probably NTLM.
Before getting here I did check the possibility of network related slowness and other things that might affect the speed like DNS resolution however couldn’t find any problems there. Also remember that once the session is connected things are working very fast, if it was network related the entire session should be affected.
It seems to be something in the authentication area though because I can’t think of anything else that happens at this point. I know that when creating new sessions it’ll take some time to load the environment for the sessions to run but once this was done and I’m using connect/disconnect the only thing left is the authentication (unless there is something else but I can’t find clear info on this). I’ll keep looking into the authentication maybe I can find something there.