Move Files from Multiple Remote Linux Servers to Windows Server

shell_l_d · December 8, 2021, 1:10am

Hi

May I ask what’s the best way to do this please?

I have this so far, which just copies the files.

I actually need to either move them from remote LINUX to local WINDOWS or copy to Windows then delete from remote LINUX servers.

Thanks

Note: some of these commands are commented out.


# INITIALLY: run the line below to create a text file 'SecureString.txt' containing the secure string of the myusername password you enter in the prompt 
# Read-Host "Enter Password" -AsSecureString |  ConvertFrom-SecureString | Out-File "SecureString.txt"

# Read the encrypted password
$encrypted  = Get-Content ".\SecureString.txt" | ConvertTo-SecureString

# Create the credential/login
$credential = New-Object System.Management.Automation.PsCredential('myusername', $encrypted)

# Variables
$serverName   = "MYLINUXSERVER"
$remoteFolder = "/app/dir1/dir2/dir3/log/AccessLog"
$localFolder  = "D:\dir1\MYLINUXSERVER\dir3"
$fileName     = "access_????-??-??.??.log"

# Copy the file from (remote) Linux server to this Windows server
#$session = New-SFTPSession -ComputerName $serverName -Credential $credential
# Cant use wildcards, not suitable
#Get-SFTPFile -SFTPSession $session -RemoteFile "$($remoteFolder)/$($fileName)" -LocalPath $localFolder

# These may require an SSH public key to be set up
#Get-SCPFile -ComputerName $serverName -Credential $credential -RemoteFile "$($remoteFolder)/$($fileName)" -LocalFile $localFolder
#pscp -v "myusername@$($serverName):$($remoteFile)/$($fileName)" "$($localFile)"

# Copy the file from (remote) Linux to Windows
# Note: also copies todays files which I don't really want: access_yyyy-mm-dd.hh.log
Get-SCPFolder -ComputerName $serverName -Credential $credential -RemoteFolder $remoteFolder -LocalFolder $localFolder

# Error: Connecting to remote server failed: Logon failure. This does not work for LINUX servers apparently.
#Invoke-Command -ComputerName $serverName -Credential $credential -ScriptBlock {Remove-Item -Path "$($remoteFile)/$($fileName)" -Verbose}

shell_l_d · December 16, 2021, 3:33am

Anyone ideas please?

Any help appreciated.

matt-bloomfield · December 16, 2021, 10:38pm

Hi, welcome to the forum

I think it’s a bit tricky for people to help you based on what you’ve posted. So much of it is commented out, it’s not clear what commands you’re actually running and which one is causing the error.

You also have a mixture of PowerShell and PuTTY in there or is pscp an alias?

If you forget about copying for a minute, are you able to SSH to the server using New-SSHSession?

shell_l_d · February 23, 2023, 11:38pm

Apologies, I had forgotten about this one, didn’t see an email notification of a comment being added.

Yes, many commands in the original post are commented out, I tried each command one at a time and I wrote a comment above the command (whether it be currently commented out or not) to say what the issue/error was with that command. Apologies if this wasn’t clear. It was all PowerShell code.

I used the following code… I’ve had to anonymise it by changing servernames & paths etc.

#-------------------------------------------------
# SUMMARY: 
# * Copies the ENTIRE OHP & SMT access log directories from <blah> Application Servers & Printer Loader Server 
#   - MYSERVERNAME1, MYSERVERNAME2, MYSERVERNAME3, MYSERVERNAME4, MYSERVERNAME5
#   - so try not to add any non access log files to these directories, unless it's in a separate subdirectory.
# * Zips the hourly access logs (access_yyyy-mm-dd.hh.log) into daily zip files (access_yyyy-mm-dd.zip) EXCEPT for today's log files
# * Removes the hourly log files from this windows server
# * Removes the original hourly log files from the remote linux server EXCEPT for today's log files
# * Moves the zip files to the archive destination.
#
# REQUIREMENTS:
# * Download Posh-SSH from https://github.com/darkoperator/Posh-SSH > Latest Release
#   Copy it to a path in the PSModule Path: $env:PSModulePath.Split(';')
#   eg: C:\Users\MyWindowsUserName\Documents\WindowsPowerShell\Modules\
# * Initially connect to each of the servers using FileZilla, it will prompt asking if you want to create an SSH key, select Yes
# * On initially running the script, click Y to proceed if prompted.
# * On initially running the script, make sure 'SecureString.txt' has been created & exists, if not, run the commented out 'Read-Host' command to create it.
#-------------------------------------------------


#---------------------------------------------------------------------------------------
# FUNCTION: Get-DateTime
#---------------------------------------------------------------------------------------

# Return the current date and time in the format: yyyy-MM-dd HH:mm:ss
function Get-DateTime {
    return Get-Date -format "yyyy-MM-dd HH:mm:ss"
}


#-----------------------------------------------------------------------------
# FUNCTION: Get-AccessLogs
#-----------------------------------------------------------------------------

<#
.SYNOPSIS
Get/copy the  Access Logs from the relevant remote LINUX server to this WINDOWS server.

.EXAMPLE
PS> Get-AccessLogs -serverName "MYSERVERNAME1" -logFileType "OHP" -remoteFolder "/app/log/AccessLog"
#>
function Get-AccessLogs (
	[CmdletBinding()]
	[Parameter(mandatory=$true)]
	[string] $serverName,                  # eg: MYSERVERNAME1, MYSERVERNAME2, MYSERVERNAME3, MYSERVERNAME3, MYSERVERNAME5
	[Parameter(mandatory=$true)]
    [string] $logFileType,                 # eg: OHP, SMT
	[Parameter(mandatory=$true)]
    [string] $remoteFolder                 # eg: "/app/log/AccessLog"
)
{
    begin {}
    process{
        # Read the encrypted password
        $encrypted  = Get-Content ".\SecureString.txt" | ConvertTo-SecureString

        # Create the credential/login
        $credential = New-Object System.Management.Automation.PsCredential('mylinuxusername', $encrypted)

        # Variables
        $localFolder  = "D:\Logs\$($serverName)\$($logFileType)_AccessLogs"
        $fileName     = "access_????-??-??.??.log"

        # Copy the file from (remote) Linux to Windows
        # Note: also copies todays files which we don't really want: access_yyyy-mm-dd.hh.log
        write-output "$(Get-DateTime): Downloading $($logFileType) logs: $($serverName) $($remoteFolder)/$($fileName)"
        try {
            Get-SCPFolder -ComputerName $serverName -Credential $credential -RemoteFolder $remoteFolder -LocalFolder $localFolder
        }
        catch {
            write-warning "$(Get-DateTime): ERROR downloading logs: $($_.Exception.Message), Cmd:$($_.InvocationInfo.MyCommand), ScriptLine#:$($_.InvocationInfo.ScriptLineNumber), Char:$($_.InvocationInfo.OffsetInLine)."
        }
    }
    end {
        write-output "$(Get-DateTime): Finished $($MyInvocation.MyCommand)"	
	}

}


#-----------------------------------------------------------------------------
# FUNCTION: Archive-AccessLogs
#-----------------------------------------------------------------------------

<#
.SYNOPSIS
Create a daily zip file (access_yyyy-mm-dd.zip) for a days hourly access logs (access_yyyy-mm-dd.hh.log),
delete the related hourly access logs, move the zip file to it's archive destination.

.EXAMPLE
PS> Archive-AccessLogs -serverName "MYSERVERNAME1"  -logFileType "OHP"
#>
function Archive-AccessLogs (
	[CmdletBinding()]
	[Parameter(mandatory=$true)]
	[string] $serverName,                  # eg: MYSERVERNAME1, MYSERVERNAME2, MYSERVERNAME3, MYSERVERNAME3, MYSERVERNAME5
	[Parameter(mandatory=$true)]
    [string] $logFileType,                 # eg: OHP, SMT
	[Parameter(mandatory=$true)]
    [string] $remoteFolder                 # eg: "/app/log/AccessLog"
)
{
    begin {}
    process{
        $filter      = "access_????-??-??.??.log"
        $sourceDir   = "D:\Logs\$($serverName)\$($logFileType)_AccessLogs"
        $zipFileName = ""

        # Zip up the hourly log files into a daily zip, delete the hourly log files, move the zip files
        Get-ChildItem $sourceDir -Filter $filter |
        Foreach-Object {
            # Only run this code for the first hourly log file for each day to zip that days log files into 1 zip file
            if ($zipFileName -ne "$($_.Name.SubString(0,18))zip") {
			
			    # Set some variables
                $zipFileName = "$($_.Name.SubString(0,18))zip"
                $fileFilter  = "$($_.Name.SubString(0,18))??.log"
                $source      = "$($sourceDir)\$($fileFilter)"
                $dest        = "$($sourceDir)\$($zipFileName)"
                $year        = "$($_.Name.SubString(7,4))"
                $zipDest     = "\\orgname\applications\WinUser\AppnLog\Logs\$($year)\$($serverName)_$($logFileType)_Access_Logs_$($year)\"
                $todaysZip   = "access_$(Get-Date -Format "yyyy-MM-dd").zip"
				$isTodays    = $zipFileName -eq $todaysZip

                # For debugging only
                #write-output "$(Get-DateTime): FileName = $($_.Name), ZipFileName = $($zipFileName), FileFilter = $($fileFilter), Year = $($year)."
				
                try {
				
                    # Create Zip file: access_yyyy-mm-dd.zip - EXCEPT for files with todays date
                    if (!($isTodays)) {
                        Compress-Archive -Path "$($source)" -Update -DestinationPath "$($dest)"
	                    write-output "$(Get-DateTime): Created zip file: $($sourceDir)\$($zipFileName)"
                    }

                    # Zip file created successfully or files have todays date
                    try {

					    # Remove the relevant hourly log source files from the windows server
						write-output "$(Get-DateTime): Removing related hourly log files: $($sourceDir)\$($fileFilter)"
                        Remove-Item "$($sourceDir)\$($fileFilter)" -Force -Confirm:$false

                        # DO NOT do this for files with todays date
						if (!($isTodays)) {
						
							# Remove the original source files from the linux server
							write-output "$(Get-DateTime): Removing linux source files: $($remoteFolder)\$($fileFilter)"
							try {
								# Read the encrypted password
								$encrypted  = Get-Content ".\SecureString.txt" | ConvertTo-SecureString

								# Create the credential/login
								$credential = New-Object System.Management.Automation.PsCredential('mylinuxusername', $encrypted)

						        # Remove ssh session if exists
	                            Get-SSHSession | Remove-SSHSession

								# Create an ssh session to the remote linux server
								New-SSHSession -ComputerName $serverName -Credential $credential #-Verbose

								# Remove the original source files from the remote linux server
								$linuxCommand = "rm $($remoteFolder)/$($fileFilter)"
								Invoke-SSHCommand -SessionId 0 -Command "$($linuxCommand)"
								# optional parameters: [-EnsureConnection] [[-TimeOut] <Int32>]
								
						        # Remove ssh session if exists
	                            Get-SSHSession | Remove-SSHSession
							}
							catch {
								write-warning "$(Get-DateTime): SSH error: $($remoteFolder)\$($fileFilter): $($_.Exception.Message)."
							}
							
						} # end if
                    } # end try
                    catch {
                        write-warning "$(Get-DateTime): ERROR removing: $($sourceDir)\$($fileFilter): $($_.Exception.Message)."
                    }

                    # Create the destination folders if they don't exist
                    if(!(Test-Path $($zipDest)))
                    {
                        New-Item -Path $($zipDest) -ItemType Directory -Force | Out-Null
                    }

                    # Move the new zip file to the destination, eg: \\orgname\applications\WinUser\AppnLog\Logs\2021\MYSERVERNAME1_OHP_Access_Logs_2021\
                    try {
						# DO NOT do this for files with todays date
						if (!($isTodays)) {
							if ((Test-Path "$($sourceDir)\$($zipFileName)") -and (!(Test-Path "$($zipDest)$($zipFileName)"))) {
								write-output "$(Get-DateTime): Moving zip file to: $($zipDest)"
									Move-Item "$($sourceDir)\$($zipFileName)" "$($zipDest)" #-ErrorAction Stop
							}
							else {
								write-warning "$(Get-DateTime): ERROR moving zip file. Source zip may not exist or destination zip may already exist"
							}
						}
                    }
                    catch {
                        write-warning "$(Get-DateTime): ERROR moving: $($sourceDir)\$($zipFileName): $($_.Exception.Message)."
                    }
                }
                catch {
                    write-warning "$(Get-DateTime): ERROR creating: $($sourceDir)\$($zipFileName): $($_.Exception.Message), Cmd:$($_.InvocationInfo.MyCommand), ScriptLine#:$($_.InvocationInfo.ScriptLineNumber), Char:$($_.InvocationInfo.OffsetInLine)."
                }
            }
        }
    }
    end {
	    # Remove ssh session if exists
	    Get-SSHSession | Remove-SSHSession
        write-output "$(Get-DateTime): Finished $($MyInvocation.MyCommand)"	
	}

}


#-----------------------------------------------------------------------------
# MAIN BODY
#-----------------------------------------------------------------------------

$logFile         = ".\Zip.log"
$theRemoteFolder = "/app/folder3/folder/FolderPlatform/log/AccessLog"

Write-Output "$(Get-DateTime): STARTED"
Write-Output "$(Get-DateTime): STARTED" | Out-File $logFile

# *** RUN THIS LINE INITIALLY ONLY ***
# Create SecureString.txt which will contain the secure string of the mylinuxusername password
# Read-Host "Enter Password" -AsSecureString | ConvertFrom-SecureString | Out-File "SecureString.txt"

Get-AccessLogs     -serverName "MYSERVERNAME1" -logFileType "OHP" -remoteFolder "$($theRemoteFolder)" *>>$logFile
Archive-AccessLogs -serverName "MYSERVERNAME1" -logFileType "OHP" -remoteFolder "$($theRemoteFolder)" *>>$logFile

Get-AccessLogs     -serverName "MYSERVERNAME2" -logFileType "OHP" -remoteFolder "$($theRemoteFolder)" *>>$logFile
Archive-AccessLogs -serverName "MYSERVERNAME2" -logFileType "OHP" -remoteFolder "$($theRemoteFolder)" *>>$logFile

Get-AccessLogs     -serverName "MYSERVERNAME3" -logFileType "OHP" -remoteFolder "$($theRemoteFolder)" *>>$logFile
Archive-AccessLogs -serverName "MYSERVERNAME3" -logFileType "OHP" -remoteFolder "$($theRemoteFolder)" *>>$logFile

Get-AccessLogs     -serverName "MYSERVERNAME4" -logFileType "OHP" -remoteFolder "$($theRemoteFolder)" *>>$logFile
Archive-AccessLogs -serverName "MYSERVERNAME4" -logFileType "OHP" -remoteFolder "$($theRemoteFolder)" *>>$logFile

$theRemoteFolder = "/app/folder/FolderPlatform/log/AccessLog"

Get-AccessLogs     -serverName "MYSERVERNAME5" -logFileType "OHP" -remoteFolder "$($theRemoteFolder)" *>>$logFile
Archive-AccessLogs -serverName "MYSERVERNAME5" -logFileType "OHP" -remoteFolder "$($theRemoteFolder)" *>>$logFile

$theRemoteFolder = "/app/folder3/templates/log/access_logs"

Get-AccessLogs     -serverName "MYSERVERNAME1" -logFileType "SMT" -remoteFolder "$($theRemoteFolder)" *>>$logFile
Archive-AccessLogs -serverName "MYSERVERNAME1" -logFileType "SMT" -remoteFolder "$($theRemoteFolder)" *>>$logFile

Get-AccessLogs     -serverName "MYSERVERNAME2" -logFileType "SMT" -remoteFolder "$($theRemoteFolder)" *>>$logFile
Archive-AccessLogs -serverName "MYSERVERNAME2" -logFileType "SMT" -remoteFolder "$($theRemoteFolder)" *>>$logFile

Get-AccessLogs     -serverName "MYSERVERNAME3" -logFileType "SMT" -remoteFolder "$($theRemoteFolder)" *>>$logFile
Archive-AccessLogs -serverName "MYSERVERNAME3" -logFileType "SMT" -remoteFolder "$($theRemoteFolder)" *>>$logFile

Get-AccessLogs     -serverName "MYSERVERNAME4" -logFileType "SMT" -remoteFolder "$($theRemoteFolder)" *>>$logFile
Archive-AccessLogs -serverName "MYSERVERNAME4" -logFileType "SMT" -remoteFolder "$($theRemoteFolder)" *>>$logFile

Write-Output "$(Get-DateTime): FINISHED. Log file: $($logFile)"
Write-Output "$(Get-DateTime): FINISHED. Log file: $($logFile)" | Out-File $logFile -Append

Topic		Replies	Views
Moving directory from one remote server to another PowerShell Help	1	205	May 16, 2024
Server backup script PowerShell Help	17	234	May 16, 2024
Iterate through many servers, archive files on remote server, copy to local backup server PowerShell Help	6	767	May 16, 2024
Need guidance server to server PowerShell Help	32	1076	May 16, 2024
copy folder to multiple machines PowerShell Help	12	526	May 16, 2024

Move Files from Multiple Remote Linux Servers to Windows Server

Related topics