Powershell script with FIleZilla Pro CLI script embedded - how to properly iterate over local and remote paths

PowerShell Help
1

0

I have a scenario where I need to ftp thousands of files from a remote server to an intermediate - the intermediate has the ftp server installed - then from there to the destination server, Due to security I cannot ftp direct from remote to destination.

I have a Powershell script for each of the 2 steps, each script creates a FileZilla Pro CLI script per subdirectory.

My 1st script works as expected, iterating over all folders and subfolders, doing a put where newer or a different size.

The 2nd script I cannot get to work, it does not seem to work out the remote(intermediate) and local paths properly, or create folders locally if not existing(where they exist on remote).

I have included only the relevant parts of each script, as an example for the 2nd script, local root would be ‘F:\MyDir\Images’ and on remote the root is ‘\Images’

1st push script - remote to intermediate:

Add-DirectoryCommands -localPath $dirPath -remotePath $remoteDirPath -filezillaCommands ([ref]$filezillaCommands)

    # Iterate over each subfolder and file in the current directory, including nested subfolders
    Get-ChildItem -Path $dirPath -Recurse | ForEach-Object {
        if ($_.PSIsContainer) {
            # Create the directory on the remote server
            $relativePath = $_.FullName.Substring($localDirectory.Length) -replace "\\", "/"
            $remoteSubDirPath = "$remoteDirPath/$relativePath"
            Add-DirectoryCommands -localPath $_.FullName -remotePath $remoteSubDirPath -filezillaCommands ([ref]$filezillaCommands)
            $dirsCreated++
        } else {
            # Increment the files checked counter
            $filesChecked++
            $globalFilesChecked++

            # Prepare local and remote file paths
            $localFile = $_.FullName -replace "\\", "/"
            $relativePath = $_.DirectoryName.Substring($dirPath.Length) -replace "\\", "/"
            $remoteFilePath = if ($relativePath) { "$remoteDirPath/$relativePath/" + $_.Name } else { "$remoteDirPath/" + $_.Name }

            # Add file upload command
            #log-message "$localFile to $remoteFilePath"
            $filezillaCommands += "put --exists size_newer `"$localFile`" `"$remoteFilePath`""
            $filesCopied++
            $globalFilesCopied++
        }
    }

2nd get script - intermediate to destination:

Get-ChildItem -Path $rootFolder.FullName -Recurse | ForEach-Object {
        if ($_.PSIsContainer) {
            # Create the directory on the remote server
            $relativePath = $_.FullName.Substring($rootFolder.FullName.Length) -replace "\\", "/"
            $remoteDirPath = "./$relativePath"
            $scriptContent += "mkdir $remoteDirPath" + "`n"
        } else {
            # Increment the files checked counter
            $global:filesChecked++

            # Prepare the download command
            $localFile = $_.FullName -replace "\\", "/"
            $relativePath = $_.DirectoryName.Substring($rootFolder.FullName.Length) -replace "\\", "/"
            $remoteFilePath = if ($relativePath) { "/$relativePath/" + $_.Name } else { "/" + $_.Name }
           log-message "$relativePath -- $remoteFilePath to $localFile"
           $scriptContent += "get --exists size_newer `"$remoteFilePath`" `"$localFile`"" + "`n"
        }
    }
2

Hard to tell. Do you have any errors ? I am kind of suprised that Filezilla does not have a sync directories command. I guess script 1 is running on a server and script 2 on the intermediate server ?
On the second script, I do see that you prepare the Filezilla command, but I don’t see any place where you are executing it. I guess you hide it out.
Also to make sure, I would first deal with all directories with the command get-childitem -Directory and verify on the target with the command `Test-Path. And I would copy files after.
Zipping may also be an easier way to deal with directories.

3

Hi, welcome to the forum :wave:

Where are you setting the value for $rootFolder?

This is the bit that jumped out at me:

If $rootFolder is a string rather than a DirectoryInfo object, it won’t have a FullName property. When Get-ChildItem runs it will run over the current folder, which won’t be the folder referenced by $rootFolder unless you happen to be running the script from that directory.

4

@ZaMotH @matt-bloomfield Thanks for the quick replies, they’ve given me something to think about.
Just for clarity, this is the full script. I think what I actually need to do is

Start from remote and iterate over all folders off root recursively,
If the folder or its children don’t exist on local then create the folder/child folders,
then create the fielzilla script as I’m doing, per folder off root.

Variables

$localDirectory = “#”
$ftpHost = “#”
$ftpUser = “#”
$ftpPass = “#”
$filezillaPath = “C:\Program Files\FileZilla Pro CLI\fzcli.exe”
$logFile = “C:\log\filezilla_log.txt”

Function to log messages

function Log-Message {
param (
[string]$message
)
$timestamp = Get-Date -Format “yyyy-MM-dd HH:mm:ss”
“$timestamp - $message” | Out-File -FilePath $logFile -Append -Encoding UTF8
}

Initialize counters

[int]$global:filesChecked = 0

Log the start of the process

$startTime = Get-Date
Log-Message “Process started at $startTime”

Iterate over each directory in the root directory

Get-ChildItem -Path $localDirectory -Directory | ForEach-Object {
$rootFolder = $_

# Log the start of checking the current root folder
Log-Message "Starting check for root folder: $($rootFolder.FullName)"

# Generate the initial part of the FileZilla CLI script for each root folder
$scriptContent = @"

open sftp://${ftpUser}:${ftpPass}@${ftpHost}:22
lcd $localDirectory
"@

# Iterate over each subfolder and file within the current root folder, including nested subfolders
Get-ChildItem -Path $rootFolder.FullName -Recurse | ForEach-Object {
    if ($_.PSIsContainer) {
        # Create the directory on the remote server
        $relativePath = $_.FullName.Substring($rootFolder.FullName.Length) -replace "\\", "/"
        $remoteDirPath = "./$relativePath"
        $scriptContent += "mkdir $remoteDirPath" + "`n"
    } else {
        # Increment the files checked counter
        $global:filesChecked++

        # Prepare the download command
        $localFile = $_.FullName -replace "\\", "/"
        $relativePath = $_.DirectoryName.Substring($rootFolder.FullName.Length) -replace "\\", "/"
        $remoteFilePath = if ($relativePath) { "./$relativePath/" + $_.Name } else { "./" + $_.Name }
       log-message "$remoteFilePath to $localFile"
       # $scriptContent += "get --exists size_newer `"$remoteFilePath`" `"$localFile`"" + "`n"
    }
}

# Finish the script with the quit command
$scriptContent += "`nquit" + "`n"

# Create a temporary script file for FileZilla CLI for each root folder
$tempScriptPath = "C:\temp\filezilla_script_$($rootFolder.Name).txt"
$scriptContent | Out-File -FilePath $tempScriptPath -Encoding ASCII

# Run FileZilla CLI with the generated script
try {
    Start-Process -FilePath $filezillaPath -ArgumentList "--script=$tempScriptPath" -Wait
} catch {
    Log-Message "Error during FileZilla CLI execution for folder $($rootFolder.Name): $_"
}

# Clean up temporary script file
Remove-Item -Path $tempScriptPath

# Log the completion for this root folder
Log-Message "Completed download for root folder $($rootFolder.Name)"

}

Log the overall completion time and counters

$endTime = Get-Date
Log-Message “Process completed at $endTime”
Log-Message “Total files checked: $global:filesChecked”