Avoid incomplete remote file download using WinSCP Sync

PowerShell Help
1

Hello,

I’m trying to make a remote file sync process as safe as possible (WinSCP, Powershell). I want to safeguard against downloading remote files which aren’t quite written to yet.

I’ll post the script, but first some caveats:

  • Powershell as a language is the preference of my management (as is WinSCP)
  • Can’t be sure on file names or extensions so I’m using “Sync and delete” instead of “Copy and delete”
  • The flow of files into the remote dir is unpredictable

Preferences aside, please please feel free to suggest whatever might make me think outside the box to satisfy the requirement.

Code follows:

    $synchronizationResult = $session.SynchronizeDirectories(

        \[WinSCP.SynchronizationMode\]::Local, $localOrdersPath, $remoteOrdersPath, $False)

    foreach ($download in $synchronizationResult.Downloads) {

        \# Success or error?

        if ($download.Error -eq $Null)

        {

            $msg = "Downloaded Order: $($order.FileName) at $(Get-Date), removing from source"

            $transferLog += $msg

            \# Download succeeded, remove file from source

            $filename = \[WinSCP.RemotePath\]::EscapeFileMask($download.FileName)

            $removalResult = $session.RemoveFiles($filename)

            if ($removalResult.IsSuccess)

            {

               $msg = "Removing of file $($download.FileName) succeeded"

               $transferLog += $msg

            }

            else

            {

                $msg = "Removing of file $($download.FileName) failed"

                $transferLog += $msg

            }

        }

        else

        {

            $msg = "Download of $($download.FileName) failed: $($download.Error.Message)"

            $transferLog += $msg

        }

    }

As this is a code snippet, please assume all variables are assigned. Current testing of the protocol has been successful but I’m looking to jazz it up a little. (I’ll upload the full if needed)

PS

I’ve seen a solution here: Locking files while uploading / Upload to temporary file name :: WinSCP. But these solutions aren’t applied to sync and I’m not sure how I would structure the commands.

All the best

2

Hi, welcome to the forum :wave:

I would look at whether it’s possible to compare without syncing the files and what data WinSCP returns when you do the comparison.
If it returns the last modified time stamp, you could then build a list of files and sync only those that haven’t been modified in the last 30 seconds, which would suggest they’re no longer being written to.

1 Like
3

Thank you for your response.

With your advice in mind, I’m now looking to develop a way that only files with matching checksums are downloaded. Using the code from the following source: Verify checksum of a remote file against a local file over SFTP/FTP protocol :: WinSCP

This does require me to create a temporary data staging area. I wanted to avoid incorporating processes which add to memory usage, but this may be a sacrifice I need to make. I haven’t yet got a completed snipped to post. But I’m still looking at advice on how to make the slickest process.
Suggestions welcome.

All the best
Leah

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.