Novice, copying files sequentially to SD card

[terrypin999]

I have no PowerShell skill but have happily been able to use occasional scripts in copy-paste-run mode. Embarrassingly I cannot recall the source of the script below, which I found in my Win10 files as a .txt file. Having first checked that the source was still correct, and with a USB stick as L:, I changed its exrension to .ps1, opened the PowerShell window,and after
PS C:\Windows\System32\WindowsPowerShell\v1.0> I entered this
C:\Users\terry\Dropbox\Scripts (Other,Sundry)\Powershell-CopyToSD.ps1
and then hit the Enter key. I hoped it would copy all files sequentially from the source to L:\ but it gave me the following error message:

At line:1 char:38
+ C:\Users\terry\Dropbox\Scripts (Other,Sundry)\Powershell-CopyToSD.ps1
+                                      ~
Missing argument in parameter list.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingArgument

At line:1 char:38
+ C:\Users\terry\Dropbox\Scripts (Other,Sundry)\Powershell-CopyToSD.ps1
+                                      ~
Missing argument in parameter list.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingArgument

That identifies the comma in the script path as the culprit, even though that folder name is OK in File Explorer. So I moved the script and ran it again.

PS C:\Windows\System32\WindowsPowerShell\v1.0> C:\PowerShell_Scripts\Powershell-CopyToSD.ps1
Copying 0002-two.mp3...
Copying 0001-one550.mp3...
Copying 0004-four.mp3...
Copying 0003-three300.mp3...
Copying 0006-six.mp3...
Copying 0005-five.mp3...
Copying 0008-eight.mp3...
.
.
Copying 0126-YesterdaysMinOutdoorWas2900.mp3...
Copying 0127-zero.mp3...
All files copied in order.

However I’m dismayed to find it didn’t do what it said, namely “All files copied in order.”

• No files have appeared in the USB drive L:\ which holds a freshly formatted micro-SD card, still empty.
• The files were NOT copied sequentially. That was the the aim, because (for obscure reasons) the SD card somehow builds its ‘index’ based on the order in which each file was physically copied to it

I’m stumped so any help on how I shpuld proceed would be appreciated please

Script

$sourceFolder = "C:\Users\terry\Dropbox\Electronics\Arduino\SKETCHES\MY SKETCHES\PROJECTS_Arduino\TalkingTemp Proj\_MP3 _files\Mon16Oct"
$destinationDrive = "L:\"

if (-not (Test-Path -Path $sourceFolder -PathType Container)) {
    Write-Host "Source folder does not exist."
    exit
}

if (-not (Test-Path -Path $destinationDrive -PathType Container)) {
    Write-Host "Destination drive does not exist."
    exit
}

$files = Get-ChildItem -Path $sourceFolder | Sort-Object CreationTime

foreach ($file in $files) {
    $destinationPath = Join-Path -Path $destinationDrive -ChildPath $file.Name
    Write-Host "Copying $($file.Name)..."
    Copy-Item -Path $file.FullName -Destination $destinationPath
}

Write-Host "All files copied in order."

[terrypin999]

Update:
Using F5 on L: revealed that it had copied the files. But here’s the wrong order in which they appear:

On inserting the card and running simple commands (in the Arduino software for this DFR Mini MP£ Player module) to list and play the MP3s, I got yet another order!
61 62 63… etc !

So that’s three diffrent orders to illustrate the issue:
1 2 3 4…
9 10 65 61 …
61 62 63 64 …

Can anyone suggest a script that will copy files in exactly the same order as they appear in File Explorer please?

Alternatively, I assume there is no way to change some property or attribute of the files before copying to get that result?

[tonyd]

I am a big fan of RoboCopy … you really dont need a script to do what you are attempting. It “should” copy them in order, but you will have to test as I am not 100% sure.

robocopy

[terrypin999]

Thanks, I’ll check it out but as the PS script failed I expect robocop will too.

Meanwhile I’ll try my macro idea, glacially slow though I expect it to be.

[tonyd]

Since I am curious, please report back on RoboCopy. Make sure to look at the options for preserving time stamps and permissions as I think you may also be running into how your explorer view is set for each location.

[Olaf]

Terrypin999,
Welcome to the forum.

This time I did it for you, B U T …

… when you post code, sample data, console output or error messages please format it as code using the preformatted text button ( </> ). Simply place your cursor on an empty line, click the button and paste your code.

Thanks in advance

How to format code in PowerShell.org 1 <---- Click

( !! Sometimes the preformatted text button hides behind the settings gear symbol. )

@terrypin999

I hoped it would copy all files sequentially from the source to L:\ but it gave me the following error message:

@terrypin999

That identifies the comma in the script path as the culprit, even though that folder name is OK in File Explorer. So I moved the script and ran it again.

If you want to use a path with spaces or other special characters you should use quotes.

about Quoting Rules - PowerShell

@terrypin999

because (for obscure reasons) the SD card somehow builds its ‘index’ based on the order in which each file was physically copied to it

Please don’t get me wrong, but that’s bullshit. An SD-Card is a data storage device without any active indexing or something like this. If you see an index it’s created by the programm showing it.
If you use the Windows Explorer you most likely set it up yourself. The screenshot you show in your second post shows that you sorted the files by their Date modified. If you use another sorting in the source folder of the files it will show them in another order.

@terrypin999

On inserting the card and running simple commands (in the Arduino software for this DFR Mini MP£ Player module) to list and play the MP3s, I got yet another order!
61 62 63… etc !

It seems like this device orders the files by name removing any leading zeros.

@terrypin999

Can anyone suggest a script that will copy files in exactly the same order as they appear in File Explorer please?

You don’t need a script or special tool. You should simply use the same order criteria on all devices to get a consistent experience.

My question would be what order would you actually expect it to be?
In your explanation you show

@terrypin999

1 2 3 4…
9 10 65 61 …
61 62 63 64 …

But in your screenshot the files have

00009-.mp3
0110-TheIndoorTempls-17...
00065-AndTheMinis1300.m...
0061-ABirthdayReminder-...

… and since you have a different amount of leading zeros you end up with an inconsistent and potentially unexpected order.

I’d recommend to name the files according to the order you want them to be in. In any case, the tool you use for copying does not matter.

[terrypin999]

Olaf,

I thought I’d successfully replied to you yesterday, from Safari on my iPad. But I don’t see see it, so I’ll try again here from my PC. It was brief, something like:

Look again. My files are all consistently padded to a 4 digit prefix.
And I’m afraid you’re mistaken about the central issue. It’s understood by experienced users of that DFR module."

I was a bit rushed at the time (probably why I might have made some slip in the sending). And I was responding with the “bullshit” accusation mostly in mind. So I’ll add a little more now. Not sure I can confidently use the forum’s quoting hierarchy so …

“If you want to use a path with spaces or other special characters you should use quotes.”

Thanks, understood.

"Please don’t get me wrong, but that’s bullshit. An SD-Card is a data storage device without any active indexing or something like this. If you see an index it’s created by the programm showing it.
If you use the Windows Explorer you most likely set it up yourself. The screenshot you show in your second post shows that you sorted the files by their Date modified. If you use another sorting in the source folder of the files it will show them in another order. "

In addition to my brief reply yesterday: you’re clearly not familiar with using the device, so I’m surprised that you contradicted my description of its behaviour so confidently. If you need more background you coulod start with
https://wiki.dfrobot.com/DFPlayer_Mini_SKU_DFR0299
at its note about 40% the way through:
'Copy your mp3 into you micro SD card
NOTE: The order you copy the mp3 into micro SD card will affect the order mp3 played , which means play(1) function will play the first mp3 copied into micro SD card.’

Maybe you didn’t read the following in my original post carfefully enough:
‘On inserting the card and running simple commands (in the Arduino software for this DFR Mini MP3 Player module) to list and play the MP3s, I got yet another order!’
Perhaps I should heve emphasised that more strongly. But I assumed it made it clear that I was not desscribing some bizarre behaviour of File Explorer!

Bottom line: GIVEN the way that this module behaves, I was looking (as I have been for a couple of years) for some way to ensure that files were physically copied to an SD in the order they were displayed on the PC in File Expolorer

[terrypin999]

Tony,

I’m curious too and hope I will eventually find time to move it up my ‘Must Study’ list.

But are you still optimistic it might work where all else has so far failed? I’m assuming that, unlike Olaf, you have understood my problem accurately.

EDIT 10 mins later. While I press on this morning trying to develop a script with Macro Experess Pro (to crudely simulate doing the job manually), is your curiosity strong enough to write me a robocopy line I can test in the Win 10 Command window please?
To copy all MP3 files from the source:
C:\Users\terry\Dropbox\Electronics\Arduino\SKETCHES\MY SKETCHES\PROJECTS_Arduino\TalkingTemp Proj_MP3 _files\Sat14Oct-24OctAdditions
to the destination, a micro SD card using a USB/SD adapter on, say the folder, L:/mp3

Terry

[Olaf]

@terrypin999

'Copy your mp3 into you micro SD card
NOTE: The order you copy the mp3 into micro SD card will affect the order mp3 played , which means play(1) function will play the first mp3 copied into micro SD card.’

So this device seems to sort the files by the creation time stamp of the files on the SD card. This creation time will be different from the one on your Win10 system.

Does PowerShell show the desired order when you output the variable $files you create on line 14 of your code?

[terrypin999]

@Olaf

So this device seems to sort the files by the creation time stamp of the files on the SD card.
This creation time will be different from the one on your Win10 system.

That sounds logical Olaf. Although I’m unclear how (or even if) a file gets its creation date changed when it’s first accessed by the software reading the SD? Is it possible instead that a new hidden file gets created, conatining some indexed list of the files in the order they are physically located on the card?
The (chinese to english) documentation doesn’t clarify this, at least not to me, with little modern programming experience.

I’m wondering how best to test this. I have a freshly re-formatted card waiting on my PC in M: waiting for me to add content., but I’ll pause in case you/anyone have suggestions.

Does PowerShell show the desired order when you output the variable $files you create on line 14 of your code?

I’m not sure exactly how to do that?

Terry

[Olaf]

@terrypin999

Although I’m unclear how (or even if) a file gets its creation date changed when it’s first accessed by the software reading the SD?

The creation date will not be changed. It is the date the file is created at this particular spot. So the moment you copy a new file from your local drive on your Windows 10 machine to the SD card, the creation date on the SD card get’s this time stamp. And even if the Windows Explorer shows only hours and minutes, there are seconds, milliseconds and microseconds as well.

@terrypin999

Is it possible instead that a new hidden file gets created, conatining some indexed list of the files in the order they are physically located on the card?

It is technically possible but I would doubt that. Since the default file system on SD cards is exFAT and exFAT does not have features like alternative data streams like NTFS you should be able to see a created index file on the card.

@terrypin999

I’m not sure exactly how to do that?

$sourceFolder = "C:\Users\terry\Dropbox\Electronics\Arduino\SKETCHES\MY SKETCHES\PROJECTS_Arduino\TalkingTemp Proj\_MP3 _files\Mon16Oct"
$files = Get-ChildItem -Path $sourceFolder | Sort-Object CreationTime
$files

I’d suggest using a small subset of files for testing until you’re sure that everything is working as desired.

@terrypin999

I’m wondering how best to test this. I have a freshly re-formatted card waiting on my PC in M: waiting for me to add content., but I’ll pause in case you/anyone have suggestions.

I’d expect that the results depend pretty much on the order you use in your PowerShell code. If you use a different order that you see in your Explorer you will surely get an undesired or unexpected order later on the SD card.
In your Explorer screenshot the files are ordered by LastWriteTime.

What order would you actually like them to be in on the SD card?

[tonyd]

@terrypin999

is your curiosity strong enough to write me a robocopy line I can test in the Win 10 Command window please

So … here is what I would “try”. Robocopy has a TON of options. These are the ones I would try.

robocopy "C:\Users\terry\Dropbox\Electronics\Arduino\SKETCHES\MY SKETCHES\PROJECTS_Arduino\TalkingTemp Proj_MP3 _files\Sat14Oct-24OctAdditions" L:\mp3 *.* /E /R:1 /W:1 /XJ /DCOPY:T /COPYALL /LOG:C:\temp\robo.log /NP

/R is retry and /W is time to wait for the next retry. Defaults are 1 million retries every 30 seconds so you can clearly see if it fails on a single file using the defaults, it will never continue.

/XJ is one I NEVER leave off. That is Exclude Juntions. Anyone who has been bitten by leaving this off only to fund a junction in the source path knows why you should never leave this off.

/E is copy everything including nested files/folders.

/DCOPY:T is preserver time stamps on folders

/COPYALL will preserver all file attributes for files such as permissions, archive bit etc.

/NP is no percentage … I always leave this off when logging the results or your log file looks major ugly.
Again, interested to see if Robo will pull through

[terrypin999]

Thanks a bunch Tony. I will try that tomorrow. I’ve been on the case for the entire day. The macro is still a WIP. I temporarily abandoned it to returned to more direct work with the SD module. I’ve been documenting my work largely for my own benefit, but include below a version for you and Olaf.

I think I’m close to a solution, working in FE, but that too will have to wait for tomorrow.

---

First I satisfied myself with the logic that the files after copying to the SD card should be in Date created order. Before exposure to whatever the sketch (an Arduino program, relying on the DFR library) might do to them. That seems to stand the best chance of their being read in the order in which they were physically copied. Although how to do it at that stage of the afternoon remained unclear to me.

But even after getting them in that apparent* order on the PC, copying to the card, and inserting that in the DFR module, the playing order still did not seem to reliably follow the order shown in FE. This was after tedious work with the timestamp editing facility of Bulk Rename Utility (BRU) to get the Date Created order
matched with Filename order - an essential requirement.

So I spent most of today trying to discover why that happens. With some limited new insights I think, but still WIP .

Here’s more detail:

1. Retaining that Date created order in FE, I copied all 137 files from source to micro SD card in USB drive L:. I used familiar FE operations (select all, copy, paste); no batch, no PS script, no robocopy yet!

2. During this I got the warning shown. I always get such a message, so I’ll cover it here, although I don’t think it’s a relevant factor.

I always get the message, and it’s always the first file being copied that triggers it. In this example you see it therefore gives an early indication that the result will be unsatisfactory, as 0137 is the last file, not the first. I responded with a Yes, and enabled the checkmark box. If instead I perform the FE copy by first selecting the LAST file, 0137 in this case, and <Shift + Home> to select all, and then drag to the empty destination, the message will show the first file, 0001, a more satisfactory state.

3. I ‘Safely removed…’ the card from L: and placed it in the DFR Mini MP3 Player module.

4. Then ran my Arduino sketch, to play all 137 files in sequence. It failed, as usual, although on this occasion with only a few errors.

5. I then did another test; again sorted an identical 137 files to Date created order and copied to card. The first sort of the card contents by
   Date created showed 0001 as the first and 0137 last. But reversing the sort by clicking the Date created column again gave this unexpected
   result. And note this is in FE - the card is not yet in the module.

6. My continued research today revealed a factor that’s probably familiar to those with more technical expertise but which was new to me.
   A significant obstacle to resolving my problem is the lack of precision in the Date created info shown in FE. It shows only to the minute. So a
   series of entries that looks neatly continuous may not be. In BRU precision is to seconds, but at milli or micro second speeds still not
   an infallible tool in this context.

[I’ve just seen that Olaf focuses in this too.]

7. For the first time today I have a 137 file card playing correctly. Tomorrow I’ll establish exactly how I did it!

• Precision seems a key factor.

Oh, pressing Send gave me “An error occurred: Sorry, new users can only put 2 lnks in a post” I’ll embed images instead before retrying the Send.

Sigh… "New users can only emebed one image! And I just deleted the links

Hopefully final attempt.

Terry

[terrypin999]

Good grief, what’s going on here? Just spent 30 mins replying in detail, handling successive messages about what new user cannot do, etc. Finally got the winning combination that allowed the Send (2 links and1 emebedded image).

But for the second time, my post has vanished. Is it recoverable. I’m assuming THIS post arrives.

[terrypin999]

Impatience got the better of me. No files were copied. I have this in the log. Back onto it in the morning.

(The problems with my post attempts earlier were apparently due to some automatic filtering called Akismet.)

@tonyd

robocopy "C:\Users\terry\Dropbox\Electronics\Arduino\SKETCHES\MY SKETCHES\PROJECTS_Arduino\TalkingTemp Proj_MP3 _files\Sat14Oct-24OctAdditions" L:\mp3 *.* /E /R:1 /W:1 /XJ /DCOPY:T /COPYALL /LOG:C:\temp\robo.log /NP

---

ROBOCOPY :: Robust File Copy for Windows

Started : Thursday, 26 October 2023 22:34:08
Source : C:\Users\terry\Dropbox\Electronics\Arduino\SKETCHES\MY SKETCHES\PROJECTS_Arduino\TalkingTemp Proj_MP3 _files\Sat14Oct-24OctAdditions
Dest = L:\mp3\

Files : *.*

Options : . /S /E /COPYALL /NP /XJ /R:1 /W:1

---

NOTE : Security may not be copied - Destination might not support persistent ACLs.

ERROR : You do not have the Manage Auditing user right.
***** You need this to copy auditing information (/COPY:U or /COPYALL).

   Simple Usage :: ROBOCOPY source destination /MIR

         source :: Source Directory (drive:\path or \\server\share\path).
    destination :: Destination Dir  (drive:\path or \\server\share\path).
           /MIR :: Mirror a complete directory tree.

For more usage information run ROBOCOPY /?

**** /MIR can DELETE files as well as copy them !

[terrypin999]

Thanks Olaf. Had problems posting earlier diue to some auto filter called Akismet.

See also my reply to Tony.

Thu 26 Oct 2023 22:43, UK

[tonyd]

@terrypin999

NOTE : Security may not be copied - Destination might not support persistent ACLs.

I suspect the SD card is formatted FAT32? Try leaving off the /COPYALL argument. Would have been nice to try that, but I suspect you need to format the SD card to NTFS which is not writable in Linux without playing some tricks.

[Olaf]

@tonyd

I suspect the SD card is formatted FAT32?

Like already mentioned above - the default is exFAT.

@tonyd

I suspect you need to format the SD card to NTFS

I wouldn’t recommend that.

[tonyd]

@Olaf

Like already mentioned above - the default is exFAT.

Sorry for missing that, thats what I get for skimming

[Olaf]

@terrypin999

I’ve been documenting my work largely for my own benefit, but include below a version for you and Olaf.

Since this is a PowerShell forum I’d like to ask you to only continue this thread if you’re using PowerShell code.

@terrypin999

During this I got the warning shown. I always get such a message, so I’ll cover it here, although I don’t th

@terrypin999

2. During this I got the warning shown. I always get such a message, so I’ll cover it here, although I don’t think it’s a relevant factor.

That’s expected since exFAT does not support alternative data streams like NTFS does.

@terrypin999

In this example you see it therefore gives an early indication that the result will be unsatisfactory, as 0137 is the last file, not the first. I responded with a Yes, and enabled the checkmark box. If instead I perform the FE copy by first selecting the LAST file, 0137 in this case, and <Shift + Home> to select all, and then drag to the empty destination, the message will show the first file, 0001, a more satisfactory state.

The Explorer does not copy files in the order it displays them. Instead it uses the order you select them. But this does not work as you might think it does.

Again - I’d recommend to reduce the number of files you use for your tests to a much lower number - I’d say maximum 10.

Again - in what order would you like the files to be copied to the SD-card? I’d assume you actually want them to be “alphabetically” ordered (by name). Since you use leading numbers would that be accordung to their numbers.

Just to show that PowerShell in fact does copy the files in the order you make it to I created a test script to demostrate it:

$StartFolder = 'C:\_Sample'

$SourceFolder = Join-Path -Path $StartFolder -ChildPath 'CopyOrder_01'
if (-not (Test-Path -Path $SourceFolder)) {
    New-Item -Path $SourceFolder -ItemType Directory | Out-Null
}
$TargetFolder = Join-Path -Path $StartFolder -ChildPath 'CopyOrder_02'
if (-not (Test-Path -Path $TargetFolder)) {
    New-Item -Path $TargetFolder -ItemType Directory | Out-Null
}

Push-Location -Path $SourceFolder

for ($i = 1; $i -le 5; $i++) {
    $NewItemSplat = @{
        Path     = '.\'
        Name     = '{0:d4}_TestFile.txt' -f $i
        ItemType = 'File'
        Force    = $true
    }
    New-Item @NewItemSplat
    Start-Sleep -Milliseconds 10
}
$FileList = 
Get-ChildItem -Path $SourceFolder

$FileList |
Sort-Object -Property CreationTime |
Select-Object Name,
@{
    Name       = 'PreciseCreationTime'
    Expression = { ($_.CreationTime).ToString('dd/MM/yy HH:mm:ss.fff') }
} |
Format-Table

$FileList |
Sort-Object -Property Name -Descending |
ForEach-Object {
    Copy-Item -Path $_.FullName -Destination $TargetFolder
}

$FileList = 
Get-ChildItem -Path $TargetFolder
        
$FileList |
Sort-Object -Property CreationTime |
Select-Object Name,
@{
    Name       = 'PreciseCreationTime'
    Expression = { ($_.CreationTime).ToString('dd/MM/yy HH:mm:ss.fff') }
} |
Format-Table

Pop-Location

This script will create 2 subfolders in a given folder and 5 new text files in the first subfolder. Since PowerShell is quiet fast I added a little delay between the creation of each file to make their difference in CreationTime more obvious. Otherwise it might be only microseconds.

Then it queries the files and shows them with more precise creation time stamps than the Windows Explorer would.
Now it sorts the files in a descending order and uses this order to copy them to the second subfolder. The subsequent query shows that they were created in the correct order.
This is the console ouptut:

Even when the files are created or copied in the same second there is in fact a difference in the creation time stamp.

And this is how it looks in a file explorer (FreeCommander):

Since my system is in German … the column “Erstellt” means “Creation Date”.

So - to accomplish what you’re looking for - copying the files in a desired order - you just have to bring them to this desired order and use a loop to copy them to their destination.

If you really want to and if it helps you could add a delay between each copy job to make it really clear that these files have been copied in order.