Greetings! I have an issue in my program that is causing Move-Item to take too long, a user could manually move files faster in some cases! Here is the code:
This code allows me to search a filter across the entire file path which includes folders and file name. I’ve truncated the code which would normally have additional languages, more selections like colors and shapes, and filters to handle multiple combinations of all the selections. This flexibility is needed because the criteria could be in a combination of folder or file names. I believe the problem is that the Move-Item is working 1 file at a time and when dealing with 50,000 files it takes much longer than it should. I am ok with going outside of powershell if needed, I have looked into robocopy but it does not seem to handle my filter criteria.
Also this code moves all the files in the filter to a single destination directory no matter what folder they were found in. The single directory should include no subfolders.
It may be an issue with the way you search and where you move the files and folders to. Since it is a subdirectory of your search root I’d suspect that your search will even go down the newly created subfolder.
I’d suggest using another folder as your destination and I’d try to use robocopy.
I didn’t get how your filter actually should look like. Could you share an example?
So you don’t want to recreate the copied/moved folder structure? Do you know for sure that there will be no duplicate filenames?
Olaf, thank you for looking at this. The newly created directory would contain the _DT filter (I missed this in posting the original example!).
$folderName = "$($selectedLanguage.Name)_DT"
100% sure there are no duplicates, but if there are I will handle that.
For the filter, in this example if you selected both languages it would run through the loop and the filter the first time is “*English*” and the second time is “*Spanish*”. Like I said in the real code there is additionally other selections so you have language, shape, color and there would be filters for many combinations of that. The output folder would be called language_shape_color ex: English_Round_Green. I can build this out further in code example if you still do not understand.
Since it is a subfolder of your root Get-ChildItem will dive into it anyway if you use the -Recurse switch.
If it has to be a subfolder in the end I’d consider creating a temporary folder next to the root folder and move it afterwards. Moving a whole folderstructure on the same drive is usually blasing fast - if it’s a local drive.
I suspected something like this. Basically that means that you enumerate ALL files and folders in your folder structure for each individual filter one time. If you use 2 filter the running time doubles. If you use 3 filters the running time tripples. That’s a hell of an inefficient approach.
How about creating a complete list of all files and folders once and use Select-String with a sophisticated pattern to filter for the desired files? I’d expect this to be way faster than your approach.
I have moved my output up a folder and that helped a bit. I also then moved the Get-ChildItem out of the loop so it only grabs a complete list once, which did not help as much as I would have thought although I was still using where-object and not select-string. I have not had success with Select-String yet but will keep trying. However, I think the issue would still exist when I get select-String working. Am I not using your idea to filter the results already with where-object vs select-string? Is select-string really going to improve my method? Take an example where all of the files are already neatly together in a folder so basically you are just moving all the files from one location to another, but it takes so long cause it is moving 1 file at a time. But this is just 1 scenario and so I don’t want to code around it, but rather improve all scenarios if possible.
The level of improvement depends pretty much on the amount of loop iterations you would have to do. As I wrote earlier. When you have to run your Get-ChildItem query more than once you should get a noticeable improvement when you only have to run it only once.
I don’t know because I didn’t get the exact requirement yet.
Of course you can use other approaches if they fit your requirements better. For example instead of collecting all files beforehand and using Select-String you could use a Foreach-Object and put your filter logic in the script block including the move operation.
About how many files and folders are we talking? Are these files on a local drive? How do the file names actually look like you’re looking for? Do they all have the same extension?
I am working with 50,000 files in this case and they are on the network. The amount of folders can vary, which is part of why I am having to use this approach. The idea of the script is that it looks for a naming convention that can be spread across 1 or many folders, or even just in file names. When there are a lot of folders or many different criteria the program is great because the program can segment the groups quicker than a user would manually. However, in the case where all the files are already in a single folder and only have 1 group (ie: all files are classified as English - Round - Blue) then the amount of time that it takes to move all those files from that folder to a new folder with the special name takes too long. Using windows explorer a user can drag and drop that folder with 50,000 files into the new folder and it takes a few seconds, but the script is taking 15 minutes. Technically I could try to build some logic to detect if the files are already grouped together nicely and just rename that folder but it is not just a simple rule because of all possible variances.
Is this network share hosted on a Windows server? If “yes” you should run the code locally on this server - either directly or by PowerShell remoting. That’ll speed up your script significantly.
Besides that I don’t know what to recommend further since I didn’t get the requirement completely yet.
If you need further assistance you may share a small sample file & folder structure and the expected end result.
