The 'For' counter is creating unnecessary duplicates

PowerShell Help
1

Approximately 5k files to procees, ultimately I needing the counter ($i) available to all within the ‘for { …}’. This didn’t seem like a big deal, but leave it to me :-/ …

The issue: While there are valid duplicate files,
the ‘for’ is causing the script to create
duplicate output of entire path’s.

Example SubDir’s: 2019 to present (20??)
Example file’s being processed:

84_3_IPC10_NAPA_20220429050913.png
100_3_IPC10_NVR16-8580_20220507193710.mp4
127_0_IPC4_ROOF_20240619200942.png
27_0_IPC2_ROOF_20240619200942.jpg
27_0_IPC11_ROOF_20240619200942.png

The key difference’s with these filename’s will be:
^(\d{1,3}_ # The beginning of the string,
\d{1,2}_ # mostly followed by a single digit with a subsequent underscore,
\w{3})($i) # the letters IPC followed by,
one or two digits, contained in $i,
i.e. IPC1 -thru- IPC14;
all of them will change as the iteration continues.

# The script ...
Set-ExecutionPolicy -ExecutionPolicy remotesigned -Scope currentuser

for ($i = 1; $i -lt 14; $i++) {
    # Only these SubDir's
	Get-ChildItem -directory -Include 20?? -Recurse | foreach {
		
	<# Get file's matching the regex. 
	   Where $i is one of IPC1 -THRU- IPC 
	   for each matching regex.
	#>
  Get-ChildItem -Recurse -File } | foreach { $newname = $_
  $_regex = "^(\d{1,3}_\d{1,2}_\w{3})($i)_.+\.(jpg|mp4|png)"
  if ($newname -match $_regex) {
	
	$newname.fullname
}}}

Please note the difference between the above ps1
and the version run from the cli (below):

PS> Get-ChildItem -directory -Include 20?? -Recurse | foreach { Get-ChildItem -Recurse -File } | foreach { 
		$newname = $_
		$_regex = "^(\d{1,3}_\d{1,2}_\w{3})(\d{1,2})_.+\.(jpg|mp4|png)"
			if ($newname -match $_regex) {
				$newname.fullname
				}
}

Both version’s of the code create duplicate’s
of both filenames and full\file\path\to\filename;
depending on whether $newname.fullname or $newname.name is used.

A final note, when comparing the output of the two code version’s,
respective outputs, neither the total file count is the same, nor are all the item’s found in both of the resulting file’s a match; some are missing.

Thank you for taking the time to review; continued Respect.

2

Your code is quite confusing. You may explain in simple terms what you actually want to do.

That’s bullshit. You cannot set the execution policy for scripts from inside a script. That’s like the chicken and egg problem!! :point_up: :wink:

Are you sure you meant -Recurse?

Are you sure you meant -Recurse again? Usually that’s an issue reading the same folder structure more than once.
And in fact you read it twice and that 14 times … so it’s actually 28 times. you read the same folders and files again and again and again and again … that’s very inefficient.

I’d recommend to read it once at the beginning of your script save it to a variable and work with that variable. That should even speed up you script significantly.

What do you mean “… will change”? Do you rename the files? You don’t show this in the code you shared. What exactly do you ant to do? :thinking: :man_shrugging:

And BTW: Wheny ou post console output or sample data you should format it as code as well, please.

Thanks in advance.

3

It has been working for\on this machine. If I remove it I’m broken, that’s not the concern (yet)

I thought I was sure until you asked. My understanding is the above snip would -Recurse only those SubDir’s for the files output in the following cmdlet.

And there is the rub, I lost site that the entire if {…} would iterate the entire pipe, I was so focused on the regex pattern properly matching with the use of ($i); as a group. What I just discovered by removing Get-ChildItem -directory -Include 20?? -Recurse |, the duplicate’s disappeared. Initially I thought by using the -include 2022 the ps1 would run faster or more efficient that searching through other SubDir’s in the same parent that do not have the file’s I need to process. Also there may be a chance that the regex pattern may match in those other unrelated SubDir’s.

So yeah, I definitely shot myself in the foot and tried to drag this forum down the rabbithole with me [Apologies]. I still have to revisit how to exclude other unrelated SubDir’s, which is not currently a priority.

I did show it in the example’s, my presentation may have been lacking clarity. To clarify “will change …” the example{s}:

84_3_IPC10_NAPA_20220429050913.png
100_3_IPC10_NVR16-8580_20220507193710.mp4
127_0_IPC4_ROOF_20240619200942.png
27_0_IPC2_ROOF_20240619200942.png
27_0_IPC11_ROOF_20240619200942.png

Compared to the example filename’s (above), the below pattern’s in the actual filename’s will change as the code matches, example: IPC1 will change during the iteration until IPC!4 is processed.

^(\d{1,3}_ # The beginning of the string,
\d{1,2}_ # mostly followed by a single digit with a subsequent underscore,
\w{3})($i) # the letters IPC

I do, when I’m not rushing, or my head isn’t to tight due to code not responding as expected.

@Olaf – Actually, Thank You! for taking the time to review my shortcoming’s.

4

You did show some file names. But what do you want to do with them? And why do you change the pattern? What’s the goal at the end? :man_shrugging: :man_shrugging:

5

In the future, please explain the end goal. It makes it so much easier to provide help.

I’m going to guess a couple of things that you don’t explain.

  • you will be doing something like copying each file with new file names (instead of the Write-Host command)
  • the $i variable will eventually be used in the generation of the new file names or folders
  • you want 13 copies of each of the files (the reason for the for loop)
  • you only want to do this for files that match the regex pattern

If those assumptions are correct, does this give you what you want? In all the 20?? folders, return all the files that match the regex in each of those folders and subfolders of those folders. Do this 13 times.

for ($i = 1; $i -lt 14; $i++) {
    Get-ChildItem -Directory -Path '20??' | Get-ChildItem -File -Recurse | ForEach-Object { 
        $newname = $_
        $regex = "^(\d{1,3}_\d{1,2}_\w{3})(\d{1,2})_.+\.(jpg|mp4|png)"
        if ($newname.Name -match $regex) {	
            Write-Host $newname.FullName
        }
    }
}

If my assumptions are not correct, please explain what you’re trying to accomplish.

1 Like
6

Hmmm … but you do query the folders and files again and again and again as well … just like Query-01 did. Thats unnecessary and inefficient as hell. :man_shrugging:

Query-01, could you please explain what your actual task is?

Thanks in advance.

7

You are so right. I added the for loop as the last step and missed that.

$FileList = Get-ChildItem -Directory -Path '20??' | Get-ChildItem -File -Recurse
for ($i = 1; $i -lt 14; $i++) {
     foreach ($File in $FileList) { 
        $newname = $File
        $regex = "^(\d{1,3}_\d{1,2}_\w{3})(\d{1,2})_.+\.(jpg|mp4|png)"
        if ($newname.Name -match $regex) {	
            Write-Host $newname.FullName
        }
    }
}
8

@Olaf @psdarwin – In this post I was focusing on why I was both getting duplicate results; nothing else. I purposely avoided posting the entire end-game (goal), because I didn’t want multiple script donations coming in from all sides, using new cmdlets or technique’s I had not yet read about, causing ‘me’ more confusion.

The short For Loop scenario I provided here, allows me to stay focused on ‘it’ and how or why I kept generating the duplicate’s. For me, this is another case of not seeing “The Forest due to The Tree’s”. That being, while I understood the For was looping, I did not understand that is was processing each file repeatedly. I thought once it found a match, it would not longer process that matched file again. I ran directly into a cider-block wall with that thought.

With the following, please don’t submit code, I would really like to get this done on my own the first time around. Perhaps improve later …

The Goal:
Ultimately rename, at a minimumn, 5k files in multiple SubDir’s using one For counter, a Get-Childitem, passed to Switch that will use a Hashtable @{}. Sample filenames are:

84_3_IPC10_NVR16-8580_20220429050913.png
100_3_IPC10_NVR16-8580_20220507193710.mp4

Filename Breakdown: Formtted by an OEM

The Underscore is a separator.
84_3_IPC10_NVR16-8580_20220429050913.png
84_3_: Some sort of indexing for the OEMs' machine. It appears to change with firmware, machine swap's and/or machine settings.
IPC10_: The [camera:channel] = IPC10, same as above can/will change.
NVR16-8580_: The particular machines' name; caan/will change.
fourteen digits.ext --take a guess.

Over the last few years all of the above has created a mini-mess when searching for specific camera’s. Depending on the machine, its settings and/or the operator’s lack of willingness to follow a naming scheme, this is only going to get worse. This script should provide a level(er) playing field; when searching for a specific camera and/or machine it was connected to.

Using the first example, 84_3_IPC10_NVR16-8580_20220429050913.png once processed will become 26_Channel 10_20220429050913.png. Using the following regex within aFor {...}: ^(\d{1,3}_)(\d{1,2}_)(\w{3})$($i)(_.+\.)(jpg|mp4|png), the counter → $i ← will retain the original camera number for reuse; ondemand. Once a match is found, pass each matched item to Switch, who will further process by employing an entry from @{}:

Switch -regex ($_) { 
  
  "<pattern" { $_ -replace "<pattern>", "<pattern>"  }

I was making good progress with both of my learning challenges and the script until, I started banging my head against the wall with the duplicate file issue. As previously stated, I thought by filtering the SubDir’s in the first cmdlet, would make for faster, more efficient running of the ps1; I guess not. I can now get back at trying to trim the length of the regex pattern’s I intend to use.

I’m sure there is a better way of handling this however, with limited time and being the only one willing to address the concern, just being able to understand the basic’s is huge. I don’t think I will ever forget how I shot myself in the foot with this For issue.

9

And that’s why I think you’re not doing yourself a favor asking others for help on how to fix a certain issue you have with a small part of your script. It’s like asking others to help you push a car forward because you don’t know that the car is supposed to be driven. People will tell you to release the hand break and you think “wow, it’s much easier to be pushed now.” That’s how you learn to do it the wrong way. :man_shrugging:

I still think your approach is way to complex for what you want to achieve.

2 Likes
10

@Olaf – Of that I have no doubt but, I’m still using PowerShell training-wheel’s. On this issue I asked for help because after two days, I could not grasp where I had screwed-up. Once a working draft is completed and successfully run, I’ll post the ps1 for further enlightenment.

Thank you ((( again ))) and again and …

1 Like