ForEach Loop through a directory

by spoony123 at 2012-10-30 16:22:57

Evening, i am new to Poweshell and have posted a few times recently as i am on a bit of a steep learning curve at the mo - so apologies for so many posts.

I now need to write a script that will aim to give me the Top 50 sft files in all directories.

a) Look through a top level directory and obtain a list of folders (essentially this is the list being created for the foreach loop)
b) Will then look at a particular sub directory in each folder
c) Check for .sft files and do a count of them
d) Present a grand total for each .sft file (there will be many duplicates in all the directories) with th

The code i have created thus far seems to work to some extent but only appears to process one of the folders, i cannot fathom out what i have done wrong.
[code2=powershell]

#Get Foldernames into Variable for ForEach Loop
$DFSFolders = get-childitem -path "c:\temp\dfs" | where-object {$.Psiscontainer -eq "True"} |select-object name

#Loop through folders in Directory
foreach ($DFSfolder in $DFSfolders)
{
#For Each Folder obtain objects in a specified directory, recurse then filter for .sft file type, obtain the filename, then group, sort and eventually show the file name and total incidences of it.
get-childitem -path "c:\temp\dfs$($DFSfolder.name)\Apps" -recurse -filter *.sft | select-object name | group {$
.name} | sort name | select-object name,count | export-csv c:\top50.csv

}



[/code2]
by Klaas at 2012-10-31 01:06:05
They may all be processed, but with every iteration the top50.csv is overwritten. You can use the -Append switch to overcome that.
If I understand it correctly, you want the top50 to be generated once over all subfolders? Now you group each subfolder apart.
by Makovec at 2012-10-31 04:29:41
And one additional note. You can rewrite first working line to this form:
$DFSFolders = get-childitem -path "c:\temp\dfs" | where-object {$.Psiscontainer} | select-object FullName it’s not necessary to use -eq "True" if comparing logical values. Another point is that IF (I don’t see the reason anyway, but IF) you want to compare, do it against $true and $false automatic variables.

Having FullName in $DFSFolders variable you can then build the path this way]get-childitem -path "$($DFSfolder.FullName)\Apps"[/powershell] which is (for me at least) a bit easier to read.

David
by spoony123 at 2012-10-31 14:19:19
Hi There - thanks for reply. We have re-jigged the code now as below.

However i have two questions now

1.The group function seems to get ignored so the files are not grouped by name, they are just showed in a single with a count value of 1 next to each of them, I am not entirely sure why this is

2.Where do i put the Export-CSV function in the code so that it gets appended too, incidentally when i tried to use the append switch i get messages about not "A parameter cannot be found that matches parameter name ‘append’."

Any further advice to the two above questions would be great, hopefully to get this snippet of code over the finish line!
[code2=powershell]$folders = get-childitem \PC1\dfs | where-object {$
.PSIscontainer}

foreach ($folder in $folders)

{

$appvpath = $folder.fullname + "" + ($folder.name + "_applications\packages")

get-childitem $appvpath -filter *.sft | group-object name | Sort-object count


}[/code2]
by Klaas at 2012-11-02 02:56:10
1. I think the group-object cmdlet reports that every name exists one time, which is very logical since you cannot have two files with the same name in one folder.

2. If you collect objects in one go, you need one export-csv, if you collect multiple times in a loop, you should export every iteration or add to an array that you export in the end.

Seems that -append doesn’t work in Powershell v2, Dmitry wrote a workaround:
http://dmitrysotnikov.wordpress.com/2010/01/19/export-csv-append/

Would this be a solution:
Get-ChildItem -recurse \PC1\dfs -include *.sft | group-object name | sort count -Desc | select count, name -first 50
by spoony123 at 2012-11-02 05:18:02
Hi Klaas - thanks for the reply.

I am really glad you have highlighted that append doesn’t work with Export-CSV. I had seen some articles alluding to this but was never sure if it was right. Is it fixed in V3? Perhaps an alternative would be Out-File?

Your suggestion works perfectly, however the reason i did the other way is that there are several million files in the directories, so i was trying to target the specific folder in each subfolder where the .sft files were to reduce run time in the script.

If i were to abandon grouping and count in the powershell script and just export the data in raw format (would use a pivot tabe in Excel to get results i need) - where would i place an out-file command to achieve this - presumaby in the For Each Loop?
by Klaas at 2012-11-02 06:07:21
I did a test in Powershell v3 piping Get-Process to Export-Csv with -Append and that succeeded. I’m not sure if it works in any case or there are some conditions under which it fails, I haven’t been searching that out.
Out-file is a possibility, but then you need to format the output carefully, so the file can be read as a csv. Export-csv takes objects and writes them in a record structure, while out-file writes strings. If you try Get-Process | Export-Csv c:\exportcsvtest.csv and then Get-Process | Out-File c:\outfiletest.csv you’ll see the difference.

Starting with your script, you could add out-file as follows:
$folders = get-childitem \PC1\dfs -directory
foreach ($folder in $folders)
{
$appvpath = $folder.fullname + "" + ($folder.name + "applications\packages")
get-childitem $appvpath -filter *.sft | Select name | Out-File C:\sftFiles.csv -Append
}

The -directory switch does the same as the [quote="spoony123"]where-object {$
.PSIscontainer}[/quote] but faster.
Another option would be to add all the names to an $sftFiles array and write that to a file only once after the foreach loop. That could be faster, you should measure the two options.
Working with objects as in my previous post is preferable, but with the number of files you should compare the options to find which is the most performant for you. You can simply add Export-csv at the end of the pipe:
Get-ChildItem -recurse \PC1\dfs -include *.sft | group-object name | sort count -Desc | select count, name -first 50 | export-csv C:\Top50sft.csv
by DonJ at 2012-11-02 08:17:18
Don’t forget that, even in v2, you could do ConvertTo-CSV and pipe to Out-File with -Append. Just tell the conversion to omit the type information and header row.