Archiving log files

[archives]

by Bueno at 2013-04-09 04:46:21

Hi There!

I need some assistance is composing a script to archive multiple files in multiple directories. These files are to be moved across the network into more or less an identical directory structure.

Let me show you the directory structure where I would like to move files from;
Directory: \server01\D$\applications\myapp\webservice


Mode LastWriteTime Length Name
---- ------------- ------ ----
d---- 09/04/2013 11:55 OrderExport
d---- 09/04/2013 12:12 OrdersImport
d---- 09/04/2013 12:22 StockExport
d---- 09/04/2013 12:22 StockStatusExport
d---- 09/04/2013 11:46 Test
-a— 24/02/2012 17:59 4666105 dbgout.log
-a— 16/11/2011 10:21 4119552 KCWarehouse 2012-01-16.exe
-a— 10/01/2012 15:39 1213952 KCWarehouse 2012-02-09.exe
-a— 09/02/2012 09:57 3954176 KCWarehouse 2012-02-24.exe
-a— 18/11/2011 09:28 1457664 KhaosMD 2012-05-25.exe
-a— 09/04/2013 12:22 5373590 KhaosMD [2013-04 (week 15)].txt
-a— 06/02/2012 10:14 239 KhaosMD.conf
-a— 09/05/2012 09:05 2301952 KhaosMD.exe
-a— 09/04/2013 11:52 1321 SOAP_ErrRequest.xml
-a— 09/04/2013 11:52 470 SOAP_ErrResponse.xml
-a— 09/04/2013 12:22 775 SOAP_Request.xml
-a— 09/04/2013 12:22 90249 SOAP_Response.xml
-a— 16/01/2013 22:53 643 web.config
-a— 08/04/2013 23:28 623344 WMS [2013-04-08].txt
-a— 09/04/2013 12:14 10536963 WMS [2013-04-09].txt
-a— 18/01/2013 17:07 296 WMS.conf
-a— 13/12/2012 09:42 9588736 WMS.exe
-a— 28/03/2012 16:15 4068864 WMS_20121213.exe

And the directory structure of where I would like to move the files to;
Directory: \server02\webservicelogs$\live


Mode LastWriteTime Length Name
---- ------------- ------ ----
d---- 04/04/2013 11:12 KhaosMD
d-r-- 04/04/2013 11:12 OrderExport
d---- 04/04/2013 13:26 OrdersImport
d---- 04/04/2013 14:10 StockExport
d---- 04/04/2013 15:23 StockStatusExport
d---- 04/04/2013 15:24 WMS


What I need to do is move the contents of each of the following directories in the webservice directory of server01to the relevant folders on server02. I have also provided example file names;

OrderExport
OrderExport [130117085433_617].xml

OrderImport
OrdersImport [120624134015_674].xml
OrdersImport [120624134015_674].log

StockExport
StockExport [121008163342_541].xml

StockStatusExport
StockStatusExport [120501140157_924].xml

There are also logs in the webservice directory of server01 which are not in sub directories but I would liek to move them into the relevant directories on server02;

WMS [2013-01-14].txt
KhaosMD [2012-05 (week 22)].txt

I want to move files which are 1 day old only(lastwritetime).

I have managed to do this successfully but only one directory at a time, my current script is big and messy. I used the following to move the contents of each sub directory (obvious I have one instance of the below code per sub directory);
Get-ChildItem ‘\Server01\c$\Folder\WebService\OrderExport*’ -Include Ordersimport*.xml, Ordersimport*.log | Where-Object {$.lastwritetime.date -le (get-date).adddays(-1).date} | foreach-object {move-item -Destination '\tycho\webservicelogs$\OrderExport'}


Then in order to move the logs in the webservice directory I do the following;

$WMSLoc = ‘\Server01\c$\Folder\WebService*’ #contains WMS and KhaosMD logs

Get-ChildItem $WMSLoc -Include WMS*.txt, KhaosMD*.txt | Where-Object {$.lastwritetime.date -le (get-date).adddays(-1).date}
Foreach ($Log in $WMSLoc){
if ($File.name -match "KhaosMD"){
Move-Item -LiteralPath $File -Destination $KDest
}
elseif ($File.name -match "WMS"){
Move-Item -LiteralPath $File -Destination $WDest
}
}


What I am looking for is some assistance in optimising my script.

Thanks in advance.

by happysysadm at 2013-04-09 05:35:52

Hi Bueno,

in the first example, it looks like ‘odersexport’ is a recurring string. To begin with, I suggest you put this and ordersimport, StockExport and StockStatusExport into an array then pipe it to a foreach inside which you can re-use your code:

[code2=powershell]@('ordersexport','ordersimport', 'StockExport,' 'StockStatusExport') | % {
(Get-ChildItem '\betty\c$\KSW\WebService$*' -Include $.xml, $_.log | Where-Object {$.lastwritetime.date -le (get-date).adddays(-1).date}) |
foreach-object {move-item -Destination '\tycho\webservicelogs$$'}
}[/code2]
Can you see what I mean?

Tell me if this helps then we will have a look at the second part and also add a bit of error handling (for failed moves).

Carlo

by MasterOfTheHat at 2013-04-09 07:09:23

I haven’t tested this, but see if the logic helps:
$sourceDir = "\betty\c$\Software\WebService"
$destDir = "\betty\c$\Software\WebService"

# get all subdirectories in the desination path
$allDestDirs = Get-ChildItem -Directory -Path $destDir

# get all .xml and .log files in the source directory and subdirectories, then process each one individually
Get-ChildItem -file -Recurse -Path $sourceDir -Include *.xml, *.log | ForEach-Object {
# save file object into a local variable so that the $_ variable can be used in nested loop
$currentFile = $
# set flag for reporting
$fileMoved = $false
# run through each destination subdirectory
$allDestDirs | ForEach-Object {
# if the file you are processing contains the name of the subdirectory, move the file
if($currentFile.Name -match $.Name)
{
Move-Item -Path $currentFile.FullPath -Destination $_.FullName
$fileMoved = $true
break
}
}
if(!$fileMoved)
{
"File $($currentFile.FullPath) was not moved from source"
}
}

by Bueno at 2013-04-10 06:26:18

Many thanks to the both of you. I marked MasterOfTheHat’s post as the answer as it worked better for me. I already had my mind set on comparing the file name to the directory name but couldn’t figure it out, now it’s just all too easy

I had to make a few changes as it was written in v3 and I’m using v2 in my environment. I will post the finished script up shortly.

by MasterOfTheHat at 2013-04-10 06:35:24

Glad it helped! And yeah, it was in v3 and I neglected to tell you. Sorry about that. Definitely let us know what the final product looks like, and maybe it will help someone else in the future.