Background jobs & Webclient.Downloadfile

PowerShell Help
1

Hey all,

I gotta warn you :wink: I am very new to PowerShell but I am not a newbie to scripting.

So, here’s my thing which I really can’t understand and hope that you can point me to the right directions:

We have a manuals site available. From time to time the guys go to places with crappy Internet access, so I thought: no biggie. I write them a short powershell to download the files to their PC.
Said, done. Works. But it is real slow (3.5 GB total, takes several hours)

Most of the time seem to be waiting time though. So I thought: No worries, I am going to parallelize the things. found the Start-Job, so said done.

But: I ve implemented a job-limit so that I wouldnt just spam jobs. There seems to be no difference between my limit being 2, 10, or whatever number. Is there a general problem that I am missing? Code appears to work and create only the limited number of jobs. But shouldn’t the total download time be reduced by using this parallel approach?

The intranet interface appears capable of high-bandwith download. Only the Webclient seems slow. I am bound to Powershell version 2.0 and cant update.. Any ideas what I could try?

My code is something like this:

$downloadcode = { param($url, $dest, $user, $pass, $domain) measure-command { $old_eap = $ErrorActionPreference $ErrorActionPreference = "Stop" $retries = 3 while ($retries -ge 0) { $wc = new-object System.Net.WebClient $wc.Credentials = new-object System.Net.NetworkCredential($user, $pass, $domain) $tcflag = $true try { $wc.DownloadFile($url, $dest) $retries-- } catch { $tcflag = $false if ($retries -lt 0) { write-host $_.Exception.Message -BackgroundColor Black -ForegroundColor Yellow } else { Start-Sleep -milliseconds 450 } } if ($tcflag -eq $true) { $retries = -1 } } $ErrorActionPreference = $old_eap } | Select-Object -ExpandProperty TotalMilliseconds }

$limit = 2

some more initializations for path, user, stuffs

for test purposes i made 2 arrays with explicit files to download

$list = “http://here.is.my.intranet.source/one.pdf”, and more
$list2 = “e:\temp\download\one.pdf”, and more

for ($i=0;$i -lt $list.Count; $i++)
{
do
{
$x = (Get-Job -State ‘Running’).Count
Start-Sleep -Milliseconds 300
} while ((Get-Job -State ‘Running’).Count -gt $limit)
$id = “job$i”
start-job -name $id -Scriptblock $downloadcode -ArgumentList @($list[$i], $list2[$i], “myuser”, “mypass”, “mydomain”)
}

While (Get-Job -State “Running”) # wait for the last jobs
{
Start-Sleep -Milliseconds 300
}

#list the results and time required
for ($i=0; $i -lt $list.count; $i++)
{
$id = “job$i”
$p = “time for $($list[$i])”
$t[$i] = Receive-Job -Name $id
#write-host “$p$x$($t[$i])”
write-host “$p`t`t`t$($t[$i])”
}
$t | Measure-Object -Average

2

Hey there,

While my post hang for several hours in Moderation :stuck_out_tongue: I am now able to add some vital information to it:

As you can see the Measure-Command in $downloadcode obviously returns the very same time for same files.

however, I found the bottleneck. And that one I still cant understand:

For some reason I cannot have more than 5 Running State jobs. Either I cannot create them fast enough to exceed this number, or something else is throttling which I don’t know about yet :wink: Is there a configuration that limits the number of jobs?

for $limit below 5 I can observe the expected linear decrease in total download time. (If I start measuring total execution time that is ^^)

Thanks in advance :smiley: