BITS transfer with github

PowerShell Help
1

Hi,

I’m trying to use BITS transfer to get binary files (release) from github.
However, I always get a 403 error. For instance:

PS> Start-BitsTransfer -Source https://github.com/Microsoft/MIMWAL/releases/download/v2.17.0414.0/MIMWAL-2.17.0414.0.zip C:\tmp\foo.zip
Start-BitsTransfer : HTTP status 403: The client does not have sufficient access rights to the requested server object.
At line:1 char:1
+ Start-BitsTransfer -Source https://github.com/Microsoft/MIMWAL/releases/download ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Start-BitsTransfer], Exception
    + FullyQualifiedErrorId : StartBitsTransferCOMException,Microsoft.BackgroundIntelligentTransfer.Management.NewBits
   TransferCommand

It is working fine with WebClient:

> $wc = New-Object System.Net.WebClient
> $wc.DownloadFile("https://github.com/Microsoft/MIMWAL/releases/download/v2.17.0414.0/MIMWAL-2.17.0414.0.zip","C:\tmp\foo.zip")
> Test-Path "C:\tmp\foo.zip"
True

Do you see any reason why it does not work with BITS?

2

That is weird. Unfortunately, BITS is a little opaque. It’s clearly doing something different, but I’d need to do, like, a packet capture to compare the two and see what was up.

3

See updated post below.

4

PROBLEM
Downloading files from GitHub.com fails with HTTP 403 when going through BITS (e.g., using Start-BitsTransfer), but succeeds when a simpler API is used (e.g., System.Net.WebClient).

ROOT CAUSE
Before beginning a download, BITS sends an HTTP HEAD request to the server in order to figure out the remote file’s size, timestamp, etc. This is especially important for BranchCache-based BITS transfers and is the reason why server-side HTTP HEAD support is listed as an HTTP requirement for BITS downloads.

Well, it turns out that GitHub download URLs redirect to Amazon S3 URLs. For instance:

https://github.com/Microsoft/MIMWAL/releases/download/v2.17.0414.0/MIMWAL-2.17.0414.0.zip
302 Found
– REDIRECT –>
https://github-production-release-asset-2e65be.s3.amazonaws.com/47265095/3c304c70-20ff-11e7-92c1-e2bb3d070b40?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180112%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180112T222609Z&X-Amz-Expires=300&X-Amz-Signature=5e5b88fbca6007f1f4faf09f9681a2f157ffa096299b27a70f1482da1584d64a&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3DMIMWAL-2.17.0414.0.zip&response-content-type=application%2Foctet-stream
200 OK

Unfortunately, Amazon S3 uses signed URLs, so the same signed URL can’t be used for both HEAD and GET because the request method is used to compute the signature. HEAD and GET will have different signatures (see this, this and this). Thus, BITS’ initial HEAD request is what’s causing the HTTP 403 failure.

WORKAROUND
BITS bypasses the HTTP HEAD request phase, issuing an HTTP GET request right away, if either of the following conditions is true:

  1. The BITS job is configured with the BITS_JOB_PROPERTY_DYNAMIC_CONTENT flag.
  2. BranchCache is disabled AND the BITS job contains a single file (always the case for Start-BitsTransfer).

Workaround (1) is the most appropriate, since it doesn’t affect other BITS transfers in the system. Sadly, the BITS_JOB_PROPERTY_DYNAMIC_CONTENT flag isn’t currently exposed by BITS’ PowerShell cmdlets. However, the BITSAdmin Tool does expose it (via /dynamic). Example BITSAdmin command that works fine:

bitsadmin /transfer mydownload /dynamic /download /priority FOREGROUND https://github.com/Microsoft/MIMWAL/releases/download/v2.17.0414.0/MIMWAL-2.17.0414.0.zip C:\foo.zip

For workaround (2), BranchCache can be disabled through BITS’ DisableBranchCache group policy. You’ll need to do “gpupdate” from an elevated command prompt after making any Group Policy changes, or it will take ~90 minutes for the changes to take effect.