BITS transfer with github


I’m trying to use BITS transfer to get binary files (release) from github.
However, I always get a 403 error. For instance:

PS> Start-BitsTransfer -Source C:\tmp\
Start-BitsTransfer : HTTP status 403: The client does not have sufficient access rights to the requested server object.
At line:1 char:1
+ Start-BitsTransfer -Source ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Start-BitsTransfer], Exception
    + FullyQualifiedErrorId : StartBitsTransferCOMException,Microsoft.BackgroundIntelligentTransfer.Management.NewBits

It is working fine with WebClient:

> $wc = New-Object System.Net.WebClient
> $wc.DownloadFile("","C:\tmp\")
> Test-Path "C:\tmp\"

Do you see any reason why it does not work with BITS?

That is weird. Unfortunately, BITS is a little opaque. It’s clearly doing something different, but I’d need to do, like, a packet capture to compare the two and see what was up.

See updated post below.

Downloading files from fails with HTTP 403 when going through BITS (e.g., using Start-BitsTransfer), but succeeds when a simpler API is used (e.g., System.Net.WebClient).

Before beginning a download, BITS sends an HTTP HEAD request to the server in order to figure out the remote file’s size, timestamp, etc. This is especially important for BranchCache-based BITS transfers and is the reason why server-side HTTP HEAD support is listed as an HTTP requirement for BITS downloads.

Well, it turns out that GitHub download URLs redirect to Amazon S3 URLs. For instance:
302 Found
200 OK

Unfortunately, Amazon S3 uses signed URLs, so the same signed URL can’t be used for both HEAD and GET because the request method is used to compute the signature. HEAD and GET will have different signatures (see this, this and this). Thus, BITS’ initial HEAD request is what’s causing the HTTP 403 failure.

BITS bypasses the HTTP HEAD request phase, issuing an HTTP GET request right away, if either of the following conditions is true:

  1. The BITS job is configured with the BITS_JOB_PROPERTY_DYNAMIC_CONTENT flag.
  2. BranchCache is disabled AND the BITS job contains a single file (always the case for Start-BitsTransfer).

Workaround (1) is the most appropriate, since it doesn’t affect other BITS transfers in the system. Sadly, the BITS_JOB_PROPERTY_DYNAMIC_CONTENT flag isn’t currently exposed by BITS’ PowerShell cmdlets. However, the BITSAdmin Tool does expose it (via /dynamic). Example BITSAdmin command that works fine:

bitsadmin /transfer mydownload /dynamic /download /priority FOREGROUND C:\

For workaround (2), BranchCache can be disabled through BITS’ DisableBranchCache group policy. You’ll need to do “gpupdate” from an elevated command prompt after making any Group Policy changes, or it will take ~90 minutes for the changes to take effect.