Invoke-webrequest breaking foreach loop

PowerShell Help
1

Hi there people!

I am struggling with a “break of completing” in my foreach loop that scans websites for DNS information.

The aparrant reason for my issue, is that i call for an invoke-webrequest besides my resolve-dnsname call.

It seems, that if my invoke-webrequest cannot complete, then neither will the rest of the code, and i simply get empty outputs on Nameserver, MX records and so on.

I have tried for quite a while to figure out how to tell powershell to ignore if invoke-webrequest cannot be completed, but ive had no luck so far.

If i remove the section “$DomainFinal = (Invoke-WebRequest -Uri $DomainName********” the scan works well enough.
The reason for my invoke-webrequest, is in order to get the “final destination” of the call, to find if the websites redirects somewhere else, which is a function i would really like to keep. the Rmatch function just checks if the final destination matches the searched domain and returns false if not.

If the invoke-webrequest fails, all i get is "The remote server returned an error: (403) Forbidden. " and no other output.

Can anyone help me figure out, how to allow a false return on the invoke-webrequest, while still getting the rest of the available data?

A picture of an output:

powershell error
powershell error1409×187 27.4 KB

The second last in the scan is the one failing, and returning ? and false on resolve, which is does not if the invoke-webrequest does not take place.

I really hope someone can help me out here!

Link to the script:
https://gist.github.com/Jonatantwn/4653cabfe8320cc461fe137c29591898

Best regards
Jonatan

2

$Rmatch in line 37 is not defined.
Line 37 should say

Rmatch = $DomainName -in $Domainfinal

instead of

 Rmatch = $Rmatch

But fundamentally, you should have separate try/catch blocks for resolve-dnsname and invoke-webrequest lines 11-47 as in:

$c1++
Write-Host "Processing $DomainName"
Write-Progress -Id 0 -Activity 'Checking servers' -Status "Processing $($c1) of $(@($m).lines)" -CurrentOperation $DomainName -PercentComplete (($c1 / @($m).lines) * 100)
try {
    $NSRecords = Resolve-DnsName -Name $DomainName -Type NS -DnsOnly -EA 1 | Where-Object { $_.QueryType -eq 'NS' }
    try {
        $DomainFinal = (Invoke-WebRequest -Uri $DomainName -UseBasicParsing -EA 1).BaseResponse.ResponseUri.AbsoluteUri 
        $DomainFinal = $DomainFinal -replace 'https://' -replace 'http://' -replace 'www.' -replace '.dk/','.dk'  -replace '.com/','.com'
        # Both command successful:
        [PSCustomObject]@{
            Resolved    = $true
            Rmatch      = $DomainName -in $Domainfinal
            DomainName  = $DomainName
            Domainfinal = $DomainFinal 
            NameServer  = $NSRecords.NameHost | select -first 1
            'A-Record'  = (Resolve-DnsName $DomainName -Type A).IPAddress | select -first 1
            'MX-Record' = (Resolve-DnsName $DomainName -Type MX).NameExchange | select -first 1
            Origin      = $FileName
            Status      = 'OK'
        }
    } catch {
        # Resolve-DNSName successful, but Invoke-WebRequest failed:
        [PSCustomObject]@{
            Resolved    = $true
            Rmatch      = '?'
            DomainName  = $DomainName
            Domainfinal = '?' 
            NameServer  = $NSRecords.NameHost | select -first 1
            'A-Record'  = (Resolve-DnsName $DomainName -Type A).IPAddress | select -first 1
            'MX-Record' = (Resolve-DnsName $DomainName -Type MX).NameExchange | select -first 1
            Origin      = $FileName
            Status      = $_.Exception.Message
        }
    }
} catch {
    # Resolve-DNSName command failed:
    [PSCustomObject]@{
        Resolved    = $false
        Rmatch      = '?'
        DomainName  = $DomainName
        Domainfinal = '?' 
        NameServer  = '?'
        'A-Record'  = '?'
        'MX-Record' = '?'
        Origin      = $FileName
        Status      = $_.Exception.Message
    }
}

I would even recommend that you have a single [PSCustomObject] block and manipulate your output there as in:

$c1++
Write-Host "Processing $DomainName"
Write-Progress -Id 0 -Activity 'Checking servers' -Status "Processing $($c1) of $(@($m).lines)" -CurrentOperation $DomainName -PercentComplete (($c1 / @($m).lines) * 100)

$Error.Clear()
try {
    $NSRecords = Resolve-DnsName -Name $DomainName -Type NS -DnsOnly -EA 1 | Where-Object { $_.QueryType -eq 'NS' }
    try {
        $DomainFinal = (Invoke-WebRequest -Uri $DomainName -UseBasicParsing -EA 1).BaseResponse.ResponseUri.AbsoluteUri 
        $DomainFinal = $DomainFinal -replace 'https://' -replace 'http://' -replace 'www.' -replace '.dk/','.dk'  -replace '.com/','.com'
    } catch { }
} catch { }
[PSCustomObject]@{
    Resolved    = $(if ($NSRecords) {$true} else {$false} )
    Rmatch      = $(if ($Domainfinal) {$DomainName -in $Domainfinal} else {'?'} )
    DomainName  = $DomainName
    Domainfinal = $(if ($Domainfinal) {$Domainfinal} else {'?'} )
    NameServer  = $(if ($NSRecords) {$NSRecords.NameHost | select -first 1} else {'?'} )
    'A-Record'  = $(if ($NSRecords) {(Resolve-DnsName $DomainName -Type A).IPAddress | select -first 1} else {'?'} )
    'MX-Record' = $(if ($NSRecords) {(Resolve-DnsName $DomainName -Type MX).NameExchange | select -first 1} else {'?'} )
    Origin      = $FileName
    Status      = $(if ($Error) {$Error[0].Exception.Message} else {'OK'} )
}
3

Try actually handling the error rather than trying to ignore it with Continue or SilentlyContinue. Even if it did “Continue”, you are then trying to get properties and perform replaces on a NULL value which should error. Try something more like this:

try {
    $response = Invoke-WebRequest -Uri $DomainName -UseBasicParsing -ErrorAction Stop
    $DomainFinal = $response.BaseResponse.ResponseUri.AbsoluteUri -replace "https://|http://|www." -replace ".dk/", ".dk"  -replace ".com/", ".com"
}
catch {
    $DomainFinal = $null
}

Another note is rather than replacing, try parsing it as a URL to get portions:

PS C:\Users\rasim> [System.URI]'https://www.google.com/search?=powershell'



AbsolutePath   : /search
AbsoluteUri    : https://www.google.com/search?=powershell
LocalPath      : /search
Authority      : www.google.com
HostNameType   : Dns
IsDefaultPort  : True
IsFile         : False
IsLoopback     : False
PathAndQuery   : /search?=powershell
Segments       : {/, search}
IsUnc          : False
Host           : www.google.com
Port           : 443
Query          : ?=powershell
Fragment       : 
Scheme         : https
OriginalString : https://www.google.com/search?=powershell
DnsSafeHost    : www.google.com
IdnHost        : www.google.com
IsAbsoluteUri  : True
UserEscaped    : False
UserInfo       :
4

Thanks a lot Rob and Sam!

To you Sam, i know the community should guide each other to get better and improve, and some prefer to give “hints” more than full usage examples. I do however greatly appreciate your example (and restructuring)!
Makes it a lot easier for a rookie like me to understand the full picture.

Just the feedback i needed :slight_smile:

Regarding your example Rob, to use parsing instead of replacing, i cant seem to make it work quite right.

For an example, one domain i am scanning is aktivude.com, which redirects to aktivude.dk.
Can parsing be used to identify this without involving a web-request?

  • sometimes the redirect presents itself in the hosting environment, but sometimes also through HTACCESS, which seems to make a difference as resolving DNS wont catch this.
    Sometimes i see domains redirected via. CNAME, but i many cases more “server locally” triggered.

Can you elaborate please? :slight_smile: