Trouble targeting JSON data

Hi there people!

I have some trouble looping through these domains.

What I’m trying to do is to scan for a specific value, in this test an ID, inside this JSON file, and notifying me when it is found.

The structure, is somewhat different than what i usually see, and i cant seem to figure out how to make PowerShell identify the sub-sections containing the data i need, as objects.
I feel that I’m missing a header to target?

This is the JSON structure:

{
    "2": {
        "id": "2",
        "url": "https://tester774.com/",
        "name": "apktest.dk"
    },
    "7": {
        "id": "7",
        "url": "https://tester223.com/",
        "name": "introtest.dk"
    },
    "17": {
        "id": "17",
        "url": "https://tester92.com/",
        "name": "sectest.dk"
    },
    "4667": {
        "id": "4667",
        "url": "https://tester432.com/",
        "name": "mnitest.dk"
    },
    "4668": {
        "id": "4668",
        "url": "https://tester112.com/",
        "name": "kptest.dk"
    }

This is the script i am testing with:

$jsondata = Invoke-WebRequest -Uri $jsonurl -Method GET

$jsondata | ConvertFrom-Json 

$jsondata | ForEach-Object {


    write-host "This is ID:" $_.id "- This is URL:"$_.url "- This is Name:" $_.name


    if ($_.id -like "*17*") {

        write-host "Number 17 found!"
        write-host "This is URL:"$_.url "- This is Name:" $_.name

    } 
    else {
        write-host "we did not find ID 17"
    }


}


This does list all the individual objects, in the terminal (using PS 7.2 with Visual Studio Code)
( if i remove the foreach-object loop that does not work anyways).

Looking like this:

4437 : @{id=2; url=https://tester774.com/; name=apktest.dk}
3508 : @{id=7; url=https://tester223.com/; name=introtest.dk}

So, i do have access to all these sections, i just cant figure out how to “grab” the data from them in a loop. I figured they need to be individually "ConvertFrom-Json"ed, but how do i target them?

Please help! :smiley:

Best regards
Jonatan

Hmmm … without beeing an JSON expert I’d say your JSON data is not valid.

But I found a way to make it work:

$JSONData = @'
{
    "2": {
        "id": "2",
        "url": "https://tester774.com/",
        "name": "apktest.dk"
    },
    "7": {
        "id": "7",
        "url": "https://tester223.com/",
        "name": "introtest.dk"
    },
    "17": {
        "id": "17",
        "url": "https://tester92.com/",
        "name": "sectest.dk"
    },
    "4667": {
        "id": "4667",
        "url": "https://tester432.com/",
        "name": "mnitest.dk"
    },
    "4668": {
        "id": "4668",
        "url": "https://tester112.com/",
        "name": "kptest.dk"
    }
}
'@ |
    ConvertFrom-Json

Foreach ($Number in $($JSONData.PSBase.ToString().trim('@{}').split(';').trim('=').trim())){
    [PSCustomObject]@{
        ID   = $JSONData.$Number.id
        URL  = $JSONData.$Number.url
        Name = $JSONData.$Number.name
    }
}

The output would be this:

ID   URL                    Name
--   ---                    ----
2    https://tester774.com/ apktest.dk
7    https://tester223.com/ introtest.dk
17   https://tester92.com/  sectest.dk
4667 https://tester432.com/ mnitest.dk
4668 https://tester112.com/ kptest.dk

once you have added |ConvertFrom-Json to the input as Olaf has done

I believe this will give the same output:-

($jsondata|Get-Member -MemberType NoteProperty).name|ForEach-Object{$jsondata.$_}

I’m not sure which would be better?

But I also notice that the ID is inside and outside of each item, while these are the same what if they are not:-

($jsondata|Get-Member -MemberType NoteProperty).name|ForEach-Object{
    $rec=$_
    $jsondata.$_|Select-Object *,@{n="RecordID";e={$rec}}
}

this outputs:-

id   url                    name         RecordID
--   ---                    ----         --------
17   https://tester92.com/  sectest.dk   17
2    https://tester774.com/ apktest.dk   2
4667 https://tester432.com/ mnitest.dk   4667
4668 https://tester112.com/ kptest.dk    4668
7    https://tester223.com/ introtest.dk 7

I have noticed that the order is alphbetical not numerical

1 Like

@Kira-Shadow @Olaf Thanks alot for your feedback :slight_smile:
I enden up with this solution, to essentially tell me what domains i have present in the JSON file, and then to crawl the DNS data (atleast some of it) for these.

This is the final structure now in use:


#log file variables.
$logtimestamp = Get-Date -Format "dd-MM-yyyy_HH-mm-ss"
$logfilepath = "REDACTED/($logtimestamp).csv"

#create log file, and insert header.
New-Item $logfilepath -ItemType File -ErrorAction 'SilentlyContinue'
$logmessageintro = "Domainname,NS,SPF,MX,PTR,A"
$logmessageintro >> $logfilepath

#The URL of the JSON file to be processed.
$mainwpurl = "https://REDACTED" 

#Get the JSON Data file, and save it locally.
$outputlocation = "c:/REDACTED" 
Invoke-WebRequest -Uri $mainwpurl -Method GET -OutFile $outputlocation

#Get the data from the locally stored JSON file and convert it to be readable. Then loop objects.
$JSONData = Get-Content -raw -path $outputlocation | ConvertFrom-Json

($jsondata | Get-Member -MemberType NoteProperty).name | Select-Object -first 10 | ForEach-Object {
    
    #Select the individual json data object for split
    $indivjson = $jsondata.$_

    #Get the URL from the indivisual json data object and replace http* prefixes.
    $domainname = $indivjson.url -replace "https://", "" -replace "http://", "" -replace "/", ""
    write-host "Domain from mainwp:" $domainname
    
    #Resolve the nameserver, and select the first.
    $NSRecords = Resolve-DnsName -Name $domainname -Type NS -DnsOnly -EA 1 | Where-Object { $_.QueryType -eq 'NS' }
    $nsforcatch = $NSRecords.NameHost | select-object -first 1

         

    #Begin DNS collection
    $domainmx = (Resolve-DnsName $domainname -Type MX).NameExchange | select-object -first 1
    $domainarecord = (Resolve-DnsName $domainname -Type A).IPAddress | select-object -first 1

    #PTR RUN IF A-RECORD IS FOUND
    if ($domainarecord -ne $null) {
        $domainptridentify = Resolve-dnsname -name $domainarecord -Type PTR
        $domainptr = $domainptridentify.namehost | select-object -first 1
    }

    #GET SPF RECORD
    $SPFRecordget = Resolve-DnsName $domainname -Type TXT | Where-Object Strings -Match "spf1"
    $SPFRecordget | Select-Object @{Name = "DomainName"; Expression = { $_.Name } }, @{Name = "Record"; Expression = { $_.Strings } }
    $SPFRecordget | ConvertTo-Json

    $SPFRecord = $SPFRecordget.Text
    write-host "SPF IS: "$SPFRecord

    #write treated object out to log.    
    ("$($domainname),$($nsforcatch),$($SPFRecord),$($domainmx),$($domainptr),$($domainarecord)") | Out-File -Filepath $logfilepath –Append

}


Import-Csv -path $logfilepath | Out-GridView

I’m pretty happy with the final result :slight_smile:

The purpose of the script is now met, and i can easily get all the domains registered online in my “domain database” and start a DNS scan of each of them, with a nice output CSV list in the end.

This is still a somewhat raw version that could sure use some finetuning, but this is atleast functionally sound!

Thanks alot for your inputs!

Best regards
Jonatan Nielsen

Thanks alot for your input!

I do however prefer the solution presented using foreach-object, as that allows me to parrallel process, as @Kira-Shadow suggested using get-member :slight_smile:

But i do have some issues still actually, in regards to parallel.

If i do not use parallel, the code works, but if i do try to run parallel, ALOT of stuff fails, as if the values are mostly null/empty inside the loop.

Any ideas of why this could be?

The attempt:


#log file variables.
$logtimestamp = Get-Date -Format "dd-MM-yyyy_HH-mm-ss"
$logfilepath = "REDACTED-$($logtimestamp).csv"

#create log file, and insert header.
New-Item $logfilepath -ItemType File -ErrorAction 'SilentlyContinue'
$logmessageintro = "Domainname,NSSTAMP,NS,SPF,MX,PTR,A"
$logmessageintro >> $logfilepath

#The URL of the JSON file to be processed.
$mainwpurl = "REDACTED" 

#Get the JSON Data file, and save it locally.
$outputlocation = "REDACTED" 
Invoke-WebRequest -Uri $mainwpurl -Method GET -OutFile $outputlocation

#Get the data from the locally stored JSON file and convert it to be readable. Then loop objects.
$JSONData = Get-Content -raw -path $outputlocation | ConvertFrom-Json

($jsondata | Get-Member -MemberType NoteProperty).name | Select-Object -first 1 | ForEach-Object -Parallel {
    
    #Set default values: 
    $domainmx = "N/A"
    $domainarecord = "N/A"
    $SPFRecord = "N/A"

    #Select the individual json data object for split
    $indivjson = $jsondata.$_

    #Get the URL from the indivisual json data object and replace http* prefixes.
    $domainname = $indivjson.url -replace "https://", "" -replace "http://", "" -replace "/", ""
    write-host "Domain from mainwp:" $domainname
    
    #Resolve the nameserver, and select the first.
    $NSRecords = Resolve-DnsName -Name $domainname -Type NS -DnsOnly -EA 1 | Where-Object { $_.QueryType -eq 'NS' }
    $nsforcatch = $NSRecords.NameHost | select-object -first 1

    #Mark common host patterns with "!"
    $nsstamp = "N/A"
    if ($nsforcatch -like "*simply*") { $nsstamp = "Simply" }
    if ($nsforcatch -like "*gratisdns*") { $nsstamp = "Gratis DNS" }
    if ($nsforcatch -like "*gigahost*") { $nsstamp = "Gigahost" }
    if ($nsforcatch -like "*curanet*") { $nsstamp = "Curanet" }
    if ($nsforcatch -like "*unoeuro*") { $nsstamp = "Unoeuro" }
                


    #Begin DNS collection
    $domainmx = (Resolve-DnsName $domainname -Type MX).NameExchange | select-object -first 1
    $domainarecord = (Resolve-DnsName $domainname -Type A).IPAddress | select-object -first 1

    #PTR RUN IF A-RECORD IS FOUND
    if ($domainarecord -ne $null) {
        $domainptridentify = Resolve-dnsname -name $domainarecord -Type PTR
        $domainptr = $domainptridentify.namehost | select-object -first 1
    }

    #GET SPF RECORD
    $SPFRecordget = Resolve-DnsName $domainname -Type TXT | Where-Object Strings -Match "spf1"
    $SPFRecordget | Select-Object @{Name = "DomainName"; Expression = { $_.Name } }, @{Name = "Record"; Expression = { $_.Strings } }
    $SPFRecordget | ConvertTo-Json

    $SPFRecord = $SPFRecordget.Text
    write-host "SPF IS: "$SPFRecord

    #write treated object out to log.    
    ("$($domainname),$($nsstamp),$($nsforcatch),$($SPFRecord),$($domainmx),$($domainptr),$($domainarecord)") | Out-File -Filepath $logfilepath –Append

} -ThrottleLimit 5


Import-Csv -path $logfilepath | Out-GridView

The Terminal erros i get are these:

(when just taking the first object)

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          20/04/2022    11:08              0 log-20-04-2022_11-08-46.csv
Domain from mainwp: Web request completed. (Number of bytes processed: 313992)                                       ]
Resolve-DnsName: 
Line |
  16 |      $NSRecords = Resolve-DnsName -Name $domainname -Type NS -DnsOnly  …
     |                                         ~~~~~~~~~~~
     | Cannot validate argument on parameter 'Name'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
Resolve-DnsName: 
Line |
  30 |      $domainmx = (Resolve-DnsName $domainname -Type MX).NameExchange | …
     |                                   ~~~~~~~~~~~
     | Cannot validate argument on parameter 'Name'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
Resolve-DnsName: 
Line |
  31 |      $domainarecord = (Resolve-DnsName $domainname -Type A).IPAddress  …
     |                                        ~~~~~~~~~~~
     | Cannot validate argument on parameter 'Name'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
Resolve-DnsName: 
Line |
  35 |  …   $domainptridentify = Resolve-dnsname -name $domainarecord -Type PTR
     |                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | N/A : DNS name contains an invalid character.
Resolve-DnsName: 
Line |
  40 |      $SPFRecordget = Resolve-DnsName $domainname -Type TXT | Where-Obj …
     |                                      ~~~~~~~~~~~
     | Cannot validate argument on parameter 'Name'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
null
SPF IS:
Out-File: 
Line |
  48 |  … ainptr),$($domainarecord)") | Out-File -Filepath $logfilepath –Append
     |                                                     ~~~~~~~~~~~~
     | Cannot bind argument to parameter 'FilePath' because it is null.
PS C:\Users\jtn>

I have some other scripts working quite similarly.
What am i missing? :smiley:

I think I’m onto the reason, working on a rework.

Will post an update later