Extracting data from a URL

PowerShell Help
1

Hello everyone,
I am unfamiliar with PowerShell and have a need to put together a script to extract data, but I am having problems with syntax.
Since I have no clue what I am doing, maybe if I explain what I am trying to do, someone would be nice enough to offer some help.
I have 3 URL’s (all 3 are similar except one is Server1, one is Server2, and Server3) that I need to pull data from. It is something that I will need to automate and has a date component that will change (always looking for the past 7 days).
If I run the command in PowerShell, it will work. However, I need to set variables for the 3 different URL’s and date calculations, but I cannot find how to do it.
Here is what works. I can set the login and password:
</> $username = “mylogin”
</> $password = Get-Content ‘C:\Programs\mysecurestring.txt’ | ConvertTo-SecureString
</> $cred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, $password
If I manually enter each URL and manually set the dates, I can get the following to successfully execute:
</> Invoke-WebRequest -Uri ‘https://login-server1.com/services/rest/access/api/v1/audit/events/1?fromDate=2022-03-24T00:00:00.000Z&toDate=2022-03-31T00:00:00.000Z&pageSize=300000’ -Credential $cred -OutFile C:\Programs\Server1-2022-03-31.xml
I need a way to set variables and pass 3 different URL’s, and programmatically pick today’s date (in the format shown) and from a week ago. I need the output to go to file names containing the server’s name and current date (so it does not overwrite each week).
Any assistance is appreciated.
Thank you.

2

BS,
Welcome to the forum. :wave:t4:

When you post code, sample data, console output or error messages please format it as code using the preformatted text button ( </> ). Simply place your cursor on an empty line, click the button and paste your code.
Thanks in advance
How to format code in PowerShell.org

In PowerShell you can get the current date with

Please always read the complete help including the examples of the cmdlets you’re about to use to learn how to use them.

Since PowerShell is about working with objects properties and methods you can even calculate with dates. So you can easily add or substract minutes, hours, days or month from a given date or time. For example:

(Get-Date).Date.AddDays(-7)
3

Thanks Olaf. I had found the Get-Date command, but I have not found how to build the command line for the get using variables.

4

Hmmm … I actually don’t know what that means. You could search for “PowerShell & Variables” or “PowerShell & Strings” and you would get something like this:

If you have some code you just got stuck with please share it along with an explanation what’s working as expected and we can try to help you further.

5

Hi again,
I appreciate your assistance. I apologize if my last post did not make sense. I am having difficulties putting the variables together in a coherent string that works. I have tried multiple variations similar to the following:

 $instances = "login-server1" , "login-server2" , "login-server3"

 ForEach ($instance in $instances) 
   {
   Invoke-WebRequest -Uri "'https://' + $instance + '.com/services/rest/access/api/v1/audit/events/1?/fromDate=' + '$((Get-Date.AddDays(-7) -Format yyyy-MM-dd))' + 'T00:00:00.000Z&toDate=' + '$(Get-Date -Format yyyy-MM-dd)' + 'T00:00:00.000Z&pagesSize=300000'" -Credential $cred -OutFile C:\Programs\$instance + '-' + $(Get-Date -Format yyyy-MM-dd)
    }

I have the cred variable set earlier for the login and password.

I’ve tried using plus signs, not using plus signs, using quotes, not using quotes, single quotes versus double quotes, parenthesis vs no parenthesis. I’ve read various things online, but it is still clear as mud when these are needed and when they are not in this type of instance. This is the area I’m having difficulty in.

Thanks again.

6

So you did not read the help I linked in my last answer. :smirk:

You basically have three options.

Assumed you have a variable with the name $Variable

$Variable = 'variableString'

and you want to insert the content of this variable in the middle of a given string.

You can use concatenation like this:

'StringStart_' + $Variable + '_StringEnd'

You basically add the parts of the string together with plus signs. And in this case it does not matter if you use single or double quotes.

Next option is to use variable substitution. Therefor you have to use double quotes for the outer quotes:

"StringStart_$($Variable)_StringEnd"

With using the subexpression operator " $() " you can make sure to properly expand the variable especially when you’re using objects with properties instead of simple plain string variables.

The third option is to use the format operator " -f "

'StringStart_{0}_StringEnd' -f $Variable

Here it doesn’t matter as well if you use single or double quotes.

2 Likes
7

Hi Olaf,
I have read the doc pages on Microsoft prior to coming here. I still do not see how to construct a line of code that works. I have tried variations like

$instances = 'login-server1' , 'login-server2' , 'login-server3'

$weekago = '$Get-Date.AddDays(-7) -Format yyyy-MM-dd'

$today = '$Get-Date.AddDays -Format yyyy-MM-dd'

ForEach ($instance in $instances) 
    {
   Invoke-WebRequest -Uri 'https://' + $instance + '.com/services/rest/access/api/v1/audit/events/1?fromDate=' + $weekago + 'T00:00:00.000Z&toDate=' + $today + 'T00:00:00.000Z&pageSize=300000' -Credential $cred -OutFile C:\Programs\file1.csv
      Invoke-WebRequest -Uri 'https://_'$instance'_.com/services/rest/access/api/v1/audit/events/1?fromDate=_'$weekago'_T00:00:00.000Z&toDate=_'$today'_T00:00:00.000Z&pageSize=300000_' -Credential $cred -OutFile C:\Programs\file2.csv
  }

No matter what I try it does not work. Can I please just get help with this statement? Thanks

8

You may do a BIG step back and restart with learning the very basics of PowerShell first. You cannot learn a complex technology like scripting in PowerShell by guessing or by piecing together some arbitrary snippets of code you’ve found on the internet.

Either you have strings you enclose in quotes or you have cmdlets or other code you do not enclose in quotes.

$instanceList = 'login-server1' , 'login-server2' , 'login-server3'
$weekago = (Get-Date).AddDays(-7) -Format 'yyyy-MM-dd'
$today = Get-Date -Format 'yyyy-MM-dd'

ForEach ($instance in $instanceList) {
    $URI = 'https://' + $instance + '.com/services/rest/access/api/v1/audit/events/1?fromDate=' + $weekago + 'T00:00:00.000Z&toDate=' + $today + 'T00:00:00.000Z&pageSize=300000'
    $FileName = 'C:\Programs\' + $instance + '_output.txt'
    Invoke-WebRequest -Uri $URI -Credential $cred -OutFile $FileName
}
9

I agree. I do want to learn the language, but I am forced to OTJ training as I have been handed duties that I am not trained in.

I do appreciate your help and patience, but even with the code you provided I was still getting errors.

Get-Date.AddDays : The term ‘Get-Date.AddDays’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At Logon-Test3.ps1:21 char:12

  • $weekago = Get-Date.AddDays(-7) -Format ‘yyyy-MM-dd’
  •        ~~~~~~~~~~~~~~~~
    • CategoryInfo : ObjectNotFound: (Get-Date.AddDays:String) , CommandNotFoundException
    • FullyQualifiedErrorId : CommandNotFoundException

However, your insight on breaking the ForEach loop down instead of trying to cram it all into a single line was a major step forward.

I wound up setting another variable for the date and then setting the $today and $weekago variable from it

$pig = Get-Date
$weekago = Get-Date $pig.AddDays(-7) -Format 'yyyy-MM-dd'
$today = Get-Date $pig -Format 'yyyy-MM-dd'

That allowed me to pass the date without getting an error.

I miss the old days of simple bash scripts. Sigh

Thank you very much for the help!!!

10

try below to fix the error.

$weekago = (Get-Date).AddDays(-7)  | Get-Date -Format 'yyy-MM-dd'

olaf explained the process really well. follow it one step at a time and you’ll get there.

11

Ooops. Sorry. My bad. I forgot the parenthesis. I corrected my code suggestion. :wink:

I think you will change your mind when you finally learned the basics and recognize the beauty, the consistancy and the power of PowerShell.