Hi,
We have a website , its a reporting site where I can filter/bookmark the filters on conditions and it creates an tabular output on the page.
I can right click the page and select export to export the data to csv. Every time I do that it generates a random name for the csv file like ae7bxtjs84snbdwwemf845d.csv and I do a save as to change it to desired filename.
the question is how do I use Invoke-Webrequest $URL -Outfile C:\xyz.csv
if i use the same temp name, it works for a while and later downloads the empty file. Probably the cookie session expired or temp name expired.
I cannot share the URL since it is official. Is there any example that you can help me or suggest to work towards?
It’s probably not going to be that straight-forward. You can logon to the page with Invoke-WebRequest and persist the session with the WebSession parameter. Open up a browser, like Chrome, hit F12 for Developer Tools and go to Network. Then watch the URL’s and the body information that is being passed to the browser, which you emulate with Invoke-WebRequest.
With that said, when you do that, the CSV is being generated Server-Side, not in the browser. You have two basic options:
- Generate the HTML table and then parse that table.
- Emulate the entire process and click the filters and the buttons to see if you can generate the CSV.
It’s hard to help unless you see the calls that are being made.
What have you tried thus far?
Show you code attempt?
You say…
'I can right click the page and select export to export the data to csv.'
SO, you mean you can Right Click anywhere on the page and you get the export option?
So, this says to me that there is code on that page that allows this.
Therefore, this…
Invoke-Webrequest $URL -Outfile C:\xyz.csv
… will not do this. You have to use web page navigation and UI automation in this use case. You scrape the page to get all the elements and then work with those in code.
$w = Invoke-WebRequest -Uri 'https://powershell.org'
$w | Get-Member
$w.StatusCode
$w.AllElements
$w.AllElements.Count
$w.Links.Count
$w.Links
$w.Forms
$w.Forms[0].Fields
$w.RawContent
$w.ParsedHtml
Thanks for the suggestion. yes, I did follow the F12 path…
The URL doesnt change untill the csv export button is clicked and that is what confuses me.
The webpage is qlikview webpage hosted on a server.
Lets say the webpage is QlikView – Powerful Interactive Analytics & Dashboards | Qlik.
I select filters like manager name, server category… so far the URL remains the and only the table content changes inside the page.
I right click anywhere on the content and select export …
Now the URL changes to QlikView – Powerful Interactive Analytics & Dashboards | Qlik and I get save / saveas option in the browser bottom.
When I look into the body of this URL , it contains the comma seperated values that is displayed in the page.
The body of the URL before that seems to contain a $URL session value and a Temp name generated by calling a javascript internally…
Even when I change the filters, the URL doesn’t change until I select export… And then I see the same session value with a new Tempname.csv.
Each filters creates a new tempname and if I go back to previous filters it creates the same tempname again untill the session expires.
Looks complicated to me 
As I mentioned in my reply above. I tried to go through the F12 options to identify where it is navigating… but that didnt help much.
QlikViewPageSample
This is the sample page where left sides are filters and the data at the bottom part of the page. I can right click anywhere in the data and choose export excel or csv.
I tried all that you mentioned above… the problem is since the main URL is not changing, I do not know where the session value and temp value is getting generated and I see its calling a javascript in the previous step.
However, since the URL is not changing, the response with your codes above remains same as in the beginning.
Talking about UI automation, Yes, the same is achievable to RPA(Robotic Process Automation) and UIPath.
However, I want to do it in powershell
If the url stays the same, then most likely a POST is being performed with a Body. Assuming you are using Chrome, hit F12 and got to Network > XHR. When you do something the page, you should see a POST and you can look at the Request Headers and Request Payload (a.k.a Body). To test, you can just grab the same JSON and try posting it, something like this:
#What you saw in Request Payload
$jsonBody = @"
{
"filter": {
"Manager": "John",
"Category": "Awesome"
}
}
"@
$params = @{
Uri = 'www.qlikview.com/someheaderparameters'
Body = $jsonBody
Method = 'POST'
ContentType = 'application/json'
}
Invoke-WebRequest @params
When you know it’s working, you can generate a PSObject that would allow you to build a dynamic query with something like this:
$filter = [pscustomobject] @{
filter = [pscustomobject] @{
Manager = 'John'
Category = 'Awesome'
}
}
$jsonBody = $filter | convertTo-Json
$params = @{
Uri = 'www.qlikview.com/someheaderparameters'
Body = $jsonBody
Method = 'POST'
ContentType = 'application/json'
}
Invoke-WebRequest @params
Another place to look is if the vendor has documentation on the API:
https://help.qlik.com/en-US/qlikview-developer/November2017/Content/APIsAndSDKs.htm
Then you can even search to see if there are examples of Powershell:
It wasnt json response after all. The links you gave for reference looks good but the API docs are in qvw extension. I donot have qlikview software installed . just using the web page… I donot think I can open it… Is there any other way?
By the way, i took notes of the XHR details and took a pic for reference… . Check if it helps anyway to guide me…
If cannot, then I started getting thoughts of skipping it 
How do I attach images here? …
Why not check with Qlik?
If they can direct you to the correct API-call etc. for what you want to do?
It will be alot more predictive than trying to use web page navigation.
Good point… Working in corporate, you end up working with support Line Of Business who works with vendor. Unfortunately, I cannot ask them to reach out to vendor to ask for API’s. Or I am sure the internal team has the documentations. However, to open them I cannot get Qlik client installed on my desktop due to policy and scope restrictions…
Anyways, I do not want to dig deep into why what when… .
Thanks for all your responses. I will find a way through IE DOM Explorer and use COM object to achieve it.
If it’s not a closed API then the whole point of it is to be able to consume the data without a specific client.
E.g. using powershell, python, java or whatever as long as it can generate the call that the API requires.
Also Qlik have their own forums so maybe use that if you don’t want to open a support case.
It’s of course up to you how you want to proceed but web navigation will most likely not be that reliable.
Good luck.