VMware esxtop reporting

[Deek_f]

Hi,
I am trying to convert a bash shell script into a powershell script, If I can get it to work.
What the bash script does, is loop through all of my esxtop csv files & puls out specific data… i.e. % ready, vm diskread & disk write latency, among others.
I can get PS to run through & list everything on screen, but I’m struggling to find a way to Only report on the bits I want/need.
This is what I have so far…

Import-csv -Path H:\Desktop\DF\headers.csv
Get-EsxTop  Select VCPUID,% Ready | ft -AutoSize | ConvertTo-Html -Head $htmlformat -Body $bodyformat | Out-File H:\Desktop\DF\csv_html_test.html
Invoke-Expression .\csv_html_test.html

which (obviously) doesn’t work correct

This is what the bash script looks like.

arg1=$1

case $arg1 in

	"ready")
		csvHeader="vCPU, %READY, status"
		entityMatch="vmx-vcpu.*Ready"
		maxPassValue="499"
		# i.e. 4.99
		;;

	"vdiskread")
		csvHeader="vDisk, Read (ms), status"
		entityMatch="Virtual Disk.*scsi.*MilliSec/Read"
		maxPassValue="499"
		;;
		
	"vdiskwrite")
		csvHeader="vDisk, Write (ms), status"
		entityMatch="Virtual Disk.*scsi.*MilliSec/Write"
		maxPassValue="499"
		;;
	*)
		echo "no argument specified, use: ready, vdiskread, vdiskwrite"
		exit

esac

esxtopDirectory="$(ls -F1 *.csv)"
outputCsv="./output/output_${arg1}.csv"

if [ ! -f "${outputcsv}" ]
then
        echo "${csvHeader}" > "${outputCsv}"
else
        echo "${outputCsv} already exists"
fi

for esxtopBatchFile in ${esxtopDirectory}
do
	columnListCount=$(head -1 ${esxtopBatchFile} | tr ',' '\n' | nl | sed -e 's/^[[:space:]]*//' | sed -r 's/\s/,/' | grep -e "${entityMatch}" | wc -l)
	if [ "$columnListCount" != "0" ]
	then
		columnList=$(head -1 ${esxtopBatchFile} | tr ',' '\n' | nl | sed -e 's/^[[:space:]]*//' | sed -r 's/\s+/,/' | grep -e "${entityMatch}")
		columnNumber=$(echo -en "${columnList}" | awk -F "," '{print $1}')
		echo -e"\n$(pwd)/${esxtopBatchFile}\n"
		printf "%+10s %+8s %+8s   %-36s\n" "Line#" "Value" "Status" "Name"
		printf -- '-%.0s' $(seq 150); echo ""
		while IFS= read -r line
		do

			entityName=$(cat ${esxtopBatchFile} | awk -F "," -v var=$line 'BEGIN{ans=var} {print $ans}' | head -1 | cut -d "\\" -f 4)
			result=$(cat ${esxtopBatchFile} | awk -F "," -v var=$line 'BEGIN{ans=var} {print $ans}' | tail -n +2 | tr -d '"'| sort -r | head -1)
			resultInteger=$(echo "${result}" | tr -dc '0-9')

			#echo ${result}
			#echo ${resultInteger}

			if [ "${resultInteger}" -ge "${maxPassValue}" ]
			then
				status="FAIL"
			else
				status="PASS"
			fi
			
			echo "${entityName},${resultInteger},${status}" >> "${outputCsv}"
			
			printf "%+10s %+8s %+8s   %-36s\n" "${line}" "${result}" "${status}" "${entityName}"
			
		done <<<"${columnNumber}"
	else
		echo "skipping ${esxtopBatchFile}, \"${entityMatch}\" didn't match any columns"
	fi
done

any pointers would be greatly appreciated… many thanks

[matt-bloomfield]

Hi, welcome to the forum

You probably won’t get much of a response posting a bash script as your example, especially if you don’t explain what it’s doing.

What exactly are you trying to do? The PowerShell you’ve posted is all a bit random and doesn’t make any sense.

[Deek_f]

Hi,
Yeh, apologies…
So, what the bash script does. It Parses through the esxtop batch files we produces from our setup.

Taking the “ready” as example. the script runs though all of the cvs files (about 178 csv files), it looks at column headers & specifically looks for vmx-vcpu & %ready, it looks at the % ready figure for each entry & assigns it a pass or fail depending on MaxPassValue, so ready pass is 5%. it then writes all of the vmx-cpu %ready into another (Manageable)csv file. which then allows us to filter & display any vcpu that are above 5%.
same process for disk read & disk write statuses.
If I can get powershell to do the same thing, that would be great. Or am I on to plums & what Im trying to achieve cant be done with powershell?

hopefully that makes some sense…

Deekl

[Olaf]

@Deek_f

It Parses through the esxtop batch files

Do you really parse batch files? What exactly is a batch file for you?

@Deek_f

the script runs though all of the cvs files (about 178 csv files),

How about sharing a few sanitized lines with the relevant elements of those CSV files along with the expected results? (Please format those CSV samples as code as well here in the forum)

[matt-bloomfield]

I think, from what I understand of your description, that you just need to import the csv file, select the relevant columns, and use a calculated property for the pass or fail. Here’s an example based on your ‘ready’ report:

function Export-VMReportData {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory=$true)]
        [String]
        $esxtopPath,
        [Parameter(Mandatory=$true)]
        [String]
        $reportType,
        [Parameter(Mandatory=$true)]
        [String]
        $reportPath
    )

    switch ($reportType) {
        ready {
            # Define calculated property for Status column
            $statusExpression = @{
                Label = 'Status'
                Expression = {
                    if ([int]$_.'%Ready' -gt 5) {
                        'FAIL'
                    }
                    elseif ([int]$_.'%Ready' -le 5) {
                        'PASS'
                    }
                }
            }
            foreach ($file in (Get-ChildItem $esxtopPath)) {
                # Import CSV only if %Ready is in the header row
                if ((Get-Content $file.FullName -TotalCount 1) -match '%Ready') {
                    Import-Csv $file.FullName | 
                        Select-Object vmx-vcpu,%Ready,$statusExpression  | 
                            Export-CSV -Path $reportPath -NoTypeInformation -Append
                }
            }                  
        }
    }
}

Export-VMReportData -esxtopPath E:\Temp\Files\vmdata -reportType ready -reportPath E:\Temp\Files\ready.csv

[Deek_f]

ok, parse in the loosest sense I`ll get a sanitised sample sorted.

[Deek_f]

I`ll give that a test, & see how i tgoes.

Thank you

[Deek_f]

@matt-bloomfield

So Matt, your script almost works…I`ll try to figure out what Is wrong, but any help would be appreciated…

This is what I get from your script…

**I can only embedd 1 file at a time, will add a sample of wht I would like to see…in next post

Also, not yet figured out why… but your script generates a 54mb file, a little bigger than I would expect… does the script look in every sub folder & go through all csv files?

[Deek_f]

this is what I would usually expect

I need to see host/server names to know what ones are haing issues

[Olaf]

@Deek_f

but any help would be appreciated…

Please do not post images of code or sample data as this is not helpful at all. Instead post the plain text formatted as code. And please do not create new posts - go back and edit and fix your existing ones.

Thanks in advance.

[Deek_f]

Hi, sorry… I was only trying to show results of previous code…I didnt think I had started a new topic, pretty sure I only hit reply…

[Deek_f]

Ok, so I figured out how to get the vm host name in place, just add vcpu to “Select-object”.
but have now run into a different problem… I can now see why the PS generates a bigger file than expected…my esxtop data is taken over 300/600 & 1200 secs, so script is listing same servers 3 times, Is it possible to remove dublicates?

[matt-bloomfield]

You can remove the duplicates by importing all the data, then selecting just unique data:

    switch ($reportType) {
        ready {
            # Define calculated property for Status column
            $statusExpression = @{
                Label = 'Status'
                Expression = {
                    if ([int]$_.'%Ready' -gt 5) {
                        'FAIL'
                    }
                    elseif ([int]$_.'%Ready' -le 5) {
                        'PASS'
                    }
                }
            }
            $csvData = foreach ($file in (Get-ChildItem $esxtopPath)) {
                # Import CSV only if %Ready is in the header row
                if ((Get-Content $file.FullName -TotalCount 1) -match '%Ready') {
                    Import-Csv $file.FullName 
                }
            }
            $csvData | Select-Object vmx-vcpu,%Ready,$statusExpression -Unique | 
                Export-CSV -Path $reportPath -NoTypeInformation -Append                  
        }
    }

[Deek_f]

bah, looks like I was wrong with the Vcpu thing in select object…I was sure it worked… but now doesn’t seem to be

Ok, re the duplicates…

& thanks again, for your help on this…

[Deek_f]

actually… Loks like I jumped the gun a little bit… so the ps does sort of work, but only on the files that have previously run through the bash script… I think it is to do with the %Ready bit… that only appears in files that have already been run throgh bash script… I`m still getting a sanitised version of a esxtop csv file.

[matt-bloomfield]

I just made a guess at the column titles based on your post.
The entityMatch field looks like it’s using a wildcard *Ready for the column header.

According to VMWare docs, the header is CPU Ready. So try 'CPU Ready' (including the quote marks) in the Select-Object statement.

[Deek_f]

Thank You Matt… (ill need to buy you a virtual beer) Im heading to the airport in 10 mins & wont be arouind till tomorrow, *lll give that a try & let you know how I get on.

Thanks
Deek

[Deek_f]

Apologies, didnt update over the weekend… esxtop batch generates a large cvs & getting it loaded is a bit of a pain… too many columns.

This is what the 1st column headers looks like…but goes on for much more…(too big to load into excel)

2098063:iscsi-rx-world-22)\% CoStop","\\chassis-name-2-blade-esxi-1.domain.name\Vcpu(1:system:2098063:iscsi-rx-world-22)\% Ready","\\chassis-name-2-blade-esxi-1.domain.name\Vcpu(1:system:2098063:iscsi-rx-world-22)\% Swap Wait","\\chassis-name-2-blade-esxi-1.domain.name\Vcpu(1:system:2098064:iscsi-rx-world-23)\% Ready"

So I am intersted in this column…
\chassis-name-2-blade-esxi-1.domain.name\Vcpu(1:system:2098063:iscsi-rx-world-22)% Ready"

& I need the full column name in the report. at the moment it is a empty column.

What I hope to achieve is a report with 3 columns, 1 with blade/server name 1 with % count & one with pass / fail
At the moment I have the pass column only. I was/am guessing its the select object statement I need to fluff around with to get the full path for the ready statement.

back to drawing board for me

[matt-bloomfield]

I’m not sure that we have the full picture here.

You could, of course, just stick the full column name in the Select-Object statement but that’s not very dynamic.

Does the existing script have any other input? For example, is vmx-vcpu actually a variable
so that when entityMatch is processed, it becomes

chassis-name-2-blade-esxi-1.domain.name.*Ready

That would then match the servername, plus .* (regex for any character, zero or more times), plus Ready.

[krzydoug]

@Deek_f

This is what the 1st column headers looks like…but goes on for much more…(too big to load into excel)

2098063:iscsi-rx-world-22)\% CoStop","\\chassis-name-2-blade-esxi-1.domain.name\Vcpu(1

I think we need to take a huge step back and look at the actual csv file. What type of property names are those? I’ve never seen anything like it from standard esxtop or Get-EsxTop. From the image you posted, it appears the property names (headers) are vcpuname, %Ready, and status. Do you mean this is the first value from the VCPuName property (column)?

Can you share more information on how the data is collected and written in the first place? Seems we are trying to do complex “parsing” of poorly structured data.