Manipulating csv with storage counters

PowerShell Help
1

Hi

I was hoping to play and learn with a task in powershell that at least sounded pretty easy in my head, I guess it is pretty easy stuff but I’m obviously doing something very wrong. After a couple of hours of trial and error I’m pretty sure I’m approaching this the wrong way.

I recently installed a tool to get some statistics/values from our storage system.
The tool produces a bunch of csv files (attached one of them)

I started with just trying to play with the first four columns: Timestamp, MDisk, Read operations (IO/s),Write operations (IO/s)

I’m trying to calculate the average read and write IOPS per MDisk.
Wit the code I got now, it actually outputs a view I was kind of happy with, at least for a start.
Problem is that the final $object only has one (the last) mdisk and its values in it.

I kind of understand that I’m doing something stupid with the foreach statement that’s causing this, and I’ve tried a couple of things like initializing an empty array to be able to use +=$object.

My hope is to be able to use these csv files to parse the data in a nice way in the future, but as I got stuck right at the beginning of playing and started to get disgruntled I was hoping someone could guide me in the right direction.

The code:

$csv = Import-Csv .\Reports\StorageV7000.csv

foreach ($mdisk in $csv | Select-Object -ExpandProperty MDisk -Unique)
{

        $avgRead = $csv | Where-Object MDisk -eq $mdisk | Measure-Object -Property "Read operations (IO/s)" -Average -Maximum -Minimum
        $avgWrite = $csv | Where-Object MDisk -eq $mdisk | Measure-Object -Property "Write operations (IO/s)" -Average -Maximum -Minimum



       $object = [pscustomobject]@{
      

                'MDisk' = $mdisk
                'Average Read IOPS' = [System.Math]::Round($avgRead.Average,2)
                'Average Write IOPS' = [System.Math]::Round($avgWrite.Average,2)
                'Average Total IOPS' = [System.Math]::Round($avgRead.Average+$avgWrite.Average,2)
        }



        $object


}
2

Looks like you just need to gather up all of your objects into a collection. Try this:

$csv = Import-Csv .\Reports\StorageV7000.csv

$report = foreach ($mdisk in $csv | Select-Object -ExpandProperty MDisk -Unique)
{
 
        $avgRead = $csv | Where-Object MDisk -eq $mdisk | Measure-Object -Property "Read operations (IO/s)" -Average -Maximum -Minimum
        $avgWrite = $csv | Where-Object MDisk -eq $mdisk | Measure-Object -Property "Write operations (IO/s)" -Average -Maximum -Minimum
 
       [pscustomobject]@{
                'MDisk' = $mdisk
                'Average Read IOPS' = [System.Math]::Round($avgRead.Average,2)
                'Average Write IOPS' = [System.Math]::Round($avgWrite.Average,2)
                'Average Total IOPS' = [System.Math]::Round($avgRead.Average+$avgWrite.Average,2)
        } 
}

$report
3

On a side note, you may find this version slightly clearer:

$csv = Import-Csv .\Reports\StorageV7000.csv

$report = $csv |
Group-Object -Property MDisk |
ForEach-Object {
    $diskGroup = $_
    $avgRead = $diskGroup.Group | Measure-Object -Property "Read operations (IO/s)" -Average -Maximum -Minimum
    $avgWrite = $diskGroup.Group | Measure-Object -Property "Write operations (IO/s)" -Average -Maximum -Minimum
    
    [pscustomobject]@{
        'MDisk'              = $diskGroup.Name
        'Average Read IOPS'  = [System.Math]::Round($avgRead.Average,2)
        'Average Write IOPS' = [System.Math]::Round($avgWrite.Average,2)
        'Average Total IOPS' = [System.Math]::Round($avgRead.Average+$avgWrite.Average,2)
    } 
}

With your calls to Select-Object -Property MDisk -Unique, and later Where-Object MDisk -eq $mdisk, you were basically recreating the functionality of Group-Object in a less efficient way.

4

Thanks Dave, I’m gonna go with your last suggestion, haven’t used Group-Object alot.
I know there a lot of ways of doing this stuff in PS, but would you (or anyone else) say this is decent approach or are there any other (better) ways?

This is what I got right now with your help:

$csv = Import-Csv .\Reports\StorageV7000.csv
 
$report = $csv |
Group-Object -Property MDisk |
ForEach-Object {
        $diskGroup    = $_
        $avgReadIO    = $diskGroup.Group | Measure-Object -Property "Read operations (IO/s)" -Average -Maximum -Minimum
        $avgWriteIO   = $diskGroup.Group | Measure-Object -Property "Write operations (IO/s)" -Average -Maximum -Minimum
        $avgReadB     = $diskGroup.Group | Measure-Object -Property "Read blocks (Byte/s)" -Average -Maximum -Minimum
        $avgWriteB    = $diskGroup.Group | Measure-Object -Property "Write blocks (Byte/s)" -Average -Maximum -Minimum
        $avgReadUSEC  = $diskGroup.Group | Measure-Object -Property "Read external response time (usec)" -Average -Maximum -Minimum
        $avgWriteUSEC = $diskGroup.Group | Measure-Object -Property "Write external response time (usec)" -Average -Maximum -Minimum


        [decimal]$avgReadMB  = $avgReadB.Average /1MB
        [decimal]$avgWriteMB = $avgWriteB.Average /1MB
        [decimal]$avgReadMS  = $avgReadUSEC.Average /1000
        [decimal]$avgWriteMS = $avgWriteUSEC.Average /1000
    
 
    [pscustomobject]@{
        'MDisk'       = $diskGroup.Name
        'Read IOPS'   = [System.Math]::Round($avgReadIO.Average,2)
        'Write IOPS'  = [System.Math]::Round($avgWriteIO.Average,2)
        'Total IOPS'  = [System.Math]::Round($avgReadIO.Average+$avgWriteIO.Average,2)
        'Read MB/s'   = [System.Math]::Round($avgReadMB,2)
        'Write MB/s'  = [System.Math]::Round($avgWriteMB,2)
        'Read MS'     = [System.Math]::Round($avgReadMS,2)
        'Write MS'    = [System.Math]::Round($avgWriteMS,2)

    } 
}

Thinking that next step somehow should be to include the Storage Pools.
For example mdisk0-5 are in one Pool, so maybe create another object that just sums some of the values of those mdisks, that should be easy, let’s see how that goes :slight_smile:
I’ll keep this post updated if that’s okay.

Also, any feedback/suggestions on how to go forward with this, other angles or code cleanup are always appreciated.