Rolling buffer?

by Dwayne Dibbley at 2012-10-06 03:20:36

Is it possible to have a rolling buffer eg 5 values, and every time a new value is added the oldest is dropped out?

The idea is to read a log file with a compass heading every line, and smooth out the data by reading for example 5 values into the buffer then reading the average from the buffer?

Thanks
by jonhtyler at 2012-10-06 04:56:07
There is nothing native in Powershell or in .NET that would do this natively. I believe you would have to roll your own class to handle the logic of removing the oldest entry when a new one gets queued. If you search for "Circular Buffer in .NET" you can find some examples that might get you on your way. Since Powershell is based on .NET, you can fairly easily create the class and have Powershell implement it.
by MattG at 2012-10-06 07:27:35
I think that a queue is what you’re looking for. The following code should get you started. I hope this helps.
[Int] $Bearings = Get-Content .\bearings.txt

$Queue = New-Object System.Collections.Queue(5)
$BufferLength = 5
$RunningTotal = 0
$Count = 0

foreach ($Bearing in $Bearings)
{
$RunningTotal += $Bearing

if ($Queue.Count -eq $BufferLength)
{
$Queue.Dequeue() | Out-Null
$Queue.Enqueue($Bearing)
}
else
{
$Queue.Enqueue($Bearing)
}

$Count++

if ($Count % $BufferLength -eq 0)
{
Write-Host "Average: $($RunningTotal/$BufferLength)"
$RunningTotal = 0
}
}

My bearings.txt consisted of the following values:
0
20
18
322
187
67
98
351
4
93
35
211
by Dwayne Dibbley at 2012-10-07 02:26:53
Thanks,

If i was reading a standard NMEA log like:

$GPRMC,090353.021,A,5137.4042,N,00246.6906,W,029.3,143.7,300912,A78
$GPGGA,090353.071,5137.4038,N,00246.6901,W,1,07,1.3,30.5,M,51.4,M,0000
76
$GPRMC,090353.071,A,5137.4038,N,00246.6901,W,029.6,143.5,300912,A70
$GPGGA,090353.121,5137.4034,N,00246.6896,W,1,07,1.3,30.6,M,51.4,M,0000
72
$GPRMC,090353.121,A,5137.4034,N,00246.6896,W,029.8,143.3,300912,A7F
$GPGGA,090353.171,5137.4030,N,00246.6892,W,1,07,1.3,30.6,M,51.4,M,0000
77
$GPRMC,090353.171,A,5137.4030,N,00246.6892,W,030.1,143.2,300912,A7A
$GPGGA,090353.221,5137.4027,N,00246.6887,W,1,07,1.3,30.6,M,51.4,M,0000
73
$GPRMC,090353.221,A,5137.4027,N,00246.6887,W,030.4,143.0,300912,A79
$GPGGA,090353.271,5137.4023,N,00246.6882,W,1,07,1.3,30.6,M,51.4,M,0000
77
$GPRMC,090353.271,A,5137.4023,N,00246.6882,W,030.7,142.9,300912,A76
$GPGGA,090353.321,5137.4019,N,00246.6877,W,1,07,1.3,30.7,M,51.4,M,0000
71
$GPRMC,090353.321,A,5137.4019,N,00246.6877,W,031.0,142.8,300912,A76
$GPGGA,090353.371,5137.4015,N,00246.6872,W,1,07,1.3,30.7,M,51.4,M,0000
7D
$GPRMC,090353.371,A,5137.4015,N,00246.6872,W,031.3,142.7,300912,A76
$GPGGA,090353.421,5137.4011,N,00246.6868,W,1,07,1.3,30.7,M,51.4,M,0000
70
$GPRMC,090353.421,A,5137.4011,N,00246.6868,W,031.5,142.7,300912,A7D
$GPGGA,090353.471,5137.4007,N,00246.6863,W,1,07,1.3,30.8,M,51.4,M,0000
76
$GPRMC,090353.471,A,5137.4007,N,00246.6863,W,031.8,142.6,300912,A78
$GPGGA,090353.521,5137.4002,N,00246.6858,W,1,07,1.3,30.8,M,51.4,M,0000
7F
$GPRMC,090353.521,A,5137.4002,N,00246.6858,W,032.1,142.6,300912,A7B
$GPGGA,090353.571,5137.3998,N,00246.6853,W,1,07,1.3,30.8,M,51.4,M,0000
7C
$GPRMC,090353.571,A,5137.3998,N,00246.6853,W,032.4,142.6,300912,A7D
$GPGGA,090353.621,5137.3994,N,00246.6848,W,1,07,1.3,30.9,M,51.4,M,0000
7D
$GPRMC,090353.621,A,5137.3994,N,00246.6848,W,032.6,142.6,300912,A7F
$GPGGA,090353.671,5137.3990,N,00246.6843,W,1,07,1.3,30.9,M,51.4,M,0000
77
$GPRMC,090353.671,A,5137.3990,N,00246.6843,W,032.9,142.6,300912,A7A
$GPGGA,090353.721,5137.3986,N,00246.6838,W,1,07,1.3,30.9,M,51.4,M,0000
78
$GPRMC,090353.721,A,5137.3986,N,00246.6838,W,033.2,142.7,300912,A7E
$GPGGA,090353.771,5137.3981,N,00246.6833,W,1,07,1.3,31.0,M,51.4,M,0000
79
$GPRMC,090353.771,A,5137.3981,N,00246.6833,W,033.4,142.7,300912,A71
$GPGGA,090353.821,5137.3977,N,00246.6828,W,1,07,1.3,31.0,M,51.4,M,0000
70
$GPRMC,090353.821,A,5137.3977,N,00246.6828,W,033.7,142.7,300912,A7B
$GPGGA,090353.871,5137.3973,N,00246.6823,W,1,07,1.3,31.0,M,51.4,M,0000
7A
$GPRMC,090353.871,A,5137.3973,N,00246.6823,W,034.0,142.7,300912,A71
$GPGGA,090353.921,5137.3968,N,00246.6818,W,1,07,1.3,31.1,M,51.4,M,0000
7D


and using your stack code to determine the course taken ( if last bearing was 10 and current bearing 12 then that would be a 2 degree turn to the right? ) does the following look correct?

$file = new-object System.IO.StreamWriter("C:\test.txt")

$lasttimestamp = [datetime]::ParseExact("000000.000","HHmmss.fff",$null)

$Queue = New-Object System.Collections.Queue(5)
$BufferLength = 5
$RunningTotal = 0
$Count = 0

$SourceFile = "C:\LOG.txt"
$data = get-content $SourceFile
foreach ( $line in $data )
{
if ($line -match ".GPRMC.")
{
$splitline = $line.split(",")

$timestamp = [datetime]],"HHmmss.fff",$null)

$course = [decimal]$splitline[8]

$RunningTotal += $course

if ($Queue.Count -eq $BufferLength)
{
$Queue.Dequeue() | Out-Null
$Queue.Enqueue($course)
}
else
{
$Queue.Enqueue($course)
}

$Count++

if ($Count % $BufferLength -eq 0)
{
$heading = $($RunningTotal/$BufferLength) - $lastbearing
$file.WriteLine("$timestamp heading: $heading")
$lastbearing = $($RunningTotal/$BufferLength)
$RunningTotal = 0
}


}
}

$file.Close()
by MattG at 2012-10-07 07:34:17
Without looking into your code too much, I do have some suggestions that you might find useful. Since the NMEA log is simply a CSV file, you could use ConvertFrom-Csv to perform some really interesting analytics on your data. For example, observe the following:
$Data = Get-Content .\log.txt
$GPRMC = $Data -match '^$GPRMC.+'
$Headers = 'MinSentence','FixTime','Active','LatVal','LatDir','LongVal','LongDir','GndSpeed','TrackAngle','Date','MagVarVal','MagVarDir','Chksum'
$Table = ConvertFrom-Csv -InputObject $GPRMC -Header $Headers
$Table | Format-Table -Property * -AutoSize

This will output the following:
MinSentence FixTime Active LatVal LatDir LongVal LongDir GndSpeed TrackAngle Date MagVarVal MagVarDir Chksum
----------- ------- ------ ------ ------ ------- ------- -------- ---------- ---- --------- --------- ------
$GPRMC 090353.021 A 5137.4042 N 00246.6906 W 029.3 143.7 300912 A
78
$GPRMC 090353.071 A 5137.4038 N 00246.6901 W 029.6 143.5 300912 A70
$GPRMC 090353.121 A 5137.4034 N 00246.6896 W 029.8 143.3 300912 A
7F
$GPRMC 090353.171 A 5137.4030 N 00246.6892 W 030.1 143.2 300912 A7A
$GPRMC 090353.221 A 5137.4027 N 00246.6887 W 030.4 143.0 300912 A
79
$GPRMC 090353.271 A 5137.4023 N 00246.6882 W 030.7 142.9 300912 A76
$GPRMC 090353.321 A 5137.4019 N 00246.6877 W 031.0 142.8 300912 A
76
$GPRMC 090353.371 A 5137.4015 N 00246.6872 W 031.3 142.7 300912 A76
$GPRMC 090353.421 A 5137.4011 N 00246.6868 W 031.5 142.7 300912 A
7D
$GPRMC 090353.471 A 5137.4007 N 00246.6863 W 031.8 142.6 300912 A78
$GPRMC 090353.521 A 5137.4002 N 00246.6858 W 032.1 142.6 300912 A
7B
$GPRMC 090353.571 A 5137.3998 N 00246.6853 W 032.4 142.6 300912 A7D
$GPRMC 090353.621 A 5137.3994 N 00246.6848 W 032.6 142.6 300912 A
7F
$GPRMC 090353.671 A 5137.3990 N 00246.6843 W 032.9 142.6 300912 A7A
$GPRMC 090353.721 A 5137.3986 N 00246.6838 W 033.2 142.7 300912 A
7E
$GPRMC 090353.771 A 5137.3981 N 00246.6833 W 033.4 142.7 300912 A71
$GPRMC 090353.821 A 5137.3977 N 00246.6828 W 033.7 142.7 300912 A
7B
$GPRMC 090353.871 A 5137.3973 N 00246.6823 W 034.0 142.7 300912 A*71

Also, I was probably premature to suggest using queues. You could simply use a five-element array and use the modulo operator to rotate through the array as can be seen here:
$BufferSize = 5
$LatArray = New-Object Single($BufferSize)
$i = 0

foreach ($Waypoint in $Table)
{
$LatArray[$i % $BufferSize] = $Waypoint.LatVal
$i++
}

Without performing a full code review of your work, I think you have a good starting point to do some really interesting NMEA parsing. Have fun!
by Dwayne Dibbley at 2012-10-07 09:48:37
Thanks again, would i be correct with the following using the measure object on the array to read the average?

foreach ($Waypoint in $Table)
{
$LatArray[$i % $BufferSize] = $Waypoint.TrackAngle
$Output = $LatArray | measure-object -average
$Output.Average
$i++
}
by MattG at 2012-10-07 10:22:12
Absolutely! That would be a great use of Measure-Object.
by Dwayne Dibbley at 2012-10-08 02:56:33
is there a way to tweak the array to account for the shift from example:358 degrees to 0 degrees? as this triggers the average to drop by 70 when i should only drop by 1 or 2?

example showing the array (CourseAverage) and real course (course) and the Diff (HeadingDiff) between last CourseAverage and Current CourseAverage

UTCTime:155837.791 CourseAverage:352 Course:356 Speed:31 HeadingDiff:2
UTCTime:155837.891 CourseAverage:354 Course:358 Speed:32 HeadingDiff:2
UTCTime:155837.991 CourseAverage:284 Course:0 Speed:32 HeadingDiff:-70
UTCTime:155838.091 CourseAverage:214 Course:2 Speed:32 HeadingDiff:-70
UTCTime:155838.191 CourseAverage:144 Course:4 Speed:33 HeadingDiff:-70
UTCTime:155838.291 CourseAverage:74 Course:7 Speed:33 HeadingDiff:-70
UTCTime:155838.391 CourseAverage:4 Course:9 Speed:34 HeadingDiff:-70
UTCTime:155838.491 CourseAverage:7 Course:11 Speed:34 HeadingDiff:3
UTCTime:155838.591 CourseAverage:8 Course:12 Speed:35 HeadingDiff:1
UTCTime:155838.691 CourseAverage:10 Course:12 Speed:35 HeadingDiff:2
UTCTime:155838.791 CourseAverage:11 Course:12 Speed:35 HeadingDiff:1
UTCTime:155838.891 CourseAverage:12 Course:12 Speed:36 HeadingDiff:1
UTCTime:155838.991 CourseAverage:12 Course:11 Speed:36 HeadingDiff:0
UTCTime:155839.091 CourseAverage:11 Course:10 Speed:36 HeadingDiff:-1
UTCTime:155839.191 CourseAverage:11 Course:9 Speed:36 HeadingDiff:0
UTCTime:155839.291 CourseAverage:10 Course:7 Speed:36 HeadingDiff:-1
UTCTime:155839.391 CourseAverage:9 Course:6 Speed:36 HeadingDiff:-1
UTCTime:155839.491 CourseAverage:7 Course:5 Speed:35 HeadingDiff:-2
UTCTime:155839.591 CourseAverage:6 Course:4 Speed:35 HeadingDiff:-1
UTCTime:155839.691 CourseAverage:5 Course:3 Speed:35 HeadingDiff:-1
UTCTime:155839.791 CourseAverage:4 Course:2 Speed:34 HeadingDiff:-1
UTCTime:155839.891 CourseAverage:3 Course:1 Speed:34 HeadingDiff:-1
UTCTime:155839.991 CourseAverage:2 Course:0 Speed:34 HeadingDiff:-1
UTCTime:155840.091 CourseAverage:73 Course:360 Speed:33 HeadingDiff:71
UTCTime:155840.191 CourseAverage:145 Course:359 Speed:32 HeadingDiff:72
UTCTime:155840.291 CourseAverage:216 Course:359 Speed:32 HeadingDiff:71
UTCTime:155840.391 CourseAverage:287 Course:358 Speed:31 HeadingDiff:71
UTCTime:155840.491 CourseAverage:359 Course:358 Speed:30 HeadingDiff:72
UTCTime:155840.591 CourseAverage:359 Course:358 Speed:29 HeadingDiff:0
UTCTime:155840.691 CourseAverage:358 Course:358 Speed:29 HeadingDiff:-1
UTCTime:155840.791 CourseAverage:358 Course:359 Speed:28 HeadingDiff:0
UTCTime:155840.891 CourseAverage:359 Course:360 Speed:27 HeadingDiff:1
UTCTime:155840.991 CourseAverage:287 Course:0 Speed:26 HeadingDiff:-72
UTCTime:155841.091 CourseAverage:216 Course:2 Speed:26 HeadingDiff:-71
UTCTime:155841.191 CourseAverage:145 Course:3 Speed:25 HeadingDiff:-71
UTCTime:155841.291 CourseAverage:74 Course:5 Speed:24 HeadingDiff:-71
UTCTime:155841.391 CourseAverage:3 Course:7 Speed:24 HeadingDiff:-71
UTCTime:155841.491 CourseAverage:5 Course:9 Speed:23 HeadingDiff:2
UTCTime:155841.591 CourseAverage:7 Course:12 Speed:23 HeadingDiff:2
UTCTime:155841.691 CourseAverage:9 Course:15 Speed:22 HeadingDiff:2
UTCTime:155841.791 CourseAverage:12 Course:18 Speed:22 HeadingDiff:3
UTCTime:155841.891 CourseAverage:15 Course:22 Speed:21 HeadingDiff:3


Thanks again
by MattG at 2012-10-08 07:11:10
What you’re asking is no longer a PowerShell question but a general computer science question. I’ll take a stab at it though. ;D

You should ask yourself, how does your algorithm differentiate between say a heading difference of 2 versus a heading difference of -358? What you would need to do to make that determination is calculate the instantaneous rate of change in the course. If the rate of change from the previous samples is positive, then it is most likely that the heading diff is 2. Conversely, if the rate of change is negative, then the heading diff should be -358. This makes sense because while it is unlikely, you could have made a really quick, nearly 360’ turn versus a slight 2’ heading difference. Therefore you must infer the heading diff based upon the previous heading diffs.

Make sense?
by Dwayne Dibbley at 2012-10-08 07:47:52
ok to bring it back to powershell land :slight_smile: if found the following excel formula that works but unsure how to powershellize it?

A1:A10 = Raw Data
B1: =IF(A1>180,A1-360,A1) <drag down to B10
C2: =IF(ABS(A1-A2)>180,1,"") <drag down to C10
D10: =IF(SUM(C1:C10)>0,IF(AVERAGE(B1:B10)<0,360+AVERAGE(B1:B10),AVERAGE(B1:B10)),AVERAGE(A1:A10))


excel result with some test data:


A B C
3 3
2 2
1 1
0 0 1
359 -1
358 -2
359 -1 1
0 0
1 1
2 2 0.5 < average bearing from A1:A10


Thanks again
by Dwayne Dibbley at 2012-10-08 08:28:11
if have the following so far but cant work out how to do the IF(ABS(A1-A2)>180,1,"") part as this requires a future number? A2. so i have currently just stuck in IF(ABS(A2-A1)>180,1,"") to show the part, maybe i could run a step behind somehow?

foreach ($Waypoint in $Table)
{
Switch($Waypoint.Course)
{
{($_ -gt 180) } { $LatArray[$i % $BufferSize] = $Waypoint.Course - 360}
default { $LatArray[$i % $BufferSize] = $Waypoint.Course }
}

if ([Math]::ABS($latArray[$i % $BufferSize] - $latArray[$i % $BufferSize - 1]) -gt 180)
{
$flag[$i % $BufferSize] = 1
}

$Output = $LatArray | measure-object -average
"Raw:$($Waypoint.Course) CourseAverage]$Output.Average) flag:$($flag[$i % $BufferSize])"
$i++

}
by Dwayne Dibbley at 2012-10-09 06:58:30
i think i have a working version now for the course that seems to work, might not be the cleanest way but im only a noob :

$Data = Get-Content .\LOG10.txt
$GPRMC = $Data -match '^$GPRMC.+'
$Headers = 'ID','UTCTime','Status','Lat','NSInd','Long','EWInd','Speed','Course','UTCDate','MagVar','MagVarInd','Chksum'
$Table = ConvertFrom-Csv -InputObject $GPRMC -Header $Headers

$BufferSize = 5
$CourseArrayA = New-Object Single($BufferSize)
$CourseArrayB = New-Object Single($BufferSize)
$CourseArrayC = New-Object Single($BufferSize)
$i = 0

$flag = New-Object Single($BufferSize)

foreach ($Waypoint in $Table)
{
$CourseArrayA[$i % $BufferSize] = $Waypoint.Course

Switch($Waypoint.Course)
{
{($_ -gt 180) } { $CourseArrayB[$i % $BufferSize] = $Waypoint.Course - 360}
default { $CourseArrayB[$i % $BufferSize] = $Waypoint.Course }
}

if ($i % $BufferSize -ne 0)
{
if ([Math]::ABS($CourseArrayA[$i % $BufferSize] - $CourseArrayA[$i % $BufferSize - 1]) -gt 180)
{
$CourseArrayC[$i % $BufferSize] = 1
}
else
{
$CourseArrayC[$i % $BufferSize] = 0
}
}
else
{
if ([Math]] - $CourseArrayA[4]) -gt 180)
{
$CourseArrayC[$i % $BufferSize] = 1
}
else
{
$CourseArrayC[$i % $BufferSize] = 0
}
}

$first = $CourseArrayC | measure-object -sum
if ($first.sum -gt 0)
{
$temp = $CourseArrayB | measure-object -average
if ( $temp.average -lt 0 )
{
$result = 360 + $temp.average
}
else
{
$result = $temp.average
}
}
else
{
$temp = $CourseArrayA | measure-object -average
$result = $temp.average
}

$result
$i++

}