Comparing 2 CSV Files and Output Differences

I am having a hard time comparing two CSV files in an intuitive way. I basically have a dump of all GPO settings with three columns: GPO Name, Setting Path, and Setting Value. When I simply compare lines via the “Compare-Object” cmdlet, it does not return what I want:

$file1 = import-csv -Path "C:\Scripts\GPOCompare\GPOExport1.csv" 
$file2 = import-csv -Path "C:\Scripts\GPOCompare\GPOExport2.csv"

Compare-Object $file1 $file2 |  Select-Object -ExpandProperty inputobject

The input files look like this:

GPO Name,Setting Path,Setting Value
GPO1,Path 1,Value 1
GPO1,Path 2,Value 2
GPO1,Path 3,Value 3
GPO1,Path 4,Value 4
GPO2,Path 1,Value 1
GPO2,Path 2,Value 2
GPO2,Path 3,Value 3
GPO2,Path 4,Value 4
GPO2,Path 5,Value 5

I basically want to compare all paths and values for each GPO in two different files. So basically, the logic would be: For GPO1, look at all paths and values and if there are any differences (the corresponding settings may not be on the same lines in the csv files), output to file or add pscustomobject for differences in that GPO to an array and then export array to csv once complete. I’m thinking I need to look at one GPO a time, compare all lines, and output differences in settings and/or values to a new CSV. Can someone please point me in the right direction?