It gets confusing using $_ and depending on where you are in the pipeline it could not be what you expect, so first thing is use $file in your first loop to ensure the correct context is used. Next, you were referencing $_ for Get-Content, which is an object, not a path, so updated to .FullName. Last, you are recursing through a directory structure, so you need to specify -File to only process files:
foreach ($file in Get-ChildItem -Path ('{0}\ReplaceTests' -f [environment]::GetFolderPath("Desktop")) -File -Recurse) {
(Get-Content $file.FullName |
foreach { $_ -replace 'FindString', 'ReplaceString' }) |
Set-Content $file.FullName
}