iOS 6.1 bug cleanup for Exchange 2010 dumpster

by deathbyvegemite at 2013-02-21 11:12:19

Recently, there was a bug in Apple’s iOS 6.1 OS that caused some calendaring issues between iOS 6.1/6.1.1 devices and Exchange 2010, noted by Apple in http://support.apple.com/kb/TS4532 and Microsoft in http://support.microsoft.com/kb/2814847 . Our company is 110k+ users and experienced this bug in a slightly different way than most smaller shops would have. Typically, backlogs in the T-logs were seen in smaller shops because of the proportional processing ability of the size of the environment. We, however, have more than enough processing power and found these calendar items spewing into the dumpster and causing massive growth of the databases because of this. We had dumpsters with millions of calendar version items that were growing in increments of 100’s per second - this equated to more than 4500 users having dumpsters in the 10 - 70 GIG range.

I wrote a few PowerShell tid bits to go through the environment and 'clean shit up" and thought I would share. Note: ANYTHING YOU RUN, YOU DO SO AT YOUR OWN RISK!!! (Sorry, legal note needed…more warnings included later…)

First of all, we put a Throttling policy in place for all users of iOS 6.1/6.1.1 devices, to at least slow things down - which it did - I am intentionally not detailing that here, as this post is about cleanup. (if you want details, send me a message)

First, I ran the following from an elevated PowerShell session with the Exchange Management Snappin loaded:
$Users=@()
Get-ActiveSyncDevice -ResultSize UnLimited | ? {$.DeviceOS -like "iOS6.1*"}| %{Get-Mailbox $.UserDisplayName | select Alias} | %{$users+=$($.Alias)}

(Yes, I know that this will also catch users who have updated to 6.1.2, you will want to review the query if you care to. We chose not to as 6.1.2 wasn’t released yet.)

This gave me a list of all the users that had iOS 6.1 devices.

Because of the size of my environment, I decided to process the devices to determine the worst cases and sort the order appropriately. To do so, I dot sourced the library that I wrote and is at the end of this post and ran the following in the same PowerShell session as the first commands:
$UsersFolderStats = Get-UserFolderStats $users

That can take a long time to run, depending on the size of your environment.
To see the top 50, I ran:
$UsersFolderStats | sort -Unique User | sort RecoverableItemsRoot -Descending | select -first 50 | ft -a

To clean up the dumpsters on the top 50, I then ran the following:
$ToProcess=@()
$UsersFolderStats | sort -Unique User | sort RecoverableItemsRoot -Descending | select user -first 15 |%{$ToProcess+=$($
.User)}
$ToProcess | Start-DumpsterCleanuUp


Now, this runs them one at a time, (I wrote a version that runs in jobs of 20, and I can send that to anyone who wants it, I am not including it here as it’s not as ‘polished’ as the library below and was a late night throw together thing) so it can take a LONG time to run and it’s intentionally REALLY verbose so expect it to write to the screen…a lot… .

NOTES AND WARNINGS: This will make changes to the Managed Folder Work Cycle property of the database of an affected user. It does reset it at the end of the processing, to 30 minutes, you can easily change it to what ever you like in the "end" section of the function "Start-DumpsterCleanUp".

This will also kill off any legitimate Calendar versioning information that may have been in the dumpster - so, do it at your own peril.

This is written as a guide to resolving a VERY specific issue, so please only use if you have had this issue.

Library:
# .SYNOPSIS
# PowerShell Library to aid in the purging of calendar items from dumpster
#
# .DESCRIPTION
# The purpose of the functions in this library is to remove calendar version items from
# the Dumpster Root Folder.
#
# This library is designed to be dot sourced and utilized by other scripts or directly from
# a PowerShell command console
#
# Dot Sourcing this file with the [-Silent] switch will hide the file Definition information
#
# Functions:
# Get-MBXFolderStats -UserName
# Start-DumpsterCleanuUp -UserName
#
# Usage:
# Start-DumpsterCleanuUp -UserName
# or
# $users | Start-DumpsterCleanUp
#
# Created by: Scott Thomas - scott@deathbyvegemite.com
# Copyright (c) 2013 Scott Thomas . All rights reserved .
#
# THIS CODE IS MADE AVAILABLE AS IS, WITHOUT WARRANTY OF ANY KIND. THE ENTIRE RISK
# OF THE USE OR THE RESULTS FROM THE USE OF THIS CODE REMAINS WITH THE USER.
#
# Filenane: DumpsterPurgeCommonLibrary.ps1
#

[CmdletBinding()]
param
(
[switch]$Silent
)

Set-StrictMode -Version Latest

Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010 -ErrorAction SilentlyContinue

function Get-MBXFolderStats
{
[CmdletBinding(SupportsShouldProcess=$true)]
param
(
[Parameter(Mandatory=$true,ValueFromPipeline=$false)][string]$UserName
)
#Error Handling
trap{Write-Host "$($_.Exception.Message) $UserName nUnable to get Mailbox Folder Statistics for user $UserName&quot; ; break}<br><br> #Get Mailbox Folder Statistics<br> $stats = Get-MailboxFolderStatistics $UserName -FolderScope RecoverableItems | select name,FolderAndSubfolderSize,ItemsInFolder<br> $RIRoot = &#40;$stats | ? {$_.Name -like &quot;*Recoverable Items*&quot;}&#41;.ItemsInFolder<br> $RIDels = &#40;$stats | ? {$_.Name -like &quot;*Deletions*&quot;}&#41;.ItemsInFolder<br> $Return = New-Object PSObject<br> $Return | Add-Member NoteProperty RecoverableItemsRoot $RIRoot<br> $Return | Add-Member NoteProperty RecoverableItemsDeletions $RIDels<br> if &#40;$Return -like $null&#41;<br> {<br> return $false<br> }<br> return $Return<br>}<br><br>function Start-DumpsterCleanuUp<br>{<br> [CmdletBinding&#40;SupportsShouldProcess=$true&#41;]<br> param<br> &#40;<br> [Parameter&#40;Mandatory=$true,ValueFromPipeline=$true&#41;]$UserName<br> &#41;<br> begin<br> {<br> Write-Host -f green &quot;Starting...&quot;<br> [Array]$MBXArray=@&#40;&#41;<br> }<br> process<br> {<br> #Error Handling<br> trap{Write-Host &quot;$&#40;$_.Exception.Message&#41; $UserName&quot; ; continue}<br> Write-Host -f Yellow &quot;Processing user: $UserName&quot;<br> $MBXFStats = Get-MBXFolderStats $UserName<br> Write-Host -f Yellow &quot;Processing User: $UserName : Recoverable Items Root count:$&#40;$MBXFStats.RecoverableItemsRoot&#41;&quot;<br> Write-Host -f Yellow &quot;Processing User: $UserName : Recoverable Items Deletions count:$&#40;$MBXFStats.RecoverableItemsDeletions&#41;&quot;<br> if&#40;$MBXFStats -ne $false&#41;<br> {<br> if&#40;$MBXFStats.RecoverableItemsRoot -gt 5&#41;<br> {<br> #set base variables<br> $UserMBX = Get-Mailbox $UserName<br> $MBXStats = Get-Mailbox $UserName | Get-MailboxStatistics<br> $MBXServer = $MBXStats.servername<br> $MBXArray += $&#40;$MBXStats.servername&#41;<br> <br> #process mailbox<br> $UserMBX | Set-Mailbox -CalendarVersionStoreDisabled:$true<br> $SearchDelete = $UserMBX | search-mailbox -SearchDumpsterOnly -DeleteContent -searchquery &quot;Kind:meetings&quot; -Confirm:$false -Force<br> if &#40;$SearchDelete.Success&#41;<br> {<br> Write-Host -f yellow &quot;Processing user: $UserName : Search removed $&#40;$SearchDelete.ResultItemsCount&#41; items&quot;<br> }<br> else<br> {<br> Write-Host -f Red &quot;Processing user: $UserName : Search Failed&quot;<br> }<br> Get-MailboxServer $MBXServer | Set-MailboxServer -ManagedFolderWorkCycle 00:00:05<br> #Start-ManagedFolderAssistant -Identity $UserName<br> $NC=0<br> $MBXFStats = Get-MBXFolderStats $UserName<br> while&#40;$MBXFStats.RecoverableItemsRoot -ge 100&#41;<br> {<br> $MBXFStats = Get-MBXFolderStats $UserName<br> $CC = $MBXFStats.RecoverableItemsRoot<br> if&#40;$NC -eq $CC&#41;<br> {<br> Write-Host -f Yellow &quot;Processing User: $UserName : Starting Managed Folder Assistant &#40;$CC,$NC&#41;&quot;<br> Start-ManagedFolderAssistant -Identity $UserName<br> $MBXFStats = Get-MBXFolderStats $UserName<br> $CC = $MBXFStats.RecoverableItemsRoot<br> }<br> else<br> {<br> Write-Host -f Yellow &quot;Processing User: $UserName : Recoverable Items Root count:$&#40;$MBXFStats.RecoverableItemsRoot&#41;&quot;<br> }<br> sleep 15<br> $NC = $CC<br> }<br> $UserMBX | Set-Mailbox -CalendarVersionStoreDisabled:$false<br> $MBXFStats = Get-MBXFolderStats $UserName<br> Write-Host -f Yellow &quot;Processing User: $UserName : Recoverable Items Root count:$&#40;$MBXFStats.RecoverableItemsRoot&#41;&quot;<br> Write-Host -f Yellow &quot;Processing User: $UserName : Recoverable Items Deletions count:$&#40;$MBXFStats.RecoverableItemsDeletions&#41;&quot;<br> Write-Host &quot;n"
}
}
}
end
{
foreach ($MBXS in $MBXArray)
{
Get-MailboxServer $MBXS | Set-MailboxServer -ManagedFolderWorkCycle 00:30:00
}

Write-Host -f green "Processing Complete…"
}
}

#Hides the libraries definition output on load
if (-not $Silent)
{
help $MyInvocation.mycommand.Definition
}
by albertwt at 2013-04-15 22:55:45
Thanks for sharing the script, does this issue has been addressed in the latest Exchange Serve 2010 Service Pack or still exist ?