Gathering emails powershell by MAPI / EWS

Hello All,

i’ve done a function to perform few automated task from Outlook, like create folders, scan mails per categories or move them to assigned folders based on a csv file. Unfortunally i’m having some issues while gathering the emails in order to move them to the folders

mailboxes are mapped as shared mailbox and the service account have full access (they’re as Online not cache)
Emails have a category and on base of this category he will lookup on the CSV to create the path and then invoke the expression, this works fine in one of the mailbox where the amount of emails it’s not high, less than 300, however there is other mailbox where the amount of messages it’s higher aprox 600 message UnRead from 1000aprox in whole inbox.

the main issue i’ve found it’s that he’s not able to gather all message information such as Categories, SenderName… which it’s important to let the tool know where to move them. I run the code below :

		$olFolders = "Microsoft.Office.Interop.Outlook.OlDefaultFolders" -as [type]
		$OlClass = "Microsoft.Office.Interop.Outlook.OlObjectClass" -as [type]
		$OlSaveAs = "Microsoft.Office.Interop.Outlook.OlSaveAsType" -as [type]
		$OlBodyFormat = "Microsoft.Office.Interop.Outlook.OlBodyFormat" -as [type]

			$mapi = $outlook.GetNameSpace("Mapi")
			$Accounts = $outlook.Session.Stores | Select-Object displayname, ExchangeStoreType, FilePath
			$Source = $Mapi.Folders[$Mailbox].Folders.Item("Inbox")
			$NotRead = $Source.Items | Where-Object { $_.Unread -eq $False -and $_.FlagIcon -eq "0" }

If i run that he will gather aprox the 50% of the messages and the others are blank. Any idea what could be the reason or possible workarounds?

I was trying to “migrate” the tool to EWS but it’s a bit more complex and i’m not very skilled with it at the moment…

here is excerpt from my EWS script for similar task (message move by subject)

$address = 'test@test.ru'
$TargetFolderName = 'Target Folder'
$Subject = 'TestSubject'

Import-Module "C:\Program Files\Microsoft\Exchange\Web Services\2.2\Microsoft.Exchange.WebServices.dll" 
$EWS = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService 'Exchange2013',([timezoneinfo]::Utc)

$EWS.AutodiscoverUrl($address)

$folderID = new-object Microsoft.Exchange.WebServices.Data.FolderId 'Inbox', $address
$folder = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($EWS, $folderID)
Write-Host "found $($folder.DisplayName) folder"

$folderview = New-Object Microsoft.Exchange.WebServices.Data.FolderView 100
$targetFolder = $null
foreach ($f in $folder.FindFolders($folderview)) {
	if ($f.DisplayName -eq $TargetFolderName) {
		$targetFolder = $f
		break;
	}
}
if ($targetFolder) {
	Write-Host "found $($targetFolder.DisplayName) folder"
}
else {
	$targetFolder = New-Object Microsoft.Exchange.WebServices.Data.Folder $EWS
	$targetFolder.DisplayName = $TargetFolderName
	Write-Host "Create $($targetFolder.DisplayName) folder"
	$targetFolder.Save($folderID)
}

$filter = New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo([Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::Subject, $Subject)
$view = New-Object Microsoft.Exchange.WebServices.Data.ItemView 100
# Так как найденное сообщение мы первым делом двигаем, то всё что берём - всё равно теряется, поэтому просим только ID
$view.PropertySet = New-Object Microsoft.Exchange.WebServices.Data.PropertySet
# А это уже нам понадобится после перемещения
$mailProperties = New-Object Microsoft.Exchange.WebServices.Data.PropertySet (
						[Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::TextBody,
						[Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::Subject,
						[Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::Sender
					)

$size = 0; $view.Offset = 0; $req = 0;
do {
	$req++
	$MailItems = $folder.FindItems($filter, $view)
	if ($view.Offset -eq 0) { Write-Host ('Messages Total: {0}' -f $MailItems.TotalCount) }
	$view.Offset += $MailItems.Items.Count
	foreach ($item in $MailItems.Items) {
		$size += $item.Size
		$item2 = $item.Move($targetFolder.Id)
		$item2.Load($mailProperties)
		# [...]
	}
} while ($MailItems.MoreAvailable)
Write-Host ('Size: {0}, Requests: {1}' -f $size, $req)

Thanks for your reply and script example, really appreciated.

While i was doing test with EWS i found i cannot gather more than 1000 items then i saw the loop (do-while) but i don’t get if doing that he will do the operation for the total amount of items. Maybe this can be done as well for the MAPI? i know EWS it’s faster than mapi but in both cases i know there is the limitation from IIS / Exchange of 1000 items per batch.

My example definitely work with folders which contain more than 1000 items (because of using $view.Offset), but can say nothing about MAPI, sorry

Do you have any suggestion how to create a Multiple Filter?
i was able to create a filter for messages received within specified time window however i don’t manage to filter the messages before the loop with the following code:

				$EWS.UseDefaultCredentials = $True
				$EWS.AutodiscoverUrl($EmailAccount, {$true})

					# Bind your Folder & Create your filter
						$folderID = new-object Microsoft.Exchange.WebServices.Data.FolderId 'Inbox', $EmailAccount
						$folder = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($EWS, $folderID)
						$mailitems = $folder.FindItems($EWSItems)

						$FilterRead = New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo([Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::IsRead, $False)
						$FilterFlag = New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo([Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::Flag, $False)

					# Create Collection and Apply your Filter
						$sfCollection = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+SearchFilterCollection([Microsoft.Exchange.WebServices.Data.LogicalOperator]::Or)
						$sfCollection.add($FilterRead)
						$sfCollection.add($FilterFlag)
						$view = new-object Microsoft.Exchange.WebServices.Data.ItemView($EWSItems)

					# Offset
					$size = 0
					$view.Offset = 0
					$req = 0

					# Loop items
					do {

						$req++
						$MailItems = $folder.FindItems($sfCollection, $view)

....

I get the following error message

"Exception calling “FindItems” with “2” argument(s): “The property can not be used with this type of restriction.”
At line:4 char:1

  • $MailItems = $folder.FindItems($sfCollection, $view)"

do you have any idea how to archive those kind of filters for “UnRead” and “Flag” messages

I’m not too late ?
sorry, I’m tried only filters like this

$filter = New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+SearchFilterCollection 'And'
$filter.Add(( New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+IsGreaterThanOrEqualTo([Microsoft.Exchange.WebServices.Data.ItemSchema]::DateTimeReceived, (Get-Date).AddDays(-7))))
$filter.Add(( New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+ContainsSubstring([Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::Sender,'microsoft.com') ))

letters from microsoft.com for last 7 days

IsRead works for me, but not a Flag

looks like the Flag is the complex property
https://msdn.microsoft.com/en-us/library/microsoft.exchange.webservices.data.flag_members(v=exchg.80).aspx

may be you should use it other way than just equal to ‘$false’

Hello Max,
thanks for the info, indeed i was looking and seems more complex, due the Exchange version too, i think i will try to create a filter by MAPI property and looking what’s the property by MFCMAPI.

however i’ve noticed now another issue, i’ve adapted a bit your code with mine, basically the operation it’s the same however i look for the folder based on a CSV file, the issue i’ve found is that he don’t do the Loop of DO, he only perform the move of the number of items i use for the view as is he’s not getting the MoreItems property

				try {

					# Bind your Folder & Create your filter
						$folderID = new-object Microsoft.Exchange.WebServices.Data.FolderId 'Inbox', $EmailAccount
						$folder = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($EWS, $folderID)
						$MailItems = $folder.FindItems($EWSItems)
						$FilterSender = New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo([Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::IsRead, $True)
						$view = new-object Microsoft.Exchange.WebServices.Data.ItemView($EWSItems)

					# Offset
					$size = 0
					$view.Offset = 0
					$req = 0

					# Loop items
					do {

						$req++
						$MailItems = $folder.FindItems($FilterSender, $view)

						if ($view.Offset -eq 0) {

							Write-Verbose ('Messages Total: {0}' -f $MailItems.TotalCount)

						}
						$view.Offset += $MailItems.Items.Count
						foreach ($item in $MailItems.Items) {

							if ($Item.Categories -ne "") {

								[int]$msgsize = $item.Size / 1MB
								$ItemSize = "{0:N0}" -f $msgsize
								$size += $ItemSize

								if ($EWSLoad -eq $True) {

									$Item.Load()

								}

								$msg = $item | Select-object DateTimeReceived,Categories,@{Name="MsgFromAddress";Expression={$_.From.Address}},
												@{Name="MsgFromUser";Expression={$_.From.Name}}, @{Name="Domain";Expression={$_.From.Address.Split("@")[1]}},
												Subject, displayTo, DisplayCC, isread, Flag, IsFromMe, Size, InternetMessageId

									# Split categories for cases where two or more categories are assigned
									# Array 		|		[string[]]$EmailCat = $msg.Categories.split(",")
									# Whole Array 	|		[String]$EmailCat = $msg.Categories
									[string[]]$EmailCat = $msg.Categories.split(",")

								[String]$MailSender = $Msg.MsgFromUser
								[String]$MailAddressSender = $Msg.MsgFromAddress
								[String]$MailCategory = $EmailCat[0]
								[String]$MailSubject = $Msg.Subject
								[String]$MailReceivedTime = $Msg.DateTimeReceived
								[String]$MailDomain = $Msg.Domain
								$UnRead = $Msg.isread
								[String]$InetMsgID = $Msg.InternetMessageId
								[String]$FromMe = $msg.IsFromMe

								# Checking Category against CSV file
								$TargetMbxFolder = $FolderStructure | Where-Object { $_.Mailbox -eq $Mailbox -and $_.MailCategory -eq $MailCategory }
								
								# Get Folder ID
									# Find and Bind to Folder based on Path
									# Define the path to search should be seperated with \ Bind to the MSGFolder Root

									[String]$FolderPath = ("\inbox\{0}\{1}" -f $CurrentMonthFolder, $TargetMbxFolder.Folder)
									Write-Debug ("Target Message | Category {0} | TargetFolder: {1} | QueryFolder: {4} | Sender: {2} | IsRead: {3}" -f $MailCategory, $TargetMbxFolder.Folder, $MailSender, $UnRead, $FolderPath)

									Try {

										$Movefolderid = new-object Microsoft.Exchange.WebServices.Data.FolderId([Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::MsgFolderRoot,$EmailAccount)
										$tfTargetFolder = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($EWS,$Movefolderid)

										$fldArray = $FolderPath.Split("\")

										#Loop through the Split Array and do a Search for each level of folder
										for ($lint = 1; $lint -lt $fldArray.Length; $lint++) {

											#Perform search based on the displayname of each folder level
											$fvFolderView = new-object Microsoft.Exchange.WebServices.Data.FolderView(1)
											$SfSearchFilter = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo([Microsoft.Exchange.WebServices.Data.FolderSchema]::DisplayName,$fldArray[$lint])
											$findFolderResults = $EWS.FindFolders($tfTargetFolder.Id,$SfSearchFilter,$fvFolderView)

											if ($findFolderResults.TotalCount -gt 0){

												foreach($folder in $findFolderResults.Folders){

													$tfTargetFolder = $folder

												}

											}
											else{

												Write-Verbose "Error Folder Not Found"
												$tfTargetFolder = $null
												$Remark = "Folder not found"

											}

										}
										if($tfTargetFolder -ne $null){

											$TargetFolder = $tfTargetFolder
											[Bool]$FolderFound = $True

										}

									}
									Catch {

										$ErrorMessage = $_.Exception.Message
										Write-Host "EWS FindFolder: " -NoNewline ; Write-Host "Folder $($TargetMbxFolder.Folder) not found | $ErrorMessage" -ForegroundColor red
										$Remark = "Folder not found"

									}

								# Move Item to TargetFolder ID
								if ($FolderFound -eq $True -and $TestMove -eq $False) {

									$item.Move($TargetFolder.ID) | Out-Null
									Write-Verbose ("Message: Moved | Category: {0} | From: {1} | Folder: {2}" -f $MailCategory, $MailSender, $TargetFolder.DisplayName)


								}
								elseif ($TestMove -eq $True) {

									Write-Verbose ("Message: TEST-Moved | Category: {0} | From: {1} | Folder: {2}" -f $MailCategory, $MailSender, $TargetMbxFolder.Folder)

								}

								$MoveMailsCollection += [PSCustomObject]@{

									Mailbox = $Mailbox
									DateOfMove = $(Get-Date -Format dd/MM/yyyy)
									TimeOfMove = $(get-date -Format HH:mm:ss)
									ExecutedBy = $ENV:USERNAME
									MsgID = $InetMsgID
									SenderName = $MailSender
									ReceivedTime = $MailReceivedTime
									Category = $MailCategory
									Folder = $TargetMbxFolder.Folder
									SenderEmail = $MailAddressSender
									MailSubject = $MailSubject
									Remark = $Remark

								}

							}

						}

					} while ($MailItems.MoreAvailable)

					$MoveMailsCollection
					$EDate = (get-date)
					$LapsedTime = $EDate - $SDate
					$LapsedMessage = "{0:00}:{1:00}:{2:00}" -f $LapsedTime.Hours, $LapsedTime.Minutes, $LapsedTime.Seconds
					Write-Host ('{0}: Finished | Messages: {1} | Size: {2} mb | Batches: {3} | Time: {4}' -f $OperationMode, $MoveMailsCollection.Count, $size, $req, $LapsedTime)

				}

since sometimes the number of items can be high is not very convenient just modify the number of items for the view, i try to debug the issue and i think it’s related to the code of the Search the subfolders:

								# Get Folder ID
									# Find and Bind to Folder based on Path
									# Define the path to search should be seperated with \ Bind to the MSGFolder 

									[String]$FolderPath = ("\inbox\{0}\{1}" -f $CurrentMonthFolder, $TargetMbxFolder.Folder)
									Write-Debug ("Target Message | Category {0} | TargetFolder: {1} | QueryFolder: {4} | Sender: {2} | IsRead: {3}" -f $MailCategory, $TargetMbxFolder.Folder, $MailSender, $UnRead, $FolderPath)

									Try {

										$Movefolderid = new-object Microsoft.Exchange.WebServices.Data.FolderId([Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::MsgFolderRoot,$EmailAccount)
										$tfTargetFolder = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($EWS,$Movefolderid)

										$fldArray = $FolderPath.Split("\")

										#Loop through the Split Array and do a Search for each level of folder
										for ($lint = 1; $lint -lt $fldArray.Length; $lint++) {

											#Perform search based on the displayname of each folder level
											$fvFolderView = new-object Microsoft.Exchange.WebServices.Data.FolderView(1)
											$SfSearchFilter = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo([Microsoft.Exchange.WebServices.Data.FolderSchema]::DisplayName,$fldArray[$lint])
											$findFolderResults = $EWS.FindFolders($tfTargetFolder.Id,$SfSearchFilter,$fvFolderView)

											if ($findFolderResults.TotalCount -gt 0){

												foreach($folder in $findFolderResults.Folders){

													$tfTargetFolder = $folder

												}

											}
											else{

												Write-Verbose "Error Folder Not Found"
												$tfTargetFolder = $null
												$Remark = "Folder not found"

											}

										}
										if($tfTargetFolder -ne $null){

											$TargetFolder = $tfTargetFolder
											[Bool]$FolderFound = $True

										}

thanks in advance for your feedback!

You reuse your inbox’s $folder variable inside folder search loop

	foreach($folder in $findFolderResults.Folders){
													$tfTargetFolder = $folder
												}

instead of it if you need the last folder you can do

$tfTargetFolder = $findFolderResults.Folders[-1]

and… if you have your folder layout in csv (based on mailbox and category) you can preload or cache it.
In your current code you do several folderfind requests for one message - it’s a huge overhead

something like

$FolderCache = @{}
#[...]
$FolderKey = $Mailbox+':'+$MailCategory
if ($FolderCache.ContainsKey($FolderKey)) {
   $TargetFolder = $FolderCache[$FolderKey]
}
else {
   $TargetFolder = GetTargetFolder $mailbox $MailCategory #its you code for foder search
   $FolderCache[$FolderKey] = $TargetFolder
}
#MoveMessage

reuse error illustration:

 C:\> $a = 'a'; $a; foreach ($a in 1,2,3 ) { $a }; $a
a
1
2
3
3

damm… i totally didn’t see i was overwriting the $folder variable, thanks Max!

About your solution to avoid do a lookup the folder for each message i was thinking implement similar solution as you suggested, currently a CSV it’s imported with 3 columns, Mailbox;Category;Folder. Folder it’s based on the Subfolder2\Subfolder3.
the whole path it’s: Inbox\Subfolder1 (based on a month variable)\Subfolder2\Subfolder3.

i will give a try at your solution too, i was thinking in similar way to store FolderID and DIsplayname in a hashtable to lookup it first instead of each message. i will give a try and let you know!

thanks!

Hello Max,

I’ve added the table for cache and it’s works like a charm, however I’ve noticed something weird. let say, I’ve 200 Items to move, however at some point i see offset it’s at the 50% of the total items and the MoreAvailable it’s false so it stop, in resume it’s like run the operation several times to move all items… do you have any idea? i’ve been trying changing the number of the items of the view, regarding the Offset and items of offset i use the same settings as you.

let me know if you need some sample of the code maybe, but I’ve some guess that it’s related to View and Offset probably

No, in my practice there were no such cases, but I did not have high loaded mailboxes.

I can assume that this happens when a new message arrives at the moment the processing of the queue is not over.

i forgot to add, i perform two operations gather emails by certain criterias - This works fine and no issues and then move emails where i see issues on the batches, by default i’m using now a lower number of items per batch, 5-10 items per batch.
I’ve run it today with 100 and i’ve verbose to show the message each time he do a new batch:

VERBOSE: Batch 1 | Count: 100 | OffSet: 0 | Total Items: 165 | More Available: True
VERBOSE: Messages Total: 165
VERBOSE: Batch 2 | Count: 0 | OffSet: 100 | Total Items: 65 | More Available: False

The script stopped just when he displayed the 2nd batch, so 65 items where not moved, the command for the verbose is as below:

Write-Verbose ("Batch {0} | Count: {1} | OffSet: {2} | Total Items: {3} | More Available {4}" -f $req, $MailItems.Items.Count, $view.offset, $mailitems.TotalCount, $mailitems.MoreAvailable)

Regarding the loop i’ve followed basically the same commands as yours… any suggestion or idea why this behavior?

My code do

$view.Offset += $FindItems.Items.Count

but in $FindItems there is a NextPageOffset property

may be we should use it ? $view.Offset = $FindItems.NextPageOffset

Anyway, I fear that we close to offtopic here and should find other way to communicate

Hello Max,

Thanks for your information, indeed i was wondering if specifying the Next offset page was the way.
Seems there is no private message option in here… any idea?

I think I know what going on

messages on server:
1
2
3
4
5
you get first 3 messages, move it away and ask server give you messages from 4 to 5
but server have only
4
5
and can’t give you anything from offset 4 becuase now max offset is 1
there is no any cookie, only olain offset…

Hello Max,

thanks for your feedback, indeed what you explained makes sense, also is what i was reading from the documentation on technet.
however was strange if i use lower number of items per view (like 5 / 10) he run few batches, however running 100 / 200 items per view he behave as explained, i will try modify the code and replace $view.Offset by $FindItems.NextPageOffset.

may be there some caching inside exchange take place

I’ve done a test by replacing it, however seems paging is not working in that way, he do a infinite loop with the first offset but is not getting the next offset page :confused: