Throttling – Webbanshee

StalledDueToSource_DiskLatency Mailbox Export

WebBanshee — Sat, 05 Feb 2022 13:29:14 +0000

Hello fellow Exchange Admins, Yesterday I exported a 30GB Mailbox as a .pst file to our SFTP server. At about 90% the MailboxExportRequest stalled with StalledDueToSource_DiskLatency.

No other exports were running from the source server. I even disabled all scheduled tasks that could have an impact on the database where the mailbox is mounted.

However, the export request always switched to: StalledDueToSource_DiskLatency

As a consequence, the export started to loop. It just created a new folder hierarchy and started to export the items again.

Therefore the export .pst was growing much bigger than the initial mailbox size. In other words, this can blow up your storage capacities if you have a large mailbox and leave the export overnight! Not to mention that the export is useless afterward.

In this post, I want to share how I worked around the StalledDueToSource_DiskLatency condition.

The solution requires that you have a DAG in place.

Let’s start at the beginning:

First, I started the Mailbox Export Request using the Priority switch with Emergency.
So the export request is being prioritized over other jobs.

New-MailboxExportRequest -Mailbox MailboxToExport -Name IndividualNameOfTheExportRequest -BatchName BatchName -BadItemLimit 0 -LargeItemLimit 0 -FilePath “\\PathToExportLocation\NameOfExportedMailbox.pst” -Priority Emergency

Once the export was running I monitored the request. The following CMDlet will show all relevant information:

Get-MailboxExportRequest IndividualNameOfTheExportRequest -BatchName BatchName| Get-MailboxExportRequestStatistics -IncludeReport | select Report,MRSServerName,Name,SyncStage,Priority,LastFailure,FailureCode,FailureType,BadItemLimit,BadItemsEncountered,LargeItemLimit,LargeItemsEncountered,OverallDuration,Status,StatusDetail,EstimatedTransferSize,BytesTransferred,EstimatedTransferItemCount,ItemsTransferred,PercentComplete

You can find more useful details on monitoring export requests here:

Monitor Mailbox Exports

The pain started whenever the export reached completion of 100%.
The job was relinquished due to large delays and unfavorable server health on the source server-side.

As can be seen in the image below the export status stayed inProgress.
While SyncStage was CopyingMessages and StatusDetail was StalledDueToSource_DiskLatency:

Therefore the export job was relinquished:

Eventually, the export started all over again. StatusDetail: CreatingFolderHerarchy:

Then PercentComplete was 10% again:

As a result, the size of the export PST grew much bigger than the initial mailbox size.

After all, the solution for me was to switch the mailbox database that hosts the mailbox to another copy within the DAG:

Work around StalledDueToSource_DiskLatency

First, check which mailbox database hosts the mailbox you want to export:

Get-Mailbox -Identity MailboxToExport | select PrimarySmtpAddress, ServerName, Database

ServerName shows the server where StalledDueToSource_DiskLatency originates from.

Next, check to which servers the passive database copies are seeded:

Get-MailboxDatabaseCopyStatus -Identity DatabaseName

The output lists the active mailbox database ( Status: Mounted ), the passive copies ( Status: Healthy ), and the lagged mailbox database copy. ( Status: Healthy with ReplayQueueLenghth greater than the others )

Now switch the active mailbox database with Status: Mounted to one of the passive copies with Status: Healthy:

Note that you don’t switch it to the copy where the ReplayQueueLenghth is greater than on the other copies!

Move-ActiveMailboxDatabase DatabaseName -ActivateOnServer OneOfTheServersFromTheOutputBefore

As has been noted before take care that you don’t try to activate the mailbox database on the server that hosts the lagged database copy!

After the database is switched to another server try to export the mailbox with a new export request.
Given that the new server has no performance or disk issues the export should complete in a normal manner.

One more thing:
After I activated the mailbox database that hosts the mailbox on another server the ServerName did not change when querying the mailbox.

Get-Mailbox -Identity MailboxToExport | select PrimarySmtpAddress, ServerName, Database

The output of the command above will still show the old server under ServerName. But this is just visual. However, you can find a good explanation for that here.

Have a nice day

The post StalledDueToSource_DiskLatency Mailbox Export appeared first on Webbanshee.

Stalled Move Request Exchange 2016

WebBanshee — Wed, 02 May 2018 17:13:14 +0000

If you are reading this you have certainly noticed a Stalled Move Request at times. Exchange 2016 ( we have CU8 in place ) utilizes throttling on move requests as well. This happens when the threshold for certain resources has been reached.

Some Stalled Move Request errors are straightforward and highlight an existing problem. Some on the other hand just make no sense because there seems to be no problem with the mentioned resources.

In our case, we have triggered just one move request at a time and the I/O operations on the disk where the target mailbox database resides were minimal. But we still got a StalledDueToTarget_DiskLatency status and a Stalled Move Request.

While move requests with a status like Stalledduetohigherpriorityjobs continued after a while , others went in status StalledDueToTarget_DiskLatency or Relinquishedwlmstall and stalled for hours just to resume and stall again and kill the move request.

You can find more on status monitoring of move requests here : Move Request Status Monitoring

The thresholds , which cause a move request to stall, and the durations are defined in the MSExchangeMailboxReplication.exe.config file located at :

C:\Program Files\Microsoft\Exchange Server\V15\Bin

You can find some useful input on this here : Social Technet Exchange 2016 Migration Status

At the end of the day the most convenient way for me to make sure that a move request completes was to prioritize each mailbox move request with the -priority switch.

To be honest I never gave high or highest a try , instead of them Emergency was the winner. We still got stalled statuses after triggering move requests with -priority Emergency , but no killer ones.

Avoid a Stalled Move Request with the -priority switch :

New-MoveRequest -Identity “Mailbox.You@WantTo.Move” -TargetDatabase “TargetMailboxDatabaseName” -Priority Emergency

As mentioned before, you will probably still see some kind of stalled statuses on your move requests, but they will not end in a loop of stall and resume.

Have a nice day !

Got a stalled export request? Read more here:
Stalled Mailbox Export

The post Stalled Move Request Exchange 2016 appeared first on Webbanshee.

Back Pressure Insufficient System Resources

WebBanshee — Sun, 26 Feb 2017 10:15:35 +0000

If you notice that your Exchange Server does not accept new mails or delivers them with a delay to user mailboxes, you can verify the symptoms regarding back pressure insufficient system resources in this post :
Verify Back Pressure Exchange 2013 / 2016

After you have determined Back Pressure as the root cause and it is disk space-related ( Event ID 15006 ) it is time to wipe parts of the logs and traces which are eating up your disk space.

Especially if you have more servers in your environment the script below can help you in a very convenient way.

It cleans the logs under the defined folder-paths. If a folder is not found the script shows an error. No worries, what does not exist can not be filled and cleaned.

All defined existing folders will be cleaned out. Feel free to adjust the folder paths for your needs or just comment out non-existing lines with paths to non-existing folders.

#———————————————————————————————————–
# Back Pressure Insufficient System Resources Clean Up Script
# Published on WebBanshee.net
# Written by : init.B
#———————————————————————————————————–

Add-PSSnapin *exchange* -ErrorAction SilentlyContinue

# datetime parameter to measure the CERES logs LastWriteTime against

$dateCERES = (Get-Date).AddHours(-4)

# datetime parameter to measure the DailyPerformanceLogs LastWriteTime against

$dateDPL = (Get-Date).AddDays(-1)

# datetime paramete for inetpub logs

$dateinetpub = (Get-Date).AddDays(-4)

# datetime parameter for all other logs

$dateother = (Get-Date).AddHours(-8)

# all mailboxservers or specified mailboxservers

$mbservers = Get-MailboxServer | Sort-Object name

#$mbservers = “Server01″,”Server02″ ,”Server03”

Write-Host “Deleting the specified logs on the following servers…”

FOREACH ($mbserver in $mbservers)

{

# tracker for human eyes

$mbserver.Name

# get and delete both types of CERES logs

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Diagnostics\ETLTraces” | Where-Object {$_.lastwritetime -lt $dateCERES -and $_.Extension -eq “.etl”} | Remove-Item

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Diagnostics\Logs” | Where-Object {$_.lastwritetime -lt $dateCERES -and $_.Extension -eq “.log”} | Remove-Item

# get and delete DailyPerformanceLogs

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\Diagnostics\DailyPerformanceLogs” | Where-Object {$_.lastwritetime -lt $dateDPL -and $_.Extension -eq “.blg”} | Remove-Item

# get and delete ews and httpproxy logs, except for powershell

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Autodiscover” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Eas” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ecp” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Mapi” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Oab” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Owa” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\OwaCalendar” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\RpcHttp” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\Ews” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

# get and delete activemonitoringtracelogs

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\Monitoring\Monitoring\MSExchangeHMWorker\ActiveMonitoringTraceLogs” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

# get and delete probecache & respondercache logs

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\Monitoring\PersistentState\ProbeCache” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

Get-ChildItem -Path “\\$mbserver\C$\Program Files\Microsoft\Exchange Server\V15\Logging\Monitoring\PersistentState\ResponderCache” | Where-Object {$_.lastwritetime -lt $dateother -and $_.Extension -eq “.log”} | Remove-Item

# get and delete inetpub logfiles

Get-ChildItem -Path “\\$mbserver\C$\inetpub\logs\LogFiles\W3SVC1” | Where-Object {$_.lastwritetime -lt $dateinetpub -and ($_.Extension -eq “.log” -or $_.Extension -eq “.zip”)} | Remove-Item

}

To prevent back pressure insufficient system resources you can set this script as a scheduled task to have the logs cleaned up on a regular basis.

If you haven’t done it already you can move the mail.que file to a location on another drive to free up even more disk space.

Change location of mail.que file Exchange 2013 / 2016

The post Back Pressure Insufficient System Resources appeared first on Webbanshee.

Message submission rate limit exceeded

WebBanshee — Thu, 09 Feb 2017 17:47:16 +0000

A customer or user wants to send bulk emails through a script or automated program. The used script or program connects via SMTP to a mailbox on your Exchange Server.

After a short while the mail submission stops with the following error message :

‘4.4.2 Message submission rate for this client has exceeded the configured limit’

This is caused by the Message Rate Limit Parameter. This one is responsible for the message submission rate from a single source per minute. It can be found on the receive-connector or can be applied on mailbox level through a Throttling Policy. Since not many users use an automated email blaster, we created a new Throttling Policy and applied it only on certain mailboxes. The global message submission rate stayed default.

Create a new Throttling Policy with ThrottlinPolicyScope set to Regular to keep it on mailbox level. The example below raises the Message Rate Limit from 5 ( default ) to 200 :

New-ThrottlingPolicy -Name “Raise Message Submission Rate” -ThrottlingPolicyScope Regular -MessageRateLimit 200

Now apply the new policy to the mailbox you want to raise the Message Rate Limit :

Get-Mailbox -OrganizationalUnit “TenantName” -Identity “MailboxName” | Set-Mailbox -ThrottlingPolicy “Raise Message Submisson Rate”

Check it with :

Get-Mailbox -OrganizationalUnit “TenantName” -Identity “MailboxName” | select ThrottlingPolicy

The output should show the name of the just applied policy.

The users who reported this have not sent any further inquiries regarding this. We take this as a positive feedback. Of course this has been tested before.

For OnPremise Syntax please visit MultiTenant VS. OnPremise in the top navigation bar.

It makes sence to apply this together with a Recipient Rate Limit :

Recipient Rate Limit Throttling Policy Exchange

The post Message submission rate limit exceeded appeared first on Webbanshee.

Verify Back Pressure Exchange 2013 / 2016

WebBanshee — Fri, 20 Jan 2017 15:23:27 +0000

This post describes Back Pressure symptoms. If you can verify Back Pressure as the cause of your problem after reading this you can find more here on how to prevent Back Pressure Insufficient System Resources.

Situation : Mails sent to recipients on your Exchange Server will be delivered with a huge delay or or will not be delivered at all. In the mail-logs of the last hop Message Transfer Agent ( MTA ) you see :

452 4.3.1 Insufficient system resources.

You can verify Back Pressure symptoms as follows …
In the event logs of your Exchange Server you will find one of the following events :

Event ID: 15004
Event log entry for an increase in any resource utilization level

Event ID: 15006
Event log entry for critically low available disk space

Event ID: 15007
Event log entry for critically low available memory

We were sure our server’s C drive [ 100GB ] had still 18 GB free space and this did not change much for a long couple of month ( since we have outsourced the .mail.que file to another drive. ) At this point Back Pressure could be taken into consideration as the culprit. Since Back Pressure requires the Exchange installation drive ( in our case C: ) to have at least 20% free space. That means in this case at least 20GB free space.

Another possibility causing Back Pressure can be the memory used. But see the criteria and thresholds yourself in the output from the command below.

First you need to add the ExchangeSnapIn for Powershell or ISE.

To verify Back Pressure status run :

$bp=Get-ExchangeDiagnosticInfo -Process EdgeTransport -Component ResourceThrottling; $bp.Diagnostics.Components.ResourceThrottling.ResourceTracker.ResourceMeter

You will get an Output ( always from the server where you trigger the cmd ) like :

Resource : SystemMemory
CurrentResourceUse : Low
PreviousResourceUse : Low
PressureTransitions : [PressureTransitions: MediumToHigh=94 HighToMedium=89 LowToMedium=88 MediumToLow=84]
Pressure : 38

Resource : UsedDiskSpace[C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data]
CurrentResourceUse : Low
PreviousResourceUse : Low
PressureTransitions : [PressureTransitions: MediumToHigh=99 HighToMedium=90 LowToMedium=89 MediumToLow=80]
Pressure : 55

This is just an example. The output contains more parameters you can verify. But here you may get an idea what causes the delays.

Regarding disc space Back Pressure starts straight away when the MediumToLow threshold has been reached !

Need to set up an Exchange Server virtual environment? Start here:

Exchange Server 2019 Hyper V Install

The post Verify Back Pressure Exchange 2013 / 2016 appeared first on Webbanshee.

Recipient Rate Limit Throttling Policy Exchange

WebBanshee — Mon, 16 Jan 2017 15:11:10 +0000

To keep the reputation of your Mail Transfer Agents healthy it is a good advise to set a Recipient Rate Limit on your Exchange Servers. With this measure you are able to prevent spamming and the transmission of bulk mails ( generally new-letters ).

This can be achieved through a Throttling Policy where you set the Recipient Rate Limit to a value which fits your customers mail sending practices, but prevents a spam critical amount of outbound mails.

In this example we will set the value to 2000 mails / 24 hours :

Apply the Recipient Rate Limit Exchange Server wide on all mailboxes : ( Exchange 2013 / 2016 )

New-ThrottlingPolicy -Name Limit2000 -RecipientRateLimit 2000 -ThrottlingPolicyScope Organization

Apply the Recipient Rate Limit for a certain organization : ( Hosted Exchange 2010 Multi-Tenant)

Get-ThrottlingPolicy -Organization “TenantName” | New-ThrottlingPolicy -Name Limit2000 -RecipientRateLimit 2000

Apply it on an Exchange 2010 OnPremise :

New-ThrottlingPolicy -Name Limit2000 -RecipientRateLimit 2000

Remember not to alter the Default Throttling Policy since this is the template for all further Throttling Policies you create.

To apply a Throttling Policy on mailbox level please see the following post :

New Throttling Policy Exchange 2013 / 2016

The post Recipient Rate Limit Throttling Policy Exchange appeared first on Webbanshee.

New Throttling Policy Exchange 2013 / 2016

WebBanshee — Sun, 15 Jan 2017 21:06:30 +0000

In an Exchange server 2013 / 2016 Multi-Tenant environment you cannot specify a New Throttling Policy on organization level anymore. ( As it was possible in a Hosted Exchange 2010 environment )
3 scopes can be specified for the new Throttling Policy, where the first scope is the global scope and represents the Out Of The Box default GlobalThrottlingPolicy.

The default GlobalThrottlingPolicy should never be changed or altered. All further Throttling Policies you create are based on this policy and are an exact copy of it. So when you want to implement a new Throttling Policy you have the following two scope choices:

Organization : will be autom. active on all mailboxes within your organization
Regular : will be active on mailbox level and needs to be directly assigned to mailboxes.

We will create now an Exchange wide new Throttling Policy , which raises the concurrent OWA sessions an Outlook Web App user can have against the Exchange Server to 10, using the parameter: -OwaMaxConcurrency. The default value is 5.

This policy , once set up , will apply to all mailboxes in your Exchange Server environment :

New-ThrottlingPolicy -Name “YourPolicyName” -ThrottlingPolicyScope Organization -OwaMaxConcurrency 10

Now we will create the same ThrottlingPolicy on mailbox level, where we limit the concurrent OWA sessions a Web App user can have to 3 :

New-ThrottlingPolicy -Name “YourPolicyNameOnMailboxLevel” -ThrottlingPolicyScope Regular -OwaMaxConcurrency 3

The new Throttling Policy needs to be directly assigned to the mailbox of choice :

Multi Tenant Environment :

Get-Mailbox -OrganizationalUnit “TenantName” -Identity “MailboxName” | Set-Mailbox -ThrottlingPolicy “YourPolicyNameOnMailboxLevel”

OnPremise Environment :

Get-Mailbox -Identity “MailboxName” | Set-Mailbox -ThrottlingPolicy “YourPolicyNameOnMailboxLevel”

The mailbox level part applies to Exchange 2010 /2013 / 2016 . For a hosted Exchange 2010 environment you want to replace -OrganizationalUnit with -Organization !

The post New Throttling Policy Exchange 2013 / 2016 appeared first on Webbanshee.