Webbanshee

StalledDueToSource_DiskLatency Mailbox Export

WebBanshee — Sat, 05 Feb 2022 13:29:14 +0000

Hello fellow Exchange Admins, Yesterday I exported a 30GB Mailbox as a .pst file to our SFTP server. At about 90% the MailboxExportRequest stalled with StalledDueToSource_DiskLatency.

No other exports were running from the source server. I even disabled all scheduled tasks that could have an impact on the database where the mailbox is mounted.

However, the export request always switched to: StalledDueToSource_DiskLatency

As a consequence, the export started to loop. It just created a new folder hierarchy and started to export the items again.

Therefore the export .pst was growing much bigger than the initial mailbox size. In other words, this can blow up your storage capacities if you have a large mailbox and leave the export overnight! Not to mention that the export is useless afterward.

In this post, I want to share how I worked around the StalledDueToSource_DiskLatency condition.

The solution requires that you have a DAG in place.

Let’s start at the beginning:

First, I started the Mailbox Export Request using the Priority switch with Emergency.
So the export request is being prioritized over other jobs.

New-MailboxExportRequest -Mailbox MailboxToExport -Name IndividualNameOfTheExportRequest -BatchName BatchName -BadItemLimit 0 -LargeItemLimit 0 -FilePath “\\PathToExportLocation\NameOfExportedMailbox.pst” -Priority Emergency

Once the export was running I monitored the request. The following CMDlet will show all relevant information:

Get-MailboxExportRequest IndividualNameOfTheExportRequest -BatchName BatchName| Get-MailboxExportRequestStatistics -IncludeReport | select Report,MRSServerName,Name,SyncStage,Priority,LastFailure,FailureCode,FailureType,BadItemLimit,BadItemsEncountered,LargeItemLimit,LargeItemsEncountered,OverallDuration,Status,StatusDetail,EstimatedTransferSize,BytesTransferred,EstimatedTransferItemCount,ItemsTransferred,PercentComplete

You can find more useful details on monitoring export requests here:

Monitor Mailbox Exports

The pain started whenever the export reached completion of 100%.
The job was relinquished due to large delays and unfavorable server health on the source server-side.

As can be seen in the image below the export status stayed inProgress.
While SyncStage was CopyingMessages and StatusDetail was StalledDueToSource_DiskLatency:

Therefore the export job was relinquished:

Eventually, the export started all over again. StatusDetail: CreatingFolderHerarchy:

Then PercentComplete was 10% again:

As a result, the size of the export PST grew much bigger than the initial mailbox size.

After all, the solution for me was to switch the mailbox database that hosts the mailbox to another copy within the DAG:

Work around StalledDueToSource_DiskLatency

First, check which mailbox database hosts the mailbox you want to export:

Get-Mailbox -Identity MailboxToExport | select PrimarySmtpAddress, ServerName, Database

ServerName shows the server where StalledDueToSource_DiskLatency originates from.

Next, check to which servers the passive database copies are seeded:

Get-MailboxDatabaseCopyStatus -Identity DatabaseName

The output lists the active mailbox database ( Status: Mounted ), the passive copies ( Status: Healthy ), and the lagged mailbox database copy. ( Status: Healthy with ReplayQueueLenghth greater than the others )

Now switch the active mailbox database with Status: Mounted to one of the passive copies with Status: Healthy:

Note that you don’t switch it to the copy where the ReplayQueueLenghth is greater than on the other copies!

Move-ActiveMailboxDatabase DatabaseName -ActivateOnServer OneOfTheServersFromTheOutputBefore

As has been noted before take care that you don’t try to activate the mailbox database on the server that hosts the lagged database copy!

After the database is switched to another server try to export the mailbox with a new export request.
Given that the new server has no performance or disk issues the export should complete in a normal manner.

One more thing:
After I activated the mailbox database that hosts the mailbox on another server the ServerName did not change when querying the mailbox.

Get-Mailbox -Identity MailboxToExport | select PrimarySmtpAddress, ServerName, Database

The output of the command above will still show the old server under ServerName. But this is just visual. However, you can find a good explanation for that here.

Have a nice day

The post StalledDueToSource_DiskLatency Mailbox Export appeared first on Webbanshee.

Export Mailbox New-MailboxExportRequest

WebBanshee — Thu, 06 Jan 2022 14:21:06 +0000

Hello folks, Today I would like to share my experience so far on how to export a mailbox with New-MailboxExportRequest. There are a couple of parameters to query to get useful information about the status of an export request.

As long as everything goes smooth you will probably just need the percentage information. But when the New-MailboxExportRequest stalls or stops with an error you will be happy to have some additional parameters to query at your disposal.

So let’s get to the point straight away:

Start to export a mailbox with New-MailboxExportRequest:

New-MailboxExportRequest -Mailbox MailboxName -Name ExportRequestName -BatchName BatchName -BadItemLimit 0 -LargeItemLimit 0 -FilePath “\\ServerName\ExportLocationFolder\MailboxName.pst” -Priority Emergency

Use the -Priority switch with Emergency to prevent stalling!

After you have started the export request check the progress:

Get-MailboxExportRequest -Name ExportRequestName -BatchName BatchName | Get-MailboxExportRequestStatistics -IncludeReport | select Report, name,Priority, FailureCode, FailureType,DiagnosticInfo,BadItemLimit,BadItemsEncountered,LargeItemLimit,LargeItemsEncountered, OverallDuration, Status, StatusDetail,EstimatedTransferSize,BytesTransferred, EstimatedTransferItemCount, ItemsTransferred,PercentComplete

As a result, you will get the values of the selected parameters.
Adjust the selected parameters if the output gets too long or intransparent.
For example remove -IncludeReport

You can find a description of the selected parameters below.

First, a short info on the -IncludeReport switch:

When this switch is used in accordance with select Report the ouptut looks like this:

The output displays the currently processed items. Depending on the mailbox size the output can get much longer. Below the report, the output of other parameters you have selected is displayed.

New-MailboxExportRequest relevant infos that can be selected:

Report: Displays the whole export request history when used with the -IncludeReport switch. Skipped items based on bad or large items limits are also displayed.
Name: The individual name of the mailbox export request.
Priority: Possible values are Lower, Low, Normal (This is the default value), High, Higher, Highest, Emergency [Use the highest priority to prevent stalling. I usually use Emergency]
FailureCode: Displays the failure code [ not really useful unless you find a list of failure codes for export requests. I couldn’t find one.]
FailureType: Description of the failure that stopped the mailbox export request.
DiagnosticInfo: This needs to be used with the -Diagnostic switch when selected to display diagnostic information. Among others, you will be able to see job pickup results and timestamps.
BadItemLimit: The max. amount of inconsistent items that will be skipped and will not be exported. When this limit is reached the export will fail. A BadItemLimit greater than 50 requires the -AcceptLargeDataLoss switch.
BadItemsEncountered: Number of bad items found during the export.
LargeItemLimit: The max. amount of items in the mailbox that exceeds the allowed message size. The default value is 0. A value higher than 50 requires the -AcceptLargeDataLoss switch. (not recommended)
LargeItemsEcountered: Number of large items found during the export.
OverallDuration: Duration of the mailbox export. Includes times of stalling as well.
Status: The current state of the export. Relevant values are inProgress, Queued, Completed, Failed, Suspended.
StatusDetail: Displays additional information on some states.
EstimatedTransferSize: Estimates and displays the amount of the exported data.
BytesTransferred: Displays the amount of data that has been really transferred. This usually differs from the EstimatedTransferSize.
EstimatedTransferItemCount: Estimates and displays the number of items that will be transferred.
ItemsTransferred: Displays the number of items that have been transferred. Can differ from EstimatedTransferItemCount. ( Among others depending on encountered bad or large items and the associated limits )
PercentComplete: The progress of the export request in percent.

See the standard reference to Get-MailboxExportRequest by Microsoft here.
Find more to Get-MailboxExportRequestStatistics by Microsoft under this link.

Modify parameters of a mailbox export request:

To modify the parameters of an existing mailbox export request you will need to suspend the request first. This can be useful when you retrospectively want to change the BadItemnLimit or want to raise the priority.

Suspend the mailbox export request:

Get-MailboxExportRequest -Name ExportRequestName | Suspend-MailboxExportRequest

In case you want to suspend all mailbox exports of the same batch:

Get-MailboxExportRequest -BatchName BatchName | Suspend-MailboxExportRequest

Now change the parameters [Example: BadItemLimit]

Get-MailboxExportRequest -Name ExportRequestName | Set-MailboxExportRequest -BadItemLimit 25

In order to change all export requests of the same batch:

Get-MailboxExportRequest -BatchName BatchName | Set-MailboxExportRequest -BadItemLimit 25

Resume the mailbox export request:

Get-MailboxExportRequest -Name ExportRequestName | Resume-MailboxExportRequest

In case you want to resume all mailbox exports of the same batch:

Get-MailboxExportRequest -BatchName BatchName | Resume-MailboxExportRequest

Note:
Modifying the BadItemLimit value makes sense when you have started the export with a low BadItemLimit ( for exmaple 5 ). While monitoring the export you assume that there will be more bad item than you have specified.

Another case would be when the export fails because the BadItemLimit threshold has been reached. In that case, you can modify the BadItemLimit after the export failed and resume the export. Suspend is not needed here as it has already failed. To be honest I usually trigger a New-MailboxExportRequest when this happens because I don’t trust the resume. I am not 100 percent sure that the resume picks up where it stopped and I want to prevent duplicated items. But if you trust the resume process more than me give it a try.

Remove-MailboxExportRequest

Here are two arguments to remove mailbox export requests you don’t need anymore:

To export a mailbox with New-MailboxExportRequest with the same request name.
Further, do some housekeeping and keep your environment clean and transparent.

Get-MailboxExportRequest -Name ExportRequestName | Remove-MailboxExportRequest

In case you want to remove all mailbox exports of the same batch:

Get-MailboxExportRequest -BatchName BatchName | Remove-MailboxExportRequest

In order to remove all mailbox export requests:

Get-MailboxExportRequest | Remove-MailboxExportRequest

Final remark:
Get-mailboxExportRequest vs. Get-MailboxExportRequestStatistics:
Whenever an export request got stalled and I monitored the request with Get-MailboxExportRequest it showed me InProgress as the current state.

Only Get-mailboxExportRequest | Get-MailboxExportRequestStatistics showed me the real status.
It seems regarding the state of the export Get-mailboxExportRequest is not as much reliable as Get-MailboxExportRequestStatistics.
For this reason I always use Get-MailboxExportRequestStatistics when I export a mailbox with New-MailboxExportRequest.

Have a nice day!

The post Export Mailbox New-MailboxExportRequest appeared first on Webbanshee.

Zoom Quit Unexpectedly Cannot Start Meeting

WebBanshee — Thu, 25 Nov 2021 08:42:04 +0000

Hello, fellow Zoom users! Yesterday I had a problem where Zoom quit unexpectedly a couple of seconds after I launched the application on a Windows 10 Version 20H2 machine.

Nonetheless, I was able to log in and noticed that the Zoom quit unexpectedly message returned permanently. Starting a meeting was not possible and I could not access the settings.

Updating graphic card drivers had no effect, Zoom kept crashing. The solution was simple once I got some hints from the Zoom log.

I’ll keep it short:

Scenario: You start Zoom and a couple of seconds after you started it the following message appears:

You are able to log in but the Zoom quit unexpectedly message reappears and you are not able to start a meeting or get into the settings. No events are written into the application log.

However, I was sure Zoom logs something so I started to google the location of the logs and found this.

You can find the logs under the path below:

C:\Users\YourUser\AppData\Roaming\Zoom\logs\

Open the crashrpt.xml and focus on the information within the exception tags.
The following lines gave me a hint:

This is the base for the next Google query. Search for:

quartz.dll 0xc0000005

Soon or later you will find this article where the comment of DaedalusWONG will point you to Microsoft Power Toys as the culprit.

A more detailed description of this Power Toys bug can be found here.

Indeed, I have Power Toys version v0.49.1 installed.

Open Power Toys as an admin and disable Video Conference Mute:

Disabling Video Conference Mute solved the problem straight away.

I hope this compiled information helps when you run into this problem and need a fast solution.

Stay healthy!

Have an ASUS router and want to install Entware on it?

The post Zoom Quit Unexpectedly Cannot Start Meeting appeared first on Webbanshee.

ADFS Server Core Token Signing Certificate

WebBanshee — Wed, 27 Oct 2021 10:38:03 +0000

Hi there, we have upgraded our servers. ADFS servers are running on Windows Server 2019 Core now. Therewith the method of the yearly renewal of the Token Signing Certificate has changed to PowerShell only.

I have just finished the renewal of the Token Signing Certificate via Powershell in our test environment. In this post, I will sum up the steps.

In case you want to renew the Token Signing certificate via GUI on an appropriate server see this post: Renew ADFS Token Signing Certificate

Let’s start with a short description of relevant ADFS properties: [Get-ADFSProperties | fl *cert*]

CertficateGenerationThreshold :

Has by default a value of 20 ( days ). That means that 20 days before the current primary ADFS Token Signing Certificate expires, a secondary certificate will be generated ( this will be the new cert after the current one expires ). You can check if the secondary certificate has already been created withe the following commands:

Get-AdfsCertificate -CertificateType Token-Signing
Get-AdfsCertificate -CertificateType Token-Decrypting

When the secondary certificate exists the ouput should list minimum two certificates. Focus on the certificate which has the attribute IsPrimary set to False. Verify that the Not Before: date is correct.

AutoCertificateRollover :

The default value of this attribute should be set to $True and should only be changed to $False for the time when the automatically created secondary certificate will be assigned as the primary ADFS certificate. When you manually renew the Token Signing Certificate this should always be set to $False. Otherwise, the secondary certificate will be promoted as the primary certificate automatically. Web logins to application servers will not be possible until the new certificate has not been introduced on the affected application servers.

CertificatePromotionThreshold :

This attribute is important and should be monitored before the upcoming expiration of the current ADFS Token Signing Certificate. It defines after how many days ( counting from the creation date of the secondary ADFS certificate ) the new certificate will be defined automatically as primary.
If the value of this attribute is set to 15 it means that the secondary certificate will be assigned as primary automatically after 15 days.
Based on the example above the servers should be updated with the thumbprint of the new certificate maximum of 15 days ( better earlier ) after the creation of the secondary ADFS Token Signing Certificate.

CertificateRolloverInterval :

Defines the interval in minutes at which ADFS checks if a new certificate needs to be generated. The default value is 720. If you change the values above accordingly to their meaning and your needs you can lower this value to 5 minutes for instance to generate the secondary certificate if it has not been generated yet. Set it back to default afterward.

Microsoft describes these properties here.

With that being said we are good to go. Log on to your ADFS Server Core.

Query the secondary Token Signing and Decrypting certificates

Get-AdfsCertificate -CertificateType Token-Signing | where {$_.IsPrimary -eq $False}
Get-AdfsCertificate -CertificateType Token-Decrypting | where {$_.IsPrimary -eq $False}

Note the thumbprints. You will need them later.

Export the secondary Token Signing Certificate

Export the certificate to a location you can reach from the application servers. I usually export the cert to a local folder on the ADFS server.

$certRefs=Get-AdfsCertificate -CertificateType Token-Signing | where {$_.IsPrimary -eq $False}
$certBytes=$certRefs[0].Certificate.Export([System.Security.Cryptography.X509Certificates.X509ContentType]::Cert)
[System.IO.File]::WriteAllBytes(“C:\PathToExportFolder\CertName.cer”, $certBytes)

Import the certificate from your ADFS Server Core to all Exchange servers

The certificate will be imported to the Trusted Root Certification Authority of LocalMachine:

Import-Certificate -FilePath \\ADFSServerName\c`$\PathToExportFolderOnADFSServer\CertName.cer -CertStoreLocation Cert:\LocalMachine\Root

You can check that the certificate has been imported successfully via remote MMC from a GUI server:

MMC > Add/Remove Snap-in > Certificates > Computername > This snap-in will manage: Another Computer > Enter the ServerName where you just have imported the certificate.

Navigate to ServerName\Trusted Root Certification Authorities > Certificates and verify that the imported ADFS Signing certificate is there.

Of course you can check it with PowerShell as well:

Set-Location -Path cert:\LocalMachine\root
Get-ChildItem | where {$_.subject -like “*ADFS*”} | fl

Promote the new secondary certificate to primary on ADFS Server Core

After this logins to web services on the involved application servers using ADFS will not be possible until the new certificate has been introduced on the application servers! ( E.g. Exchange OWA )

Set AutoCertificationRollover to False to be able to promote your secondary certificate to primary:

Set-ADFSProperties -AutoCertificateRollover $False

Query the thumbprint of the new Token Signing and Decrypting certificates:

Get-AdfsCertificate -CertificateType Token-Signing | where {$_.IsPrimary -eq $False}
Get-AdfsCertificate -CertificateType Token-Decrypting | where {$_.IsPrimary -eq $False}

Note the thumbprint of both certificates.

Promote both secondary certificates ( Token Signing and Decrypting ) to primary:

Set-AdfsCertificate -IsPrimary -CertificateType “Token-Signing” -Thumbprint ThumbprintGoesHere
Set-AdfsCertificate -IsPrimary -CertificateType “Token-Decrypting” -Thumbprint ThumbprintGoesHere

However, I got an error here stating that I need to add the certificate first. When I tried to add the Token Signing certificate on ADFS Server Core:

Add-AdfsCertificate -CertificateType Token-Signing | where {$_.IsPrimary -eq $False}

The output was a message stating the certificate is already added.

I think you need to wait a little bit after you set AutoCertificateRollover to True. Several tries later it succeeded. To be honest I could not identify the cause – I just assume it could have been the elapsed time after I set AutoCertificateRollover to True.

Verify that the new certificates have the primary status:

Get-AdfsCertificate -CertificateType Token-Signing | where {$_.IsPrimary -eq $True}
Get-AdfsCertificate -CertificateType Token-Decrypting | where {$_.IsPrimary -eq $True}

Switch AutoCertificateRollover back to True:

Set-ADFSProperties -AutoCertificateRollover $True

Check it:

Get-ADFSProperties | fl *cert*

AutoCertificateRollover should be True!

Introduce the new Token Signing Certificate to the Exchange organization

It is enough to fire the following command once from an Exchange Server within your organization:

Set-OrganizationConfig -AdfsSignCertificateThumbprint ADFSTokenSigningCertThumbprint

Verify the AdfsSignCertificateThumbprint:

Get-OrganizationConfig | select adfs*

Restart IIS:

iisreset /noforce

Or Restart IIS remotely:

invoke-command -computername “ServerName” -scriptblock {iisreset /noforce}

Now try to login via web frontend to your application servers. ( try an Outlook on the Web login )

That’s it

Stay healthy!

The post ADFS Server Core Token Signing Certificate appeared first on Webbanshee.

Mailbox Locked Mailbox Move Issue

WebBanshee — Tue, 25 May 2021 07:58:02 +0000

In this post, I would like to share how I got around a Mailbox Locked Mailbox Move issue. After we have successfully moved several hundred mailboxes to their new location some issues occurred with a few mailboxes.

Moves for those mailboxes could not be started with the -priority switch due to the fact that the target location wasn’t on-premise.

As a result, some of those boxes just stalled. Restarting the Mailbox Replication Service on the relevant source server did the trick. The stalled move requests could be reinitiated. Only one mailbox remained.

However, the remaining mailbox was locked. I tried to move the mailbox to another mailbox database but received the same error as with the move-request:

Informational: The request has been paused temporarily because the mailbox is locked. The Microsoft Exchange Mailbox Replication service will attempt to continue processing the request after …

It seemed like the mailbox had a locked flag ( or something similar ) set in the information store of the source server that mounts the mailbox database for this mailbox. If so, I would have to restart the information store or reboot the server to wipe this flag. Not the first choice for a production server.

I never had this kind of mailbox locked mailbox move issue before.
So I started to google and fortunately found this article on dirteam.com.

The article made me confident that it is most probably locked in the information store on the source server. Switching the mounted mailbox database where the locked mailbox resides to another server just did it. Simple as that. Thank you, DirTeam!

Here the short procedure to solve the mailbox locked mailbox move issue:

Check which mailbox database hosts the locked mailbox:
get-mailbox -Identity“MailboxName” | select PrimarySmtpAddress, database

Note the database-name and display servers hosting the copies of this mailbox db:
Get-MailboxDatabaseCopyStatus “DatabaseName”

Mount the mailbox database on another server within the DAG:
Move-ActiveMailboxDatabase “DatabaseName” -ActivateOnServer “ServerHostingCopy”

After the mailbox database was mounted on another server I was able to start and complete the move of the last mailbox successfully.

Have a nice day!

If you run into other issues with move requests this could also be of interest:

Mailbox Moves

The post Mailbox Locked Mailbox Move Issue appeared first on Webbanshee.

Exchange 2019 CU9 with KB5001779

WebBanshee — Thu, 22 Apr 2021 05:46:46 +0000

Microsoft released another security update in April for Exchange 2019 CU9 and Exchange 2016 CU20. Here are our experiences on installing Exchange 2019 CU9 with KB5001779. We also installed Exchange 2016 CU20 with KB5001779 on our Exchange 2016 servers.

If you run a mixed environment consisting of Exchange 2019 and Exchange 2016 servers you may want to check the schema requirements before. You can find information on that in my post from March:

Exchange 2019 CU8 and Exchange 2016 CU19.

First, I want to mention installing the CUs went well and smoothly. Updating from Exchange 2019 CU8 and Exchange 2016 CU19 required less time than upgrading from Exchange 2019 CU5 and Exchange 2016 CU16 to the mentioned CUs. Also installing security update KB5001779 took only about 30-40 minutes.

These were the good things. What are the backlashes?

KB5001779 can break processes initiated by CMDlets you run from a 3rd party or self-developed application against Exchange.

To be more specific quoting Microsoft:

After application of the Exchange Server April security update CMDlets executed against the Exchange Management Console using an invoked runspace might fail with the following error message:
The syntax is not supported by this runspace. This can occur if the runspace is in no-language mode.
This behavior is expected; please change any code using .AddScript() to use .AddCommand() for continued compatibility.

This Microsoft article was quite helpful in terms of what to expect:

April 2021 Exchange Server Security Updates

Tasks before installing Exchange 2019 CU9 with KB5001779 and Exchange 2016 CU19 KB5001779:

Backup your web.config files where you have modified parameters or have added new keys.

As we experienced it still only the web.config file in the OWA directory takes over parameters and keys you have modified before. To be safe I recommend backup every web.config file you have ever touched/modified. So you can easily reapply the changes you have made.
Disable Check for publisher’s certificate revocation in IE.
Make sure you have enough free space on the install drive.
Take a look at the supportability matrix if you need to upgrade your .NET version.
Check Auth methods on service directories like Powershell, OWA, and ECP and note the output.
Switch mailbox databases and put the server in maintenance mode.

Order of install:

Install the regular Windows patches ( if necessary )
Install Exchange 2019 CU9 or Exchange 2016 CU20
Install the April security update KB5001779 from an elevated Command Prompt

Sources:

Exchange 2016 CU20:

- Download Exchange 2016 CU20 here.
- Download Exchange 2016 April security update KB5001779 here.

Exchange 2019 CU9:

Experiences after installing Exchange 2019 CU9 with KB5001779 and Exchange 2016 CU20 KB5001779:

Cumulative Updates CU9 and CU20:

We had mailtips disabled. Mailtips were enabled afterward.
The web.config files in the following directories have been overwritten. Changes needed to be reapplied.

Active Sync – [ We have an increased attachment size ] :

C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\sync
C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\sync

ECP – [ We have additional keys in place. ] :

C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\exchweb\ews

Anyway, I always check the OWA web.config as well:

OWA – [ We have additional keys in place ] :

C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\Owa
Virtual directories for additional OWA and ECP we have in place needed to be recreated.
Problems with KB5001779:

As mentioned the security update breaks CMDlets you run from an external application against Exchange. For example provisioning mailboxes. Although we have read the Microsoft statements and have tested it we ran into this issue.

Since building a new release for our front-end application took a little bit more time than a change window grants I uninstalled KB5001779 on that Exchange server that the application uses for provisioning. Our development team has released a Hot-Fix since.

Here is how to get rid of KB5001779:

Since our Exchange 2019 servers are Core Servers I could not do it via GUI – Installed Updates. It would have been possible to do it via our management server and the Windows Admin Center. In the end, the local way was more sympathetic to me and I have uninstalled the patch via the uninstall string in the registry. You can find the string here:

HKEY_LOCAL_MACHINE\SOFTWARE\Micorsoft\Windows\CurrrentVersion\Uninstall\Patch

Copy the value of the uninstall string and paste it into an elevated command prompt, press enter and follow the uninstall wizard. The process took about 30-40 minutes. If the ISO of the Exchange CU isn’t mounted anymore then you will be asked to insert the Exchange Server source disk. Browse to the required file and continue. Restart the server after the patch has been uninstalled.

In case you need information on other CUs take a look into the CU Archives

… or find a general description here: CU Install Exchange

Stay safe!

The post Exchange 2019 CU9 with KB5001779 appeared first on Webbanshee.

Exchange 2019 CU8 and Exchange 2016 CU19

WebBanshee — Thu, 25 Mar 2021 08:12:09 +0000

Hello fellow Exchange Admins! I know I am late on this. This post is for the records to have all CUs we installed covered. Anyway, we started to install Exchange 2019 CU8 and Exchange 2016 CU19 on the 4th of march straight after the Microsoft out of band session regarding Hafnium.

Altogether our environment consists of Exchange 2016 and Exchange 2019 servers. Therefore we needed to apply the highest schema prior to installing Exchange 2019 CU8 and Exchange 2016 CU19 in this mixed environment.

Accordingly, we ran /prepareschema,/preparead,/preparedomain from an Exchange 2019 server. That was the first server in our environment we upgraded.

In order to determine your current schema version use the commands below:

Forest Range Upper:
Get-ADObject “CN=ms-Exch-Schema-Version-Pt,$((Get-ADRootDSE).schemaNamingContext)” -Property Rangeupper

Forest Object Version:
Get-ADObject (dir “AD:\CN=Microsoft Exchange,CN=Services,$((Get-ADRootDSE).configurationNamingContext)”).DistinguishedName -Property objectVersion

Domain Object Version:
Get-ADObject “CN=Microsoft Exchange System Objects,$((Get-ADRootDSE).defaultNamingContext)” -Property objectVersion

After you have determined your current schema versions check the minimum schema version required to install Exchange 2019 CU8 or Exchange 2016 CU19 from this handy list on eightwone.com.

Tasks before installing Exchange 2019 CU8 and Exchange 2016 CU19:

Backup your web.config files where you have modified parameters or have added new keys.

As we experienced it still only the web.config file in the OWA directory takes over parameters and keys you have modified before. To be safe I recommend backup every web.config file you have ever touched/modified. So you can easily reapply the changes you have made.
Disable Check for publisher’s certificate revocation in IE.
Make sure you have enough free space on the install drive.
Take a look at the supportability matrix if you need to upgrade your .NET version.
Check Auth methods on service directories like Powershell, OWA, and ECP and note the output.
Switch mailbox databases and put the server in maintenance mode.

Order of install:

Install the regular Windows patches ( if necessary )
Install Exchange 2019 CU8 or Exchange 2016 CU19
Install the Hafnium patch KB5000871 from an elevated Command Prompt

Sources:

Exchange 2016 CU19:

Download Exchange 2016 CU19 here.
Download Exchange 2016 Hafnium patch KB5000871 here.

Exchange 2019 CU8:

Download Exchange 2019 CU8 here. [Can only be downloaded via volume license center.]
Download Exchange 2019 Hafnium patch KB5000871 here..

Experiences after Exchange 2019 CU8 and Exchange 2016 CU19 has been installed:

We had mailtips disabled. Mailtips were enabled afterward.
The web.config files in the following directories have been overwritten. Changes needed to be reapplied.

Active Sync – [ We have an increased attachment size ] :

C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\sync
C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\sync

ECP – [ We have additional keys in place. ] :

C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\exchweb\ews

Anyway, I always check the OWA web.config as well:

OWA – [ We have additional keys in place ] :

C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\Owa
Virtual directories for additional OWA and ECP we have in place needed to be recreated.

Problems with Exchange 2019 CU8 and Exchange 2016 CU19:

As mentioned above we started to upgrade our Exchange 2019 servers first. After the first server was upgraded customers reported that delegated users lost their access rights on shared mailboxes. This was a minor issue since only a few customers were affected and could be solved by reapplying those access rights.

Another issue that started after we had the CU in place was time-outs from a 3rd party archiving software. Nothing fatally critical but from time to time some mailboxes could not be archived because the process ran into a time out. On the next run, those mailboxes were archived but new ones ran into a time-out.

However, in the event log of the patched servers, we could find the event IDs Microsoft mentions in this article. Microsoft states that upgrading to Exchange 2019 CU9 will solve the problem.
As soon as we upgrade to Exchange 2019 CU9 I will update this post.

In case you need information on other CUs take a look into the CU Archives

… or find a general description here: CU Install Exchange

Stay safe!

The post Exchange 2019 CU8 and Exchange 2016 CU19 appeared first on Webbanshee.

Mailbox Audit Logging – Enable and Search Logs

WebBanshee — Fri, 12 Mar 2021 07:21:35 +0000

Did you ever receive inquiries about strange mailbox behavior like users state that mails have been deleted without any interaction of the mailbox owner? In cases like this, you are happy to enable Mailbox Audit Logging on the affected mailbox.

This post is about how to enable Mailbox Audit Logging on a certain mailbox and configure the auditing parameters to provide the most relevant insights. Let’s get to the point straight away:

Enable Mailbox Audit Logging on a mailbox

Check the audit status on a certain mailbox:

get-mailbox -Identity MailboxName | select PrimarySmtpAddress, *audit* | fl

In the following examples, I will use intern@testlab.local as -Identity
The output shows that Mailbox Audit Logging is not enabled on this mailbox:

Enable Mailbox Audit Logging:

get-mailbox -Identity MailboxName | Set-Mailbox -AuditEnabled $True

When you check the status again using the first command it will show AuditEnabled as True now.
Operations audited by default are shown as well:

If your output is truncated with ellipses (…) change the $FormatEnumerationLimit value.

Define operations that should be audited:

Depending on what circumstances make a mailbox audit logging necessary you can change the predefined audit operations to more relevant ones. Find a list of actions logged by mailbox audit logging here.

Usually, I completely change the actions for AuditOwner. AuditOwner = MailboxOwner
To do so use the following command:

Set-Mailbox MailboxName -AuditOwner “Create, SoftDelete, HardDelete, Update, Move, MoveToDeletedItems,MailboxLogin”

Adjust the logged actions according to the list by Microsoft mentioned above.
Use -AuditAdmin or -AuditDelegate instead of -AuditOwner to change logged operations for Admin or Delegates.

Search Mailbox Audit Logs

Define your query:

The Search-MailboxAuditLog CMDlet seems only to work out of an Exchange Management Shell. Out of an ISE loaded with an Exchange PS-Snapin, I always get an error.

Undoubtedly the filter for date ranges in connection with the Search-MailboxAuditLog CMDlet is somewhat imprecise. If you want to query a range from 3 days ago until today for instance I recommend setting the end date of the query to the date of tomorrow. Otherwise, it can happen that no results will be shown for the current day or even the day before.

Search-MailboxAuditLog -Identity MailboName -StartDate “MM/DD/YYYY” -EndDate “MM/DD/YYYY” -LogonTypes Owner -ShowDetails | select lastaccessed, operation, logontype ,logonuserdisplayname,folderpathname,ClientIPAddress,ClientInfoString,ClientMachineName,ClientProcessName,MailboxOwnerUPN,DelegateUserDisplayName,MailboxResolvedOwnerName,SourceItemSubjectslist | sort lastaccessed |ft -autosize

Some notes to the query above:

-StartDate/-EndDate: The date format can vary depending on the regional settings of your system!
-LogonTypes: Narrow down your query to logon types you are interested in.

Selectors:

Operation: Displays the logged action.
ClientIpAddress: Shows the IP address of the client respectively the source network outgoing IP address.
ClientInfoString: Shows client connection types like RPC, ActiveSync, OWA
ClientProcessName: For example OUTLOOK.exe
SourceItemSubjectslist: The subject of mails where an action has been logged with.

I recommend to use only relevant selectors when searching an audit log.
Displaying all selectors can make the output confusing or will not fit in at all.

Stay safe folks!

The post Mailbox Audit Logging – Enable and Search Logs appeared first on Webbanshee.

Add Secondary IP Address Windows Server

WebBanshee — Mon, 07 Dec 2020 11:22:19 +0000

If you want to add a secondary IP Address without using the Change Adapter Options in the GUI ( especially on Windows Core servers ) you could use sconfig or simply add the IP address with PowerShell.

In my case, I wanted to add a secondary IP address to a certain interface on some Windows Server Core servers. Unfortunately, an Invalid Index error was thrown when I chose the appropriate interface under sconfig – Network Settings:

However, I faced this error a couple of months ago already when I wanted to Disable IPv6 in my test lab.

Because this time it was production servers I didn’t want to change any settings while adding secondary IP addresses to the servers. The only option was to add a secondary IP address via PowerShell.

… and here we go:

Add Secondary IP Address to Network Interface:

List Network Interfaces with name and Interface Index

Get-NetAdapter

Display existent IPs on chosen Network Interface

To list IPv4 and IPv6 IP addresses:
Get-NetAdapter -ifIndex ifIndex Number | Get-NetIPAddress | select IPAddress | ftTo display only IPv4 addresses:
Get-NetAdapter -ifIndex ifIndex Number | Get-NetIPAddress | where {$_.AddressFamily –eq “IPv4”} | select IPAddress

Add new IP address

New-NetIPAddress –IPAddress “IpAddress goes here” –PrefixLength 24 –InterfaceIndex ifIndex Number

The command adds the IP address to the ActiveStore and the Persistent Store.
You can read more about the PolicyStore values here.

Verify that add a secondary IP address was successful

Get-NetAdapter -ifIndex ifIndex Number | Get-NetIPAddress | where {$_.AddressFamily –eq “IPv4”} | select IPAddress

Thats all. You don’t need to specify a default Gateway since this has most likely be done already with the initial IP address.

Remove IP Address from Network Interface:

List Network Interfaces with name and Interface Index

Get-NetAdapter

Display existent IP addresses that can be removed on chosen Network Interface

We focus here only on IPv4 addresses. See point above to display IPv6 as well.
Get-NetAdapter -ifIndex ifIndex Number | Get-NetIPAddress | where {$_.AddressFamily –eq “IPv4”} | select IPAddress

Copy IP address to remove from the output.

Remove IP address

Insert copied IP address in the command below:
Get-NetIPAddress –IPAddress IPAddressToRemove –InterfaceIndex ifIndex Number | Remove-NetIPAddressYou will need to confirm it for both PolicyStores.
Otherwise you could use the command with -Confirm:$false

add secondary IP address verify

Get-NetAdapter -ifIndex ifIndex Number | Get-NetIPAddress | where {$_.AddressFamily –eq “IPv4”} | select IPAddress

I have compiled the above steps in a script to make it more convenient:

Script to Add/Remove IP Address

The post Add Secondary IP Address Windows Server appeared first on Webbanshee.

Remote PowerShell Session to Server Core

WebBanshee — Tue, 22 Sep 2020 14:59:13 +0000

Hello Folks, this is just a simple post on how to open and enter a Remote PowerShell Session from one server to another. A lot of tasks on different servers can be accomplished in a convenient way from one server.

For instance if you want to Change the Keyboard Layout on the login screen of a new Server Core installation.

We will use the cmdlets below to open, enter, exit, and remove a Remote PowerShell session.

Remote PowerShell Prerequisites:

To receive PowerShell remote commands PS-Remoting must be enabled on the computer. PS-Remoting is enabled by default. Further the WinRM Service needs to be started.

If you want to go for sure firstly check the PS-Remoting state of the remote computer with:

Test-WSMan -ComputerName NameOfRemoteComputer

Afterward, verify the state of the WinRM service on the remote computer that it is running:

Get-Service WinRM -ComputerName NameOfRemoteComputer | Select MachineName,Name,Status

Remote PowerShell Session Cmdlets:

Get-PSSession – [Shows current sessions]
New-PSSession – [Create a new persistent connection to the target host]
Enter-PSSession – [Enter a session as interactive session]
Exit-PSSession – [Exit session. The session will still be alive in the background]
Disconnect-PSSession – [Disconnect from the session]
Connect-PSSession – [connect to a disconnected session]
Remove-PSSession – [Remove an earlier created persistent session]

Ok, here we go …

Create a Remote PowerShell Session with New-PSSession:

New-PSSession -Name NameOfSession -ComputerName NameOfRemoteComputer -Credential Domain\Username.

In case you don’t want to give the remote PowerShell session a name it will be named WinRM*

Check the newly created session with Get-PSSession:

Get-PSSession

The output will show the state of the session:
I gave my session the name “ServerCore”. It connects to an Exchange 2019 server running on Server 2019 Core with hostname “exchange2019”

If you have more sessions you can go for the computer name:
Get-PSSession -ComputerName NameOfRemoteComputer

… or explicit for the session name:
Get-PSSession -Name NameOfSession

… and also for the Id:
Get-PSSession -Name ID

Enter a session directly as an interactive session with Enter-PSSession:

Enter-PSSession -ComputerName NameOfRemoteComputer
Enter-PSSession -name NameOfSession.

As a result, PowerShell switches to the entered interactive session:

You can perform tasks on the target server via PowerShell from the localhost within this session now.
Enter-PSSession does not require a session that has been created with New-PSSession before.

Find more on how to run Remote Commands from a Remote PowerShell Session here.

Exit an interactive session with Exit-PSSession:

Within the remote PowerShell type:
Exit-PSSession

Instead of Exit-PSSession, you can just type Exit as well. It will have the same effect.

Thereafter, the session remains open but you have left the interactive session into the localhost’s PowerShell:

Disconnect a session with Disconnect-PSSession:

You can disconnect a remote PowerShell session which has been created with New-PSSession.
But you can’t disconnect an interactive session that has been initiated with Enter-PSSession.

To disconnect a session run the command below from your localhost’s PowerShell:

Disconnect-PSSession -name NameOfSession

…or disconnect it based on the ID:
Disconnect-PSSession -Id ID

Further, you can disconnect all remote session for a certain remote host based on ComputerName:
Disconnect-PSSession -Id ComputerName

Following you get an output like this:

You can later reconnect by using the Connect-PSSession cmdlet.

Reconnect a disconnected PowerShell session:

As shown below you can reconnect an existing session that has been created with New-PSSession:

Connect-PSSession -name NameOfSession… or you connect it with its ID:
Connect-PSSession -Id ID… as well as with ComputerName:
Connect-PSSession -Id Computername

Remote PowerShell Session to Exchange Server

To connect with a Remote PowerShell Session to Exchange use:

$UserCred = Get-Credential

$ExSession = New-PSSession -ConfigurationName Microsoft.Exchange –Name ExchangeSession -ConnectionUri “http://RemoteExchangeFQDN/powershell” -Credential $UserCred -Authentication Kerberos

Import-PSSession $ExSession

You can verify the session with name ExchangeSession you just openend with Get-PSSession:

The Exchange CMDLets will be available now.

The same can be done from an Exchange Management Shell to connect through a Remote PowerShell session to another Exchange Server:

Open an Exchange Management Shell

If you are on a Server Core type LaunchEMS into the command window.

Use the CMDlets from above

That’s all.

To display the output of a PowerShell command completely without ellipses take a look here:

Expand PowerShell Output

The post Remote PowerShell Session to Server Core appeared first on Webbanshee.

Webbanshee

StalledDueToSource_DiskLatency Mailbox Export

Work around StalledDueToSource_DiskLatency

First, check which mailbox database hosts the mailbox you want to export:

Next, check to which servers the passive database copies are seeded:

Now switch the active mailbox database with Status: Mounted to one of the passive copies with Status: Healthy:

Export Mailbox New-MailboxExportRequest

Start to export a mailbox with New-MailboxExportRequest:

After you have started the export request check the progress:

First, a short info on the -IncludeReport switch:

New-MailboxExportRequest relevant infos that can be selected:

Modify parameters of a mailbox export request:

Suspend the mailbox export request:

Now change the parameters [Example: BadItemLimit]

Resume the mailbox export request:

Remove-MailboxExportRequest

Zoom Quit Unexpectedly Cannot Start Meeting

ADFS Server Core Token Signing Certificate

CertficateGenerationThreshold :

AutoCertificateRollover :

CertificatePromotionThreshold :

CertificateRolloverInterval :

Query the secondary Token Signing and Decrypting certificates

Export the secondary Token Signing Certificate

Import the certificate from your ADFS Server Core to all Exchange servers

Promote the new secondary certificate to primary on ADFS Server Core

Introduce the new Token Signing Certificate to the Exchange organization

Mailbox Locked Mailbox Move Issue

Here the short procedure to solve the mailbox locked mailbox move issue:

Exchange 2019 CU9 with KB5001779

Tasks before installing Exchange 2019 CU9 with KB5001779 and Exchange 2016 CU19 KB5001779:

Backup your web.config files where you have modified parameters or have added new keys.

Order of install:

Sources:

Exchange 2016 CU20:

Exchange 2019 CU9:

Experiences after installing Exchange 2019 CU9 with KB5001779 and Exchange 2016 CU20 KB5001779:

Active Sync – [ We have an increased attachment size ] :

ECP – [ We have additional keys in place. ] :

OWA – [ We have additional keys in place ] :

Problems with KB5001779:

Here is how to get rid of KB5001779:

Exchange 2019 CU8 and Exchange 2016 CU19

Tasks before installing Exchange 2019 CU8 and Exchange 2016 CU19:

Backup your web.config files where you have modified parameters or have added new keys.

Order of install:

Sources:

Exchange 2016 CU19:

Exchange 2019 CU8:

Experiences after Exchange 2019 CU8 and Exchange 2016 CU19 has been installed:

Active Sync – [ We have an increased attachment size ] :

ECP – [ We have additional keys in place. ] :

OWA – [ We have additional keys in place ] :

Problems with Exchange 2019 CU8 and Exchange 2016 CU19:

Mailbox Audit Logging – Enable and Search Logs

Enable Mailbox Audit Logging on a mailbox

Check the audit status on a certain mailbox:

Enable Mailbox Audit Logging:

Define operations that should be audited:

Search Mailbox Audit Logs

Define your query:

Some notes to the query above:

Selectors:

Add Secondary IP Address Windows Server

Add Secondary IP Address to Network Interface:

List Network Interfaces with name and Interface Index

Display existent IPs on chosen Network Interface

Add new IP address

Verify that add a secondary IP address was successful

Remove IP Address from Network Interface:

List Network Interfaces with name and Interface Index

Display existent IP addresses that can be removed on chosen Network Interface

Remove IP address

add secondary IP address verify

Remote PowerShell Session to Server Core

Remote PowerShell Prerequisites:

Remote PowerShell Session Cmdlets:

Create a Remote PowerShell Session with New-PSSession:

Check the newly created session with Get-PSSession:

Enter a session directly as an interactive session with Enter-PSSession: