Open a Socket!

I had a problem in my lab environment yesterday when trying to fix a corrupted search catalog on one of my Exchange 2010 mailbox servers.  I knew the search catalog was corrupted because I was seeing the following events:

Log Name:      Application
Source:        ExchangeStoreDB
Date:          21/06/2010 12:02:03 p.m.
Event ID:      123
Task Category: Database recovery
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MXB02.contoso.com
Description:
At ‘21/06/2010 12:02:01 p.m.’ the Microsoft Exchange Information Store Database ‘DB01′ copy on this server experienced a corrupted search catalog. Consult the event log on the server for other “ExchangeStoreDb” and “MSExchange Search Indexer” events for more specific information about the failure. Reseeding the catalog is recommended via the ‘Update-MailboxDatabaseCopy’ task.

I knew I had a good copy of the database on my other server (MBX01), so I went ahead and issued the following Powershell command:

Update-MailboxDatabaseCopy -Identity “DB01\MBX02″ -CatalogOnly

This generated a nasty looking erorr, which corresponded to the following Application event log entry:

Log Name:      Application
Source:        MSExchange Configuration Cmdlet - Remote Management
Date:          21/06/2010 4:54:08 p.m.
Event ID:      4
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MBX02.contoso.com
Description:
(PID 9940, Thread 21) Task Update-MailboxDatabaseCopy writing error when processing record of index 0. Error: Microsoft.Exchange.Cluster.Replay.CiSeederGenericException: A server-side seed operation has failed. Error: An error occurred while performing the seed operation, which may indicate a problem with the source disk. Error: An error occurred while updating the search catalog files from server ‘MBX01′ to ‘MBX02′. Error: Can’t dismount the search catalog. Error: Microsoft.Exchange.Search.Common.FteCatalogNotFoundException: SearchCatalog.Dismount failed, error 0×80043629 —> System.ComponentModel.Win32Exception: Unknown error (0×80043629)
   — End of inner exception stack trace —
   at Microsoft.Exchange.Cluster.Replay.CiFilesSeederInstance.<>c__DisplayClass5.<SeedThreadProcInternal>b__2(Object , EventArgs )
   at Microsoft.Exchange.Cluster.Replay.CiFilesSeederInstance.RetryCiOperation(EventHandler evt) —> Microsoft.Exchange.Search.Common.FteCatalogNotFoundException: SearchCatalog.Dismount failed, error 0×80043629 —> System.ComponentModel.Win32Exception: Unknown error (0×80043629)
   — End of inner exception stack trace —
   at Microsoft.Exchange.Cluster.Replay.CiFilesSeederInstance.<>c__DisplayClass5.<SeedThreadProcInternal>b__2(Object , EventArgs )
   at Microsoft.Exchange.Cluster.Replay.CiFilesSeederInstance.RetryCiOperation(EventHandler evt)
   — End of inner exception stack trace (Microsoft.Exchange.Search.Common.FteCatalogNotFoundException) —
   at Microsoft.Exchange.Cluster.Replay.CiFilesSeederInstance.SeedThreadProcInternal()
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
   — End of stack trace on server (MBX02.contoso.com) —
   at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
   at Microsoft.Exchange.Cluster.Replay.SeedProgressReporter.GetException()
Event Xml:

After some fruitless retries and even more fruitless Googling, I finally found a Technet article that indicates I should have suspended the database copy before running the Update-MailboxDatabaseCopy command.  Sure enough, once I suspended the database copy I was able to run the command successfully.

Interestingly (and somewhat worryingly) the corrupted search index appeared immediately after an attempt to perform a server switchover.  If I didn’t know better I would think that the switchover generated the corruption.

Hopefully this will help others who run into the same problem.

The other day I had to uninstall Exchange 2010 from a server with the CAS and Hub Transport roles.  It all went well until the point where it was uninstalling the Hub Transport role.  The uninstall then ground to a halt with the following error:

Log Name:      Application
Source:        MSExchange Configuration Cmdlet - Management Console
Date:          20/05/2010 4:11:01 p.m.
Event ID:      4
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      HUBCAS1.contoso.com
Description:
(PID 6920, Thread 31) Task Uninstall-MsiPackage writing error when processing record of index 0. Error: Microsoft.Exchange.Configuration.Tasks.TaskException: Couldn’t open package ‘D:\Program Files\Microsoft\Exchange Server\V14\Mailbox\MSFTE.MSI’. Another version of this product is already installed. Installation of this version cannot continue. To configure or remove the existing version of this product, use Add/Remove Programs on the Control Panel. Error code is 1638. —> System.ComponentModel.Win32Exception: Another version of this product is already installed. Installation of this version cannot continue. To configure or remove the existing version of this product, use Add/Remove Programs on the Control Panel
   — End of inner exception stack trace —
   at Microsoft.Exchange.Management.Deployment.MsiUtility.GetProductCode(String packagePath)
   at Microsoft.Exchange.Management.Deployment.MsiUtility.IsInstalled(String packagePath)
   at Microsoft.Exchange.Management.Deployment.UninstallMsi.InternalValidate()

After scratching my head for a few minutes, I uninstalled Update Rollup 3 for Exchange Server 2010 and then re-tried the uninstall for Exchange.  It went through perfectly. 

I hadn’t realised it was necessary to remove any Update Rollups before uninstalling Exchange, but it it seems it is required in this case.

This is an updated version of the script that I wrote for Exchange Server 2007.  The new version works for both E2007, E2010 as well as environments where both versions coexist.

Ok, so the script is a bit messy and could use some polish - but, hey, it works for me

##########################################################
# Name: GetExchangeBuild.ps1
# Author: Tony Murray
# Version: 2.0
# Date: 22/04/2010
# Comment: PowerShell script to list build info
# for each Exchange Server in the organisation
#
#########################################################    

Add-PSSnapin Microsoft.Exchange.Management.Powershell.Admin -ErrorAction SilentlyContinue
Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010 -ErrorAction SilentlyContinue     

$exsrvs = (get-exchangeserver)     

foreach ($exsrv in $exsrvs)
{
$version = (get-exchangeserver -identity $exsrv).admindisplayversion
$edition = (get-exchangeserver -identity $exsrv).edition
write-host “=====================================================”
write-host “Exchange Server: $exsrv”
write-host $version
write-host “Edition: $edition”
write-host “Installed Update Rollups:”
$baseKey = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey(’LocalMachine’, $exsrv)
$Version8 = "Version 8."
If ($version -match $Version8)
{
$regKey = “SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\
S-1-5-18\Products\461C2B4266EDEF444B864AD6D9E5B613\Patches\”
}
Else # Version is 14
{
$regKey = "SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\
S-1-5-18\Products\AE1D439464EB1B8488741FFA028E291C\Patches\"
}
$baseKey = $baseKey.OpenSubKey($regKey)
$Updates = $baseKey.GetSubKeyNames()
ForEach($Update in $Updates)
{
$fullPath= $regKey + $Update
$UpdateKey = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey(’LocalMachine’, $exsrv)
$UpdateKey = $UpdateKey.OpenSubKey($fullPath)
$values = $UpdateKey.GetValueNames()
ForEach($value in $values)
{
if ($value -eq “DisplayName”)
{Write-host $UpdateKey.GetValue($value)}
}
}
write-host “=====================================================”
}

Watch out for two the lines beginning with $regKey.  They might wrap in the window here but should be on one line.  Note also that Wordpress does something funky with the character codes and you may need to replace the double-quote characters if you copy/paste the code. You can download the file here: getexchangebuildv2ps1.txt

You receive the following message when installing Update Rollup 3 for Exchange Server 2010:

“Setup Wizard for Update Rollup 3 for Exchange Server 2010 (KB981401) ended prematurely because of an error. Your system has not been modified. To install this program at a later time, please run the installation again.”

In my case this turned out to to be a User Account Control (UAC) gotcha.  Basically, I didn’t launch the *.msp file using “Run as Administrator”.  Interestingly, the error is misleading in that it does modify the system.  If you look at the installed updates on the system the Update Rollup appears as being installed.  In other words it is partly installed.

The steps to resolve the issue are:

1. Go to Control Panel, Programs, Programs and Features, Installed Updates.  Highlight the update and then click Uninstall.

2. Open a CMD prompt using right-click “Run as Administrator”.  Browse to the folder where the file is located and launch the file named Exchange2010-KB981401-x64-en.msp.

One of these days I’m really going to get the hang of UAC . Until I do, it would be really nice if the Microsoft product teams could provide more helpful error messages for UAC failures.

The Get-MailboxStatistics cmdlet is great for showing a range of detailed information about the mailboxes in your organisation.  Sometimes you need to pare down the information to just the really useful stuff (such as mailbox size, item count, etc.) and push it to a CSV file.  Here’s one that I find helpful:

get-exchangeserver | where-object {$_.IsMailboxServer -eq $true } | `

Get-MailboxStatistics | Sort-Object TotalItemSize -Descending | `

Select-Object DisplayName,@{label="TotalItemSize(MB)";`

expression={$_.TotalItemSize.Value.ToMB()}},`

ItemCount,ServerName,StorageGroupName,DatabaseName | `

export-csv c:\stats.csv –NoTypeInformation

If you’re creating contact objects in Active Directory the Exchange cmdlets New-MailContact, Set-MailContact and Set-Contact are usually sufficient.  On the other hand I haven’t found a way using these cmdlets to set all the attributes that I might need.  For example, the “description” attribute doesn’t appear to feature anywhere.

Things have obviously changed with the AD Powershell Provider and associated cmdlets in Windows Server 2008 R2, but here’s a script to bulk create contacts  from CSV file if you’re still using Powershell 1.0.

The format of the requried CSV file looks like this:

givenName,sn,displayName,mail,description
Bob,Smith,”Bob Smith”,bob.smith@gmail.com,”External Supplier”
Sue,Jones,”Sue Jones”,sue.jones@hotmail.com,”Hadware Sales”
Graeme,Turner,”Graeme Turner”,graeme.turner@yahoo.com,partner

#########################################################
#
# Name: BulkCreateContacts.ps1
# Author: Tony Murray
# Version: 1.0
# Date: 13/12/2009
# Comment: PowerShell 1.0 script to
# bulk create AD Contact objects from csv file
#
#########################################################

# Set the target OU where the contacts will be created
$ContactOU=[ADSI]“LDAP://ou=Contacts,dc=mycompany,dc=com“

# Find our current working directory
$working = $(Get-Location)

# Specify the folder and CSV file to use
$folder = “C:\util\Powershell\CSV”
Set-Location $folder
$csv = Import-Csv “contacts.csv”

# Parse the CSV file line by line
foreach($line in $csv) {

# Assign variables to each attribute
$givenName = $line.givenName
$sn = $line.sn
$displayName = $line.displayName
$mail = $line.mail
$description = $line.description
$targetAddress = $line.mail

# Go ahead and create the contact object
$Contact = $ContactOU.create(“Contact”,”cn=$displayName”)
# Set the attributes on the contact object
$Contact.Put(“givenName”,$givenName)
$Contact.put(“sn”,$sn)
$Contact.put(“displayName”,$displayName)
$Contact.put(“mail”,$mail)
$Contact.put(“description”,$description)
$Contact.put(”targetAddress”,$targetAddress)
# Commit the changes
$Contact.setinfo()
# Mail-enable the contact (if you need to)
Enable-MailContact -Identity $displayName -ExternalEmailAddress $targetAddress
}
# Go back to the original working directory
Set-Location $working

The other day I applied Update Rollup 1 for Exchange Server 2007 Service Pack 2 in my test lab and wanted to check that the update was successfully applied.  Being something of a Powershell fan I first tried the command shown below:

(get-exchangeserver MyE2K7Server).admindisplayversion

The output showed Version 8.2 (Build 176.2), which is the same as Exchange Server 2007 SP2 without the Update Rollup.  Next I look at the value using the Exchange Management Console (EMC), which showed me the same information.

I even checked the registry under HKLM\SOFTWARE\Microsoft\Exchange\v8.0\<role> and the ConfiguredVersion value shows the same (8.2.176.2).

The only place I noticed the build number had been incremented was in the EMC under Help->About.

After some Googling I found the statement below from Microsoft’s Ananth Ramanathan

Successful installation of an update rollup should not be ascertained by the version number shown in the admin tools. The version number displayed by Exchange Management Console or other administrative mechanisms is obtained from the Exchange Server Object in Active Directory. If we update the Exchange Server Object in Active Directory to show a new version number for every rollup, it would require additional privileges for the account being used to install the rollup. Since many large customers have split level administrative model where the Exchange administrators do not have permissions to the Active Directory we do not update the Active Directory with an updated version number when the rollup is installed.

Another factor is that the rollup is no different than a Hotfix/Update for other products from Microsoft. So as an administrator you should do what you usually do for validating that the Hotfix is correctly installed/ In Windows Server 2003, you will need to go to Add//Remove programs in control panel and make sure the “Show Updates” checkbox is selected at the top. In Windows Server 2008, you will need to go to “View Installed Updates” in the Control Panel.

Ananth’s method is fine you want to look at one Exchange Server, but what if you want to list the build levels for all Exchange servers in your organisation?  Powershell to the rescue!  Here’s a script I put together to show the names, editions and build numbers for each Exchange server in the organisation.

#########################################################
#
# Name: GetExchangeBuild.ps1
# Author: Tony Murray
# Version: 1.0
# Date: 24/11/2009
# Comment: PowerShell script to list build info
# for each Exchange Server in the organisation
#
#########################################################

#Uncomment the line below if running from Powershell outside the EMS
#add-pssnapin Microsoft.Exchange.Management.Powershell.Admin

$exsrvs = (get-exchangeserver)
foreach ($exsrv in $exsrvs)
{
$version = (get-exchangeserver -identity $exsrv).admindisplayversion
$edition = (get-exchangeserver -identity $exsrv).edition
write-host “=====================================================”
write-host “Exchange Server: $exsrv”
write-host $version
write-host “Edition: $edition”
write-host “Installed Update Rollups:”
$baseKey = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey(’LocalMachine’, $exsrv)
$regKey = “SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Products\461C2B4266EDEF444B864AD6D9E5B613\Patches\”
$baseKey = $baseKey.OpenSubKey($regKey)
$Updates = $baseKey.GetSubKeyNames()
ForEach($Update in $Updates)
{
$fullPath= $regKey + $Update
$UpdateKey = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey(’LocalMachine’, $exsrv)
$UpdateKey = $UpdateKey.OpenSubKey($fullPath)
$values = $UpdateKey.GetValueNames()
ForEach($value in $values)
{
if ($value -eq “DisplayName”)
{Write-host $UpdateKey.GetValue($value)}
}
}
write-host “=====================================================”
}

Let me know if you have any problems with the script, or if you can think of ways to improve it.

Tony

The other day I had a customer ask what the recommended maximum database size is for Exchange Server 2007 mailbox stores.  I knew I had seen the information somewhere on-line, so I did some Googling.  Initially, I didn’t have much luck. 

First of all I found the article below, which recommends stores grow no larger than 50GB. On the other hand it also mentions .stm files (no longer used in Exchange 2007), so I wasn’t convinced it is still relevant.

http://technet.microsoft.com/en-us/library/aa995715.aspx

Then I found KB823144, which also recommends a 50GB max, but the article only applies to Exchange 2003.

After that, I found the following Blog entry written by Gerod Serafin.  The article references some Microsoft on-line content that talks about different recommendations, but comes back to the 50GB recommendation.

http://blogs.technet.com/gerod_serafin/archive/2007/06/11/200-gb-limit-on-database-sizes.aspx

Feeling somewhat confused, I put the question to the Exchange MVP brains trust.  I was referred to the on-line Technet article that covers Mailbox Server Storage Design.  This provides the authoritative recommendation for Exchange Server 2007, as quoted below.

We recommend the following maximum database sizes for Exchange 2007:

• Databases hosted on a Mailbox server without continuous replication: 100 GB
• Databases hosted on a Mailbox server with continuous replication and gigabit Ethernet: 200 GB

  Note:

  Large databases may also require newer storage technology for increased bandwidth to accommodate repair scenarios.

  Important:

  The true maximum size for your databases should be dictated by the SLA in place at your organization. Determining the largest size database that can be backed up and restored within the period specified in your organization’s SLA is how you determine the maximum size for your databases.

For me, the key element is the comment regarding the SLA in place within your organisation.  When carrying out a high level design the 100GB/200GB recommended maximums shown above are good starting points, but these may need to be adjusted during your proof of concept phase.

Yesterday I fired up an old Exchange Server 2007 VM lab environment.  The first thing I saw when I started the Exchange Management Console (EMC) was a pop-up message saying, “The following servers in your organization are currently unlicensed”, as shown below.

After clicking OK, I got the another pop-up message saying, “The server ‘<server_name>’ is unlicensed and has exceeded its trial period for licensing.

Fair enough, it had been a long time since I had installed the test environment.  After obtaining a valid license key from MSDN I went back to the EMC to update the product key.  The only problem was I couldn’t find the option anywhere!  I found the following information on Technet:

“Open the Exchange Management Console.

• In the console tree, expand Server Configuration.
• In the result pane, select the server that you want to license.
• In the action pane, under the server name, click Enter Product Key. The Enter Product Key wizard appears.
• On the Enter Product Key page, type the product key for the Exchange server, and then click Enter.”

Even with these instructions I still couldn’t find the option. 

Then it struck me that I was using the 32-bit version of Exchange Server 2007, which clearly can’t be licensed because it is not supported for production use.  Doh!

After some further digging I even found wording in the Help documentation that talks about entering the product key:

“Note: 
This action is available only if you installed the 64-bit version of Exchange 2007″

RTFM anyone?

Because you cannot license a 32-bit version of Exchange Server 2007 your options are to either re-install the trial version or to live with the warning messages.  Interestingly the license warnings are just that - i .e. they do not impede or remove functionality.

I thought I would blog about this to save others from going through the same time-wasting process