Zoho Banner September 2011

Archive for the ‘Exchange Server’ Category

You know you’re getting old when you come across a Usenet post you wrote almost 20 years ago. I came across this little memento while Googling for a much more recent item. Given the vintage of the post, I must have been referring to Exchange 4.0.  Exchange has come a long way since then, although I do kind of miss X.400.

x400_tony

PS. I’m still waiting for an answer to my question. :-)

It seems that a couple of weeks ago my standalone Exchange Online Protection (EOP) configuration was changed without me being involved. Basically, it looks like my default Accepted Domain was changed from type “Internal Relay” to “Authoritative” without my knowledge or consent.

The first I knew of this was when I noticed my on-premises mail server was no longer receiving email. The current usage is low, so I didn’t notice it for a couple of weeks. After some troubleshooting I pinned the problem down to the fact that the Accepted Domain was showing as “Authoritative”. After changing it back to “Internal Relay” mail started getting delivered to my on-prem server almost immediately.

Accepted Domains

I have no delegated admins for this service, so nobody could have gone rogue on me. I have also checked the admin audit logs and the only entries shown for modifying the Accepted Domains configuration are a) when I originally set it up last September and b) when I changed it back yesterday. Here are a few screenshots to show the evidence.

Firstly the graph below shows when mail stopped being received…

graph

 

…then the audit entries showing when I made modifications to the mail.activedir.org Accepted Domain. It only shows the two entries. The first was when I set up the service last September and the second was when I made the change from “Authoritative” to “Internal Relay”yesterday.

 

EOP_audit2

 

 

EOP_audit1

It looks like I don’t have access to the external admin audit log report. It doesn’t appear in my EAC view (see below), so perhaps it is simply not available to EOP-only subscriptions. This might have been insightful as the the log apparently shows actions performed by datacentre administrators, which is where I believe the change was made.

 

Audit_View

Given the external admin audit log report wasn’t available via the EAC, I thought I would try to invoke it via Powershell. All I got from the output was the changes that I had made in the portal, i.e. no external admin entries.

 

PS C:\> Search-AdminAuditLog -ExternalAccess $true

RunspaceId         : 4e7bfd93-6f40-493b-b294-4f936506f863
ObjectModified     : FFO.extest.microsoft.com/Microsoft Exchange Hosted Organizations/fisheaglelimited2014.onmicrosoft.com/Configuration/mail.activedir.org
CmdletName         : Set-AcceptedDomain
CmdletParameters   : {MatchSubDomains, Identity, DomainType}
ModifiedProperties : {AcceptedDomainFlags, AcceptedDomainType}
Caller             : tony@mail.activedir.org
ExternalAccess     : 
Succeeded          : True
Error              : None
RunDate            : 5/03/2015 1:44:26 a.m.
OriginatingServer  : DB3FFO11WS056 (15.01.0099.000)
Identity           : e7054efb-d9f5-461a-9c85-08d224fd0c3a
IsValid            : True
ObjectState        : New

PS C:\> $now = get-date

PS C:\> $start = $now.AddYears(-1)

PS C:\> Search-AdminAuditLog -ExternalAccess $true -StartDate $start -EndDate $now

RunspaceId         : 4e7bfd93-6f40-493b-b294-4f936506f863
ObjectModified     : FFO.extest.microsoft.com/Microsoft Exchange Hosted Organizations/fisheaglelimited2014.onmicrosoft.com/Transport Settings/FE Outbound
CmdletName         : New-OutboundConnector
CmdletParameters   : {TlsDomain, CloudServicesMailEnabled, TlsSettings, Enabled...}
ModifiedProperties : {ConfigurationUnit, SmartHostType, Id, OrganizationId...}
Caller             : tony@fisheaglelimited2014.onmicrosoft.com
ExternalAccess     : 
Succeeded          : True
Error              : None
RunDate            : 7/09/2014 8:57:44 p.m.
OriginatingServer  : AM1FFO11WS040 (15.00.1010.011)
Identity           : ec85e346-1d12-4ab0-2067-08d198f581a9
IsValid            : True
ObjectState        : New

RunspaceId         : 4e7bfd93-6f40-493b-b294-4f936506f863
ObjectModified     : FFO.extest.microsoft.com/Microsoft Exchange Hosted Organizations/fisheaglelimited2014.onmicrosoft.com/Transport Settings/FE Inbound
CmdletName         : New-InboundConnector
CmdletParameters   : {SenderIPAddresses, CloudServicesMailEnabled, RestrictDomainsToCertificate, Enabled...}
ModifiedProperties : {ConfigurationUnit, Id, OrganizationId, RawName...}
Caller             : tony@mail.activedir.org
ExternalAccess     : 
Succeeded          : True
Error              : None
RunDate            : 8/09/2014 1:17:15 a.m.
OriginatingServer  : AM1FFO11WS002 (15.00.1019.000)
Identity           : 50f7f697-a501-4106-56a9-08d19919c2fb
IsValid            : True
ObjectState        : New

RunspaceId         : 4e7bfd93-6f40-493b-b294-4f936506f863
ObjectModified     : FFO.extest.microsoft.com/Microsoft Exchange Hosted Organizations/fisheaglelimited2014.onmicrosoft.com/Configuration/FE Outbound
CmdletName         : Set-OutboundConnector
CmdletParameters   : {TlsDomain, CloudServicesMailEnabled, Identity, TlsSettings...}
ModifiedProperties : {RecipientDomains, RecipientDomainsEx, SmartHosts}
Caller             : tony@mail.activedir.org
ExternalAccess     : 
Succeeded          : True
Error              : None
RunDate            : 8/09/2014 1:19:30 a.m.
OriginatingServer  : DB3FFO11WS013 (15.00.1019.000)
Identity           : 9f184a42-929c-4a98-54c8-08d1991a134d
IsValid            : True
ObjectState        : New

RunspaceId         : 4e7bfd93-6f40-493b-b294-4f936506f863
ObjectModified     : FFO.extest.microsoft.com/Microsoft Exchange Hosted Organizations/fisheaglelimited2014.onmicrosoft.com/Configuration/mail.activedir.org
CmdletName         : Set-AcceptedDomain
CmdletParameters   : {MatchSubDomains, Identity, DomainType}
ModifiedProperties : {AcceptedDomainFlags, AcceptedDomainType}
Caller             : tony@mail.activedir.org
ExternalAccess     : 
Succeeded          : True
Error              : None
RunDate            : 8/09/2014 1:24:06 a.m.
OriginatingServer  : AM1FFO11WS002 (15.00.1019.000)
Identity           : 55b909e6-abbd-43af-8c21-08d1991ab767
IsValid            : True
ObjectState        : New

RunspaceId         : 4e7bfd93-6f40-493b-b294-4f936506f863
ObjectModified     : FFO.extest.microsoft.com/Microsoft Exchange Hosted Organizations/fisheaglelimited2014.onmicrosoft.com/Configuration/mail.activedir.org
CmdletName         : Set-AcceptedDomain
CmdletParameters   : {MatchSubDomains, Identity, DomainType}
ModifiedProperties : {AcceptedDomainFlags, AcceptedDomainType}
Caller             : tony@mail.activedir.org
ExternalAccess     : 
Succeeded          : True
Error              : None
RunDate            : 5/03/2015 1:44:26 a.m.
OriginatingServer  : DB3FFO11WS056 (15.01.0099.000)
Identity           : e7054efb-d9f5-461a-9c85-08d224fd0c3a
IsValid            : True
ObjectState        : New

 

I’ve opened a support incident with Microsoft about this, so I’ll post a follow-up here when that it resolved.

Anyone else out there experienced something similar?

 

 

 

I recently had a interesting issue where I was seeing the 1121 error below in the Application log on one of my Exchange 2013 servers every minute.

 

Log Name: Application
Source: MSExchange Mailbox Replication
Date: 14/01/2015 2:23:34 p.m.
Event ID: 1121
Task Category: Request
Level: Error
Keywords: Classic
User: N/A
Computer: ex1.contoso.com
Description:
The Microsoft Exchange Mailbox Replication service was unable to process a request due to an unexpected error.
Request GUID: '60b5375b-0f27-4561-a0f2-af428dd3c501'
Database GUID: '43ce3977-2c7f-4c6a-937b-b94966c81122'
Error: An Active Directory Constraint Violation error occurred on dc1.contoso.com. Additional information: The name reference is invalid.
This may be caused by replication latency between Active Directory domain controllers.
Active directory response: 000020B5: AtrErr: DSID-03152CB1, #1:
0: 000020B5: DSID-03152CB1, problem 1005 (CONSTRAINT_ATT_TYPE), data 0, Att 9fc1287c (msExchMailboxMoveSourceMDBLink)
--> A value in the request is invalid..
 

The reference to “msExchMailboxMoveSourceMDBLink” was the only indication the error related to a mailbox move request. Frustratingly, no requests showed up from running Get-MoveRequest.

After some ineffective troubleshooting I eventually found the solution from Exchange MVP Jeff Guillet. The error message he shows in his article is different to mine, but the fix is the same.

I had to run the following command:

Remove-MoveRequest -MoveRequestQueue '43ce3977-2c7f-4c6a-937b-b94966c81122' -MailboxGuid '60b5375b-0f27-4561-a0f2-af428dd3c501'

I still don’t know the cause of the problem, but I have a suspicion it followed a reverse mailbox move from Exchange 2013 (CU6) to Exchange 2007 (SP3 RU13).

Hopefully this helps if you ever come across the same issue.

I came across a weird error the other day while running Exchange Server 2013 CU6 setup with the /PrepareAD switch.  The error (from the Exchange setup logs) was this:

[12/02/2014 01:14:16.0727] [2] [ERROR] Could not find the Exchange Mailbox Administrators Universal Security Group through its well-known GUID 29a962c2-91d6-4ab7-9e06-8728f8f842ea.  Please make sure that Setup /prepareAD has been run.

I sat there scratching my head for a while because what (wtf?) is the Exchange Mailbox Administrators group and what does it have to do with Exchange 2013.  Anyway, after a journey of discovery, here’s what I found out.

For some time, Exchange has used an attribute named otherWellKnownObjects on the Exchange container in the Configuration partition of AD to store information about the location of certain Exchange objects, including the default groups.  One of these entries corresponds to the Organization Management group (as shown below).

attribute values

The strange-looking string next to the DN for the Organization Management group (C262A929D691B74A9E068728F8F842EA) is a Hex representation of a GUID.  To get to the string representation I found a handy function called Convert-OctetStringToGuid.  The result was as follows:

convert hex

The resulting GUID string (29a962c2-91d6-4ab7-9e06-8728f8f842ea) matches that shown in the original error.  So, basically the setup error was saying that it couldn’t find the Organization Management group.  The guff about the Exchange Mailbox Administrators group must be a legacy piece of code referencing an earlier version.  Once I had the new information I had a look to see if the group actually existed (it did).  I then concluded that maybe AD replication hadn’t finished by the time setup got to looking for the group.  I then made the executive decision to run setup with the /PrepareAD switch again.  Lo and behold it finished successfully.

I thought this write-up might help if you find yourself in the same boat.

 

You might come across a slight gotcha with SPF records if are using Exchange Online Protection (EOP) to provide protection for your on-premises mail environment and you don’t need to use EOP to scan your outbound (i.e. on-premises to Internet) messages. If this scenario is of interest to you, read on…

You may be aware that I run a mailing list running on a single server over at ActiveDir.org that provides a forum for disucssions on the topic of Active Directory. I set up Exchange Online Protection (EOP) to cover the mail.activedir.org namespace a little while ago as a lower-cost alternative to my previous anti-malware provider. When you first set up EOP with there are some DNS records that you are asked to configure. In fact, these *appear* to be mandatory based on the wording (see below).

EOP1

Without really thinking too deeply about I went ahead and configured the TXT record in DNS for SPF. With hindsight I shouldn’t have done this as I only wanted the traffic from the Internet to my on-premises mail server to be scanned for malware. I don’t need the SMTP traffic in the other direction to be scanned for malware because the only mail generated on the server is from the mailing list (list server) and as such as been scanned on the way in.

All appeared fine for a month or so and then fellow MVP Brian Arkills queried why he wasn’t receiving any email from the list. After a bit of digging, his infrastructure guy pointed to a problem with my SPF TXT record. Basically, the “-all” part of the record indicates that only the servers matching spf.protection.outlook.com are responsible for sending mail from mail.activedir.org. If a different server tries to deliver mail from that address the receiving MTA should fail the SPF check. Interestingly, most organisations treat this as a soft fail and deliver the email anyway. Not so the University of Washington, which is why Brian wasn’t receiving the list emails.

Once alerted to the problem, I modified the record like this:

v=spf1 a -all

What this says to the receiving MTA is to only pass the SPF check if sending server matches the A record for the namespace (i.e. my mail server).

That all seemed fine but then I looked at the message headers for emails that I was receiving from the list.  I am subscribed to the list with a standard cloud-based O365 email address. The header contained the following:

Received-SPF: Fail (protection.outlook.com: domain of mail.activedir.org does not designate 157.55.234.61 as permitted sender)  receiver=protection.outlook.com; client-ip=157.55.234.61;  helo=emea01-db3-obe.outbound.protection.outlook.com; Authentication-Results: spf=fail (sender IP is 157.55.234.61)  smtp.mailfrom=activedir-owner@mail.activedir.org;

The 157.55.234.61 IP address maps to emea01-db3-obe.outbound.protection.outlook.com (very clearly not my mail server). So what was going on? Why was the sending server showing as one of the outlook.com servers and not my mail server? I don’t know for sure but my best guess is that the O365 cloud servers are also EOP servers (makes sense). The EOP servers *think* they are responsible for the mail.activedir.org namespace and as such will effectively take over as the perceived sending server. The upshot was that I needed to go back to my SPF record and change it to include the EOP servers. It now looks like this:

v=spf1 a include:spf.protection.outlook.com -all

To summarise then:

  • Emails coming from my mail server to non-O365 addresses need to have a SPF TXT record that matches my mail server (via the A record for mail.activedir.org).
  • Emails coming from my mail server to O365 addresses need to have a SPF TXT record that matches the name required by EOP (spf.protection.outlook.com)

More than slightly confusing isn’t it? But then I guess I have a slightly unusual configuration.

For more information on SPF, this web site is a great starting point: http://www.openspf.org/

To be honest I’m only writing this one because I know I’ll get confused in the future and will want to refer to something authoritative. :-)

There are two configurable values when considering how to change the Exchange Server 2013 Offline Address Book (OAB) generation schedule from its default of 1 day.  These are:

  • OABGeneratorWorkCycle
  • OABGeneratorWorkCycleCheckpoint

Both are set to 1 day by default.

The values are not well documented and there are conflicting sources on the Internet that describe how they should be set to modify the generation schedule.  From testing, the key value is the OABGeneratorWorkCycleCheckpoint.  If you set this down to, e.g. 2 hours, the OAB will be generated every two hours as the screenshot below clearly shows:

OAB_WorkCycle

The command to set the value on all mailbox servers is shown below:

Get-MailboxServer | Set-MailboxServer -OABGeneratorWorkCycle 01:00:00 -OABGeneratorWorkCycleCheckpoint 02:00:00

Hopefully, this clears up the confusion for you as well as me :-)

If you have Send As permissions on another recipient object (e.g. mailbox, distribution group) you have the ability to specify that object’s primary SMTP address as your “From” address when sending from Outlook and Outlook Web App (OWA).  How you achieve this in OWA is not immediately obvious, so here are some (hopefully helpful) screenshots.  The method works for both Exchange 2013 on-premises and Office 365 OWA.

1. In the New Mail pane, click on the three dots to expose additional options.  Select “Show From”.

002

 

2. With the From field visible, right-click the email address shown and select “Remove”.

003

3. Click on the drop-down arrow in the From field.  From there select the address you which to send from/send as.

004

4. The address should now appear in the From field.  When you send the email it will send from the address shown.

005

That’s all there is to it.  It’s not rocket science, but then it’s not blindingly obvious either.

 

I upgraded a client’s CAS servers recently from Exchange 2007 Service Pack 2 Update Rollup 5 to Service Pack 3 Update Rollup 13 in preparation for upgrade to Exchange 2013.  Following the restart after the Update Rollup had completed I found I could not log into OWA.  Instead I got a blank page with a URL similar to the one shown below.

https://MyCAS/owa/auth/logon.aspx?url=https://MyCAS/owa&reason=0

I found several suggestions for what might be the problem on the web, but the one that worked for me was to run the UpdateOwa.ps1 Powershell script in the Bin folder under the Exchange installation path.

It was required on both CAS servers, so at least it was consistent :-)

When you install a new version of Exchange or apply a Cumulative Update certain AD attributes are updated to reflect the change.  The updates are made in three different directory partitions (also known as naming contexts): Schema, Configuration and Domain.  The following Microsoft TechNet article is a good reference for the different versions and the corresponding attribute values.

http://technet.microsoft.com/en-us/library/bb125224(v=exchg.150).aspx

You can check the values manually….or you could do it the easy way with Powershell.  Here’s a Powershell sample to give you values across the three partitions (assumes a single domain forest):

 

# Exchange Schema Version
$sc = (Get-ADRootDSE).SchemaNamingContext
$ob = "CN=ms-Exch-Schema-Version-Pt," + $sc
(Get-ADObject $ob -pr rangeUpper).rangeUpper

# Exchange Object Version (forest)
$cc = (Get-ADRootDSE).ConfigurationNamingContext
$fl = "(objectClass=msExchOrganizationContainer)"
(Get-ADObject -LDAPFilter $fl -SearchBase $cc -pr objectVersion).objectVersion

# Exchange Object Version (domain) - assumes single domain forest
$dc = (Get-ADRootDSE).DefaultNamingContext
$ob = "CN=Microsoft Exchange System Objects," + $dc
(Get-ADObject $ob -pr objectVersion).objectVersion

The output will looking something similar to screenshot shown below (showing the values for Exchange Server 2013 CU5):

Schema

I’ve had my Lenovo Ideapad Yoga 13 for a little over a year now. Generally, I’m very happy with it.  It has two internal SSDs, 8GB RAM and an Intel Core i7 processor.  Windows 8.1 runs very nicely on it.  I use the Ideapad for my day-to-day work as well as running test labs in Hyper-V.  Memory is generally my main limitation with Hyper-V, but mostly I can starve the VMs of RAM as performance isn’t a key issue for me for demos and/or testing purposes.  [As an aside, Exchange 2013 is a complete resource hog and won't run nicely unless you give each machine at least 4GB of RAM, which makes running a DAG near impossible for me].   Recently, I noticed that my disk latency (average response time) on the SSD that I run the VMs off was really high (around 11,000ms).  Ok, I was running 3 VMs simultaneously, but still!  So I downloaded AS SSD Benchmark to see how my SSD was performing.  The overall result was 438, which is not great when compared with what others have posted on line with the same SSD.

as-ssd-bench M4-CT256M4SSD3 15.07.2014 8-31-44 a.m.

After some deep thinking (i.e. staring idly into space over a coffee), the idea struck me that Bitlocker might be the culprit.  So I disabled Bitlocker for that drive and tried again. The difference was significant (around 20%) without being remarkable.  Interestingly, the read times before and after were almost identical.  The write times were where the difference was appreciable.

as-ssd-bench M4-CT256M4SSD3 15.07.2014 4-29-01 p.m.

The disk is still performing slowly compared with others online.  I checked my other SSD (a Samsung) and it was also slow, so the conclusion I’ve reached is that there must be some other factor (controller?) causing the slowness.  It would be interesting to hear what others with Ideapads are seeing, or if you have any ideas on how to improve performance.  Windows 8.1 is apparently optimised for SSD use, so I haven’t found any silver bullet for speeding things up.