Flipping Bits

Background Database Maintenance in Exchange 2010

Database size is a critical aspect to consider in Exchange 2010 designs where DAG is in play. This is regardless of the disk backend you choose. JBOD, DAS RAID, and SAN designs all need planning where small passive databases are in play. Since Microsoft offers no configurability of BDM on passive databases, there is nothing you can do but be aware of and plan for the workload (aside from going with a standalone configuration).

Many Exchange administrators are used to having small databases (100-200GB) so that ESE maintenance tasks and restore SLAs are easier to address.  I find that this can trip up otherwise solid storage architectures.  BDM can lead to problems, primarily because admins aren’t necessarily aware that BDM schedules that they set via EMC or powershell applies only to the active copy of the database.

Microsoft calls background database maintenance “online database scanning” or “database checksumming” or “background database maintenance”.  It can be associated with page zeroing.  It’s a googlicious challenge – and a tagging nightmare for bloggers. 

Let’s see what Technet says about BDM (aka “checksumming”):

Background database maintenance I/O is sequential database file I/O associated with checksumming both active and passive database copies. Background database maintenance has the following characteristics:

• On active databases, it can be configured to run either 24 × 7 or during the online maintenance window. Background database maintenance (Checksum) runs against passive database copies 24 × 7. For more information, see "Online Database Scanning" in the New Exchange Core Store Functionality topic.
• Reads approximately 5 MB per second for each actively scanning database (both active and passive copies). The I/O is 100 percent sequential, so the storage subsystem can process the I/Os efficiently.
• Stops scanning the database if the checksum pass completes in less than 24 hours.
• Issues a warning event if the scan doesn't complete within three days (not configurable).

Let’s reiterate interesting bit:

Background database maintenance (Checksum) runs against passive database copies 24 × 7.

Now let’s look at what another technet page says about this:

Exchange scans the database no more than once per day. This read I/O is 100 percent sequential (which makes it easy on the disk) and equates to a scanning rate of about 5 megabytes (MB)/sec on most systems.

The important thing to remember is that when you set a maintenance schedule for a database as described on this technet page, the setting applies only to the active copy of the database.

Now, how do you plan for this workload?  Frequently, you don’t have to worry about it at all.  Let’s say you have 5000 users with 2GB mailboxes on a server.  That’s 10 TB of mailbox data.  If you have 2 TB mailbox databases, you have about 5 databases, or about 25 MB/s in BDM workload.  Not a problem.

However, if you limit your databases to 100GB, that can present a problem.  That 5000 users translates to 100 databases, each of which can launch a 5MB/s read workload (or more) at any given time.  Aggregated over a whole single mailbox server, 500MB/s is nothing to sneeze at, especially if it’s mixed with user workload on the same disk. 

What you can do to limit the impact of BDM:

1. Use larger databases (1-2TB in size).  Although it will increase the amount of time required to maintain the databases, remember that with a DAG configuration, it’s often more expedient to reseed than it is to do ESE maintenance. 
2. If you are concerned about restore times, use an array that can act as a VSS provider and store your first-tier backups in snapshots, clones, or CDP bookmarks.  In these scenarios, restore times are largely uniform whether you have a 1TB database or a 100GB database (log replay notwithstanding).
3. If after considering hardware-assisted VSS, you STILL can’t leverage larger databases, confer with your storage vendor to architect for what will turn out to be large block reads at unpredictable rates and unpredictable intervals.  To the best of your ability, isolate the passive from active databases so that BDM doesn’t impact user mailboxes.
4. Ask Microsoft why passive database maintenance is not configurable.  As it is today, a completely passive Exchange server can generate over 500MB/s in read bandwidth.  Let’s be crystal clear about this: 500+ MB/s is data warehouse territory, and it’s absurd to think that this can occur with no active users on it, and no backups running against it.  It makes little sense, especially in configurations where the silent corruption they’re looking for is detected by other techniques.

Now, I have a couple questions I haven’t been able to get an answer to:

1. What happens to the schedule when checksum pass completes in less than 24 hours?  Does it start again in 24 hours?  Is that from the start of the prior run, or from the end of the prior run?  An authoritative answer would be helpful for those who have to plan for this workload.
2. What factors can make the scanning run faster or slower than “most” systems?  Does the system issue a 256KB every XX ms?  Does that depend on the processor clock rate?  Can we thus expect the read rate to increase with clock rate?  The difference between 5-8 MB/s doesn’t really matter on a single database, but it really does matter when you’re talking about hundreds or thousands of databases on a single disk array.

Best Practices for Windows Mounts Points

Mount points – they’re understandably popular.  And although they’ve been around for quite a while, some people have questions around their implementation.  Yes, they’re really easy to set up, but you should follow some guidelines so you can take leverage advanced technologies down the road.

For the uninitiated, here’s an quick overview of mount points:

Traditionally, a windows volume (disk drive, LUN, etc) had to be mounted to a drive letter for Windows to access it.  This results in a limitation of 24 hard drives that can be mounted to a system.  To overcome this limitation, Microsoft introduced the ability to mount a drive on an empty directory within any NTFS filesystem.  You can try it yourself next time you format a USB stick.  It’s quite neat – now you can have a virtually unlimited number of drives attached to your system.  You know, like UNIX.  .  It also makes it easier to find stuff.

Here’s an example:  Let’s say I have a database with 2 data files, and 1 transaction log, and I want to keep them on separate drives for recovery and performance purposes.  Instead of mounting them to letters g:\, h:\ and i:\, I can do this:

• g:\dbname\dbfile01
• g:\dbname\dbfile02
• g:\dbname\tlog

Where dbfile01, dbfile02, and tlog are all empty directories on which a drive is mounted (mount point).  The directory structure is clear to anyone who looks at it, and if I need to add more drives to my database I just create a directory in dbname, and put a drive on it.  I try not to put the mount points on my system drive (c:\), although I’m allowed to.  The reason is that the mount point looks just like a directory – you have to know that it’s a mount point.  When I have it on another drive letter, it’s clear that there’s another drive there. 

So what’s a nested mount point?  As the name implies, it’s where you have a mount point within a mount point.  In our case, the dbname directory could be a mount point.

Are nested mount points categorically a bad idea?  Absolutely not – in fact it’s a pretty common practice.  The key is that there shouldn’t be unique data higher in the directory structure than a mount point.  For example, this is a bad idea:

• g:\dbname\dbfile
• g:\dbname\dbfile\tlog

Where both dbfile and tlog are both mount points.  First, it wouldn’t occur to another DBA that tlogs is actually a different drive.  It also has implications for backup and recovery if you’re trying to leverage a volume-based backup and recovery system.  The reason for this becomes clear when you start playing through a recovery scenario.  Let’s say I want to restore my database, but keep my transaction log around for replay.  VSS and the SQL Virtual Device Interface (VDI) backup and restore at the disk level rather than the file level.  So I’ve completely replaced not just g:\dbname\dbfile, but also g:\dbname\dbfile\tlog, where the transaction log lives.

I haven’t necessarily lost any data – I can just go find the tlog drive, mount it up, and continue my recovery.  But it clearly poses problems if I want to automate the process.

So this causes all sorts of confusion around whether nested mount points are supported.  In the case of Replication Manager, the answer is yes, nested mount points are supported, as long the volumes in the application set are not nested within each other.  To give examples, this is supported:

• g:\dbname\dbfile
• g:\dbname\tlog

Where dbname, dbfile, and tlog are all mount points, but only dbfile and tlog are part of the application set being protected.

On the other hand, this is not supported:

• g:\dbname\dbfile
• g:\dbname\dbfile\tlog

Where dbfile and tlog are mount points and both part of the application set being protected.

If you have any lessons learned about mount points you think are worth sharing, post a comment below!

I/O Density and Exchange: A short history

Turns out that some people are dubious that Exchange 2010 and 7.2k drives are a good fit.   So I’m sometimes asked to justify spending less on storage for Exchange (a seemingly odd, but ultimately honest position to take).

Up until a short while ago, I just trotted out Microsoft’s graphs regarding the IO per “heavy use” mailbox they see in production.  It shows a 10x decrease in IO per mailbox between Exchange 2003 and Exchange 2010.

This can be unconvincing to some of the folks who fought Exchange 5.5, 2000, and 2003 in the trenches, where it was common knowledge that not only did you put Exchange on the fastest storage available, you often had to allocate more disks than you needed for capacity to meet the performance demands of the application.  A 10x reduction in IO workload doesn’t convincingly take you from “put it on the fastest storage you can afford” to “put it on the slowest storage you can buy”.

The story is actually two-fold – at the same time that IO per mailbox was being driven down, the size of the mailboxes were being driven up.  Since storage sizing is and always has been a balance between performance (how quickly you need to move data on and off the drives) and capacity (how much data you need to store on the drive), IO density (or IO per GB) is far more relevant than IO per mailbox.  In short, we don’t just look at how many IOs Exchange is driving per mailbox, we have to factor in how big those mailboxes are as well.  Typical mailboxes were only 50MB-100MB during the Exchange 2003 days.  Nowadays they’re between 10 and 100 times that.  When we do factor in mailbox sizes, it turns out that IO density isn’t 10 times lower with Exchange 2010, it’s orders of magnitude lower in most modern Exchange configurations.

I’ve come up with a graph to illustrate this notion. 

So while we saw “only” a 10x decrease in the IO per mailbox, we saw over a 400x decrease in IO per gigabyte.

Now let’s take a look at the IO density that spinning disks have offered throughout the years.

Clearly these are simple examples – the mailbox IO was far more variable in the 32-bit versions of Exchange, and the IO/GB for the disk drives doesn’t include RAID overheads, there’s no free space factored in and so forth.  But when we interpolate the IO/GB driven by mailboxes and the IO/GB provided by disks, it matches up pretty well with the typical disk topologies I recommend:

In the Exchange 5.5 to 2003 days, every IO counted, and you had allocate more disks to the Exchange workload than you needed for capacity.  Things got a lot better in Exchange 2007, where you could place your average mailboxes on either smaller 10k FC drives or larger 15k FC drives.  With Exchange 2010 and large mailboxes, the sweet spot for the typical deployment is somewhere between 1TB and 2TB drives.

Now this isn’t to say that 2TB or 1TB drives are appropriate for all Exchange deployments.  A large number of factors go into sizing storage for Exchange.  But if you’ve done your homework and factored in all the variables, and it looks like the slower drives are the way to go, then you should trust your sizing and save your organization some money.

Option 4: Third Party Replication (Or: How Stella Got Her Single Copy Cluster Back)

This is the fifth and final post in a series about the various options to achieve HA and DR with Exchange 2010.  In the first, I broke the DAG into its basic components (Active Manager and DAG replication).  In the second, I gave a quick overview of Native DAG.  In the third, I covered a hybrid approach that combined DAG replication and Active Manager for local HA, and array/SAN based replication for remote site recovery.  In the fourth, I described an option that deploys Exchange in a standalone configuration and leverages a hypervisor to achieve local high availability.

This one will cover Exchange’s Third Party Replication.  This one definitely has a lot of cool factor in it.  It actually leverages DAG (in the form of Active Manager) with array or SAN-based replication technology.  You get all the automatic failover, live patching, Exchange-aware coolness of DAG, zero data loss, and you only have to deploy one copy of the data at each site.

Although synchronous operation is possible with both Options 2 and 3, this is the only option shown that combines synchronous replication with automatic failover.

This option will use a plugin from EMC to coordinate the replication engine with Active Manager. This would be either Replication Enabler for Exchange 2010 (free!), or AutoStart 5.3 with the Exchange 2010 module. This option can also use a virtualization platform like Hyper-V or VMware, but a hypervisor is not necessary to leverage the benefits of this option.

Here are the cost factors:

• Storage: 2
• Network: 2

Architects and managers will typically consider this option when:

• Lossless, automatic failover is required
• Minimal hardware footprint is desired
• There is minimal latency between the sites
• A hardware VSS protection scheme is available for rapid recovery in the event of database corruption
• Live patching is required

Option 3: Virtualized Host Clustering

This is the fourth in a series of posts about the various options to achieve HA and DR with Exchange 2010.  In the first, I broke the DAG into its basic components (Active Manager and DAG replication).  In the second, I gave a quick overview of Native DAG.  In the third, I covered a hybrid approach that combined DAG replication and Active Manager for local HA, and array/SAN based replication for remote site recovery.

I call this option “Virtualized Host Clustering”, because like the Virtualized Local DAG option, this options leverages a hypervisor, but it leverages the HA capabilities of the hypervisor instead.  The Exchange mailbox role is deployed in a VM as a standalone server.  This is by far the least expensive option from all perspectives: acquisition, operation, complexity, footprint, and power.

As you can see, there are some clear cost benefits to this option. You are replicating only one copy of the database, and since the HA function is handled by the hypervisor, a second copy of the database is unnecessary. It’s also the most flexible option – the full suite of workload management tools provided by the hypervisor can be used, and we add a fourth potential replication engine – VPLEX.

I suppose it’s also worth noting that one does not necessarily need a virtualization layer to accomplish this. With the right operational recovery plan to replace the server and restore from backup, three nines (99.9% availability) could easily be achieved without any HA facility whatsoever (either at the application or hypervisor layer).

This is by far the least expensive option from all perspectives: acquisition, operation, complexity, footprint, and power.

Here are the cost factors broken down:

• Storage: 2
• Network: 2 (SRDF, MirrorView), .5 (RecoverPoint)

Administrators and managers will typically choose this option when want to:

• Minimize complexity of the deployment
• Use advanced virtualization features such as Live Migration/Vmotion, DRS, etc
• Achieve consistency with other line of business applications
• Control failover with scripts or tools like VMware Site Recovery Manager
• Control their RPO from zero data loss to minutes
• Have multiple recovery points at each site
• Control bandwidth utilized by replication
• Meet failover requirements not achievable with native Exchange’s Best Copy Selection

This solution is not without its drawbacks however.  Here are a couple of things to consider:

• This is the only option that where the administrator does not have the ability to do non-disruptive patching.  However, one should consider that boot times are pretty quick on virtual machines.  It’s very possible to achieve four 9’s (99.99% availability) with this solution despite the lack of live patching, and reboots can be scheduled for non-peak hours.
• Since only one copy of the data is available at each site, a rapid recovery mechanism for logical and physical failure modes is well advised.  This is usually achieved through hardware based snapshots or bookmarks at minimal cost.  It’s usually a good idea to have a rapid recovery scheme outside of the context of the application anyway, for a variety of reasons.

Option 2: Virtualized Local DAG

This is the third in a series of posts about the various options to achieve HA and DR with Exchange 2010.  In the first, I broke the DAG into its basic components (Active Manager and DAG replication).  In the second, I gave a quick overview of Native DAG.  In this post, I’m going to cover a pretty popular option for folks who’ve already made the decision to virtualize their Exchange environment.

I call it “Virtualized Local DAG” because it uses DAG replication and Active Manager for local HA, but third party technologies for remote replication and failover.

This configuration utilizes a hypervisor such as Hyper-V or VMware’s. A two member DAG group is created, and then both copies are replicated from one site to the other. It’s important to note that because HA is dealt with by Exchange natively, any HA or workload migration features offered by the hypervisor (such as VMHA, DRS, Live Migration and VMotion) should be disabled for the mailbox VMs (other Exchange roles can utilize those features).

As for the network cost, it’s going to be more expensive operationally – you’re replicating two copies of the databases and two copies of the logs, and the storage cost is identical to Native DAG Replication.  However, this can be mitigated through the use of interesting compression and data reduction techniques made available by products like RecoverPoint.

• Storage: 4
• Network: 4 (SRDF, MirrorView), 1 (RecoverPoint)

Why would anyone choose this option? Well first, the network cost isn’t much more than Native DAG replication when you need to make sure you can reseed your databases in a reasonable amount of time. And if you use a replication appliance like RecoverPoint, you can end up using even less bandwidth than Native DAG replication, owing to the very good compression and data reduction an appliance like that offers.

Basically, for customers who've decided to virtualize Exchange, and already have working replication technologies for other applications, this option offers a lot of flexibility, integration with their data center strategies, and costs nothing in terms of storage footprint when compared to traditional DAG.

Ultimately, people may choose this option when there’s a need for or concern about:

• Uptime while patching
• Simplified and coordinated failover of all Exchange roles and services
• The ability of the WAN to absorb re-seeding operations that will occur more frequently with log-based replication
• Better control of failover operations than what native Exchange’s Best Copy Selection can provide
• Full site failover of multiple applications (such as provided by PowerShell scripting with Hyper-V or VMware Site Recovery Manager)
• Consistency with other line of business applications
• Controllable RPO
• Synchronous replication (loss-less cross-site failover)
• Multiple recovery points are desired at each site
• Controllable bandwidth utilization
• Compliance and inclusion in a DR plan that includes other applications
• Business requirements that can’t be met by DAG replication or Active Manager

S#!t Happens (Or: What we can learn from the latest gmail outage)

So here's evidently what happened:

Some time around February 27, gmail was affected (mid-upgrade) by a bug that effectively deleted the mail data associated with about 40,000 email accounts.  Now, Google maintains multiple copies of users' data, so this bug affected all the available copies of the data for these users.  Google had the foresight to backup their data rather than relying on data replication as its sole protection against data loss, but that backup data resides on tape, which clearly takes time to restore.  Just to give you an idea of how much time, you need an idea of the scale of the data loss.  If each of those users had 5GB of data in their mailboxes, the restore operation requires about 200TB of data - not unmanageable, but clearly something that would take on the order of days to weeks to restore unless something really very cool is used.  One of the interesting aspects of the restore process is that users report having no access to the email services while their data is being restored.  An Exchange administrator would have the ability to spin up some dial tone databases and use something like recover-mailbox, recovery storage groups or a more robust tool like Ontrack PowerControls to merge data from the backup sets back into the dial tone databases.

Now, this is not a schadenfreude post.  I have a lot of respect for Google, especially around how they've transformed messaging, and provided consumers with a very viable and attractive alternative to what was a pretty miserable corner of IT when it was first introduced in 2004 (my, how time flies, huh?). They've delivered a remarkably reliable infrastructure for a massive number of users at an incredible price.

However, as a technologist, I'd like to look at what happened, as well as the users' reactions to get an idea of how I can architect messaging systems so that when stuff inevitably hits the fan, the impact can be minimized.

1. Avoid backups at your own risk.  It's tempting, especially with three, four, or five copies, to think to yourself "Well, how many copies do I need before I don't need to back up?"  The fact is that all of those copies are in a single failure domain.  As my friend and colleague Jim Cordes says, "This will work, up to the point where it won't."  In this case, a storage bug (likely associated with Google Filesystem) created data loss.  But it could as easily have been an application bug, administrator error, or a security breach.  In this case, the data also resides outside of the failure domain (on tape).  It's generally advisable that critical data be available outside the context of the application.
2. Users don't just care about service availability - they care about their data too.  Many people live in their email accounts.  Whether we like it or not, their email account is where they keep their most important data.  So make sure your SLAs (either internal or with a service provider) cover data availability and not just service availability.
3. Users don't just care about service and data availability - they care about metadata too. Complaints about the loss of starred emails and labels abound.  This shouldn't surpise us.  If people live in their email accounts, then they'll organize it.  Think of it like a filing cabinet.  If your "backup" of your filing cabinet entails copying everything and putting in all in a fireproof canister, then "restoring" those files to a usable state where you can actually find something is going to be a problem.  This could be a problem for folks who use a compliance archive as a last-ditch resort for data restore.
4. Set distinct SLAs for service and data/metadata availability.  "Distinct" doesn't mean "different" in this context.  With a robust email solution you can get service availability up quickly and cheaply after a disaster.  Getting the actual data back is the longer pole in the tent, and where the bulk of investment is required.  If you tier your data through the use of archives, cost can be mitgated by assigning different SLAs for the active and archive data.
5. Make sure your backup/restore solution meet SLAs for data/metadata availability.  As we've seen, even the best run organizations running top notch software can experience data loss, even when multiple copies of the data are deployed.  If the copy of the data outside of the application failure domain takes 100 hours to restore, then that's the SLA you can sign.  If a disaster requires that you restore many terabytes of data and you have a data availability SLA of under a day, then it's advisable to look at hardware-based snapshots or bookmarks as a solution.
6. Fast backups ≠ Fast restores.  There are many solutions out there that help people meet aggressive backup windows (incremental forever with synthetic fulls are widely available, for example).  Administrators and managers are well-advised to examine the restore speeds of these solutions.  Basically, if the solution calls for the full dataset to be moved from one place to another in order for it to be used, then you need to examine the restore speed.

 

Dissecting Database Availability Groups

I wanted to post a good picture of dissection at the beginning of this post, and it brought me back to middle school biology.  I was in eighth or ninth grade.  We are the World was a chart-topper, and the Asian Tiger Mosquito (awesome name for species, eh?) managed to migrate to Houston.  The important thing is that I wore an onion on my belt (which was the style of the time). 

Well, no, I guess that wasn't the most important thing.  The really important thing is that I couldn't find a picture of a real frog dissection that I want to post on this particular blog.  So I'll post this instead:

   

Onto what I'd really like to get at with this post:

DAG is not the only way to achieve high availability or remote site disaster recovery with Exchange 2010.  The reality is that there are quite a few options for replication and HA, and they run the gamut of requirements, from synchronous replication with automatic, lossless failover, to asynchronous replication with selectable recovery points, and from a single copy of the database(s) at each sites to whatever you choose.

So over the next few posts, I'll be going over the options that administrators and managers can consider when deploying Exchange 2010.  Most importantly, I'll be covering the factors that should be included in that consideration.

Before I start, I need to conceptually take Database Availability Groups (or DAG) apart, show a little bit about how the technology works, and define the terminology associated with it.

Most Exchange administrators have a good idea of what DAG is, and know enough about how it works to implement it.  But many administrators don't know that there are two components to DAG.  As with any true remote disaster recovery technology, there are two mechanisms:

1. There's a technology to replicate the data, known as the replication engine.  In native Exchange 2010, we'll call this "DAG Replication." It consists of asynchronous block-level replication of the transaction logs over an IP network between the members of a DAG.
2. There's a technique or technology that manages which copy of the data is "active."  A "technique" would be something a human does to decide which copy is active.  A "technology" is an automated process that determines which copy is active.  In Exchange 2010, this is a technology (in the form of a role) called Active Manager that runs on ALL Exchange mailbox servers (even standalone servers).

I'll take a few bullet points here to describe the various types of replication.  All replication I'll discuss in these posts involve taking the writes to storage or application, duplicating them, and sending them to another location.  If these locations are on the same server, disk, array, or site, it could be generally termed local replication.  If it's sent to another site, it can be called remote replication.  The different replication types are:

• Application replication:  This is where an application determines what is to be replicated at the application level.  In most cases, this is the application itself.  An example would be DAG replication, where writes to the Exchange transaction log are replicated to other members of the DAG.  Other examples would be SQL log shipping, Oracle DataGuard, or Active Directory replication.  An example where a middleware application determines what's to be replicated at the app level would be GoldenGate.
• Host replication:  This is where an agent running on the server, but outside the context of the application is configured to replicate data directed to one or more disks.  Examples of this type of replication would be Replistor, NSI Doubletake, and so forth.
• Array replication:  This is where the storage controller is configured to replicate writes destined for certain LUNs (or logical disks) configured on the array.  Examples of this type of replication would be EMC MirrorView and SRDF, NetApp SnapMirror and the like.
• SAN replication:  This is where the replication is handled by an appliance residing on a storage area network (SAN), but outside of the host or array.  Examples of this would include EMC RecoverPoint, NetApp ReplicatorX, IBM SVC, and technologies of that ilk.

Here's a little cartoon showing the types of replication:

Clearly, this is a simplification, and details differ between the implementation details the various vendors take.  Some third party products include both replication and application failover.  A book could be written on replication schemes, so I apologize for the brief treatment of the topic.  But all replication technologies I'm aware of can fit fairly comfortably in one of those four buckets, and it should suffice for this series of blog posts.

<whew>

Getting back to the topic at hand:

How can you replicate and fail over Exchange (other than DAG)?  Turns out there are a lot of possibilities.  The rules are pretty simple:

• You can't use DAG replication without Active Manager
• You should only use one high availability technique at a time

But you CAN use Active Manager without DAG replication.  This bears repeating:  You can get Exchange high availability without having multiple copies of your database.  And believe it or not, you don't have to use Active Manager at all.

I'm going to cover four different options for Exchange 2010 HA and replication, and this is not intended to be an exhaustive list: there are variants on each, and there might even be different methods.  Stay tuned for more...

Cognitive Dissonance and Storage

The Exchange group's obsession with direct-attached storage (DAS) is going to cost Microsoft market share in the virtualization space.

Quick quiz: How many Exchange servers does it take to provision 300 highly available Exchange mailboxes? The answer is six - and that's if you put client access and hub transport roles on the same servers.  There are some good reasons for that sort of suprising number (although it would be nice to be able to put more than one role on a clustered mailbox server).  But when I look at that ratio of servers to users, I see that Exchange is a classic candidate for server virtualization.  Indeed Microsoft now formally supports Exchange on Hyper-V, as well as third-party virtualization platforms.

Obviously VMWare isn't alone in their vision of a virtualized data center - Microsoft also realizes that in the very near future, the default platform for new applications will be virtualized even in the most conservative of IT environments.  Hence the heavy investment and incentives to adopt Hyper-V - they want customers to use Microsoft's hypervisor - not anyone else's.  That's great news for IT buyers.  Competition nearly always results in better value for the consumer.

Which makes the the Exchange team's latest spreadsheet effort a little more baffling than usual.  This spreadsheet is supposed to calculate the TCO of direct-attached storage to SAN storage.  There's a few warts on their approach even leaving out the virtualization discussion: 

• First, VSS clones and snapshots complement, rather than compete with CCR (since it allows rapid recovery from a variety of external threats, and error conditions that CCR cannot).  After all, SQL Server has had CCR-like functionality for many years now.  Lots of people use it (primarily on their SANs), but the SQL Server team does not claim it's a direct replacement for a rapid backup and restore system that leverages VSS or VDI. And neither CCR or database mirroring preclude the use of a SAN.

• Second, there's no way a generic spreadsheet can estimate the TCO of a given VSS-protected Exchange install, since each SAN vendor implements their VSS Provider differently. 

• Lastly, their effort leaves out a number of factors that are included in more evolved TCO studies (which show time after time that shared storage is more cost-efficient than direct-attached storage silos).

But what's really got me scratching my head is how strategically disconnected the Exchange team is from the rest of Microsoft (and indeed the rest of the IT world) when it comes to virtualization (which currently requires a storage network if you want to do high availability, live or quick migration, or any of the good stuff layered on top of it).  Most folks deploying Exchange today will expect their installation to last for a 3 to 5 year lifecycle.  During that time, most organizations will either deploy a new virtualization platform in production, or move forward with virtualizing more production applications.  So people who decide to deploy Exchange on direct-attached storage will end up buying entirely new storage to take advantage of the incredible ROI that virtualization offers.

Keep in mind that both of the major virtualization vendors offer simple methods to migrate existing production servers to their virtual platforms (and even free or inexpensive tools to determine likely candidates for virtualization).  So even while some folks have decided to deploy Exchange on physical servers, a lot of those same folks will decide to move it to virtualized servers in the next year or so.  A downturn in the economy is only going to accelerate that process as people seek to deliver more services with less infrastructure. 

The funny thing is that people planning a virtualized Exchange environment pretty much ignore the DAS message already.  The only folks considering deployment of Exchange on DAS are the very same folks who Microsoft hopes will be net new virtualization customers (deploying on Hyper-V) over the next 12 to 36 months (after they've released Windows 08 R2 and the features that will put Hyper-V on par with VMWare).  And the fact that they will be on DAS will prevent virtualization.

Dissonance in a large company like Microsoft is not unusual.  But I have to wonder when they'll realize they're shooting themselves in their virtual foot.

eseutil error 1216 JET_errAttachedDatabaseMismatch

I'm really lucky in that I get to work with a lot of very smart and experienced people.  So when I encounter something that has us all running fruitlessly to Google, I figure it might be worthwhile to share the knowledge.

Doing a reduced-capacity disaster recovery site is relatively common.  Basically, this is a scenario where you have multiple production sites, and while your DR site has all the data from all the sites, there's only enough processing power to run one services for the largest production site.  Depending on how many production sites there are, the savings can be considerable.

So we have a customer with 5 production mailbox servers at different sites around the US and a single mailbox server at their DR site.  They had all their databases, checkpoint files, and transaction logs, but mounting the database failed, and running eseutil /r failed with the following:

Operation terminated with error -1216 (JET_errAttachedDatabaseMismatch, An outstanding database attachment has been detected at the start or end of recovery, but database is missing or does not match attachment info) after 1.937 seconds.

That was brand new to us.  When the guy with 11 years of very deep Exchange experience asks, "what's a 1216 error?" you know you have something interesting.  After futzing around with a bunch of different settings, we changed the drive letters to be identical to the production server, and not only did eseutil run properly, we didn't actually have to run it to mount the databases.  It should work when the drive letters different, but troubleshooting this is not worth the effort.