Flipping Bits

Enterprise Flash Drives: not just performance

I often encounter the misconception that EFDs are not beneficial unless you need to either reduce latencies below what traditional disks can get you, or you’re short-stroking your disk in order to maintain performance.  So I figured I’d go through the three general use cases I talk about with EFDs:

1. Do more stuff
2. Do the same stuff faster
3. Do the same stuff, but with less gear

These are not mutually exclusive.  In most cases, EFDs allow people to do more stuff, faster, with less gear.  But your goals for EFDs will certainly flavor how to best deploy them.

Do more stuff (increase scale)

Let’s use an entirely contrived order processing system (like a trading desk). Let’s say this system can support 1,000 trades a minute. But during peak trading times, you’re getting more trades than you can process.

The business case for EFDs here would be that you can increase revenue by processing more orders.  Here’s an example where six EFDs supported seven times the transactions of six traditional fibre channel drives.  And this is a perfect example of how increased scale and reduced latency are not mutually exclusive – the response times on the EFD drives were 7 times lower than the response times on the spinning disks.

source

Note that both of the cases thus far really depend on how much your EFDs cost, and how much productivity improvement you’re going to see from their deployment.  That’s significantly different than this one:

Do the same stuff faster (reduce latency):

Let’s take a large manual order-entry system, where user wait time for a query is 5 seconds, and users do about a one query per minute. Let’s say the performance gate in this scenario is storage and it’s getting about 5-7 ms latency (about as good as you can get with a performance HDD at scale due to rotational latency).

The business case for EFDs here would be that employees in this role spend about 8% of their time waiting on the database. If you can reduce that to 1.6%, you realize massive productivity improvements.

Here’s some data: Note the graphs are not on the same scale.

source

Do the same stuff, but with less gear (decrease footprint): 

Let’s say that you’ve got an application that’s fat and happy residing on ninety 10k performance HDDs.  You’re not short-stroking them too badly, but it’s still taking about $6,000 a year to power them, $2,000 a year to cool them, about 18U of rack space to store them, not to mention the maintenance cost associated with them.  But the fact is that in most environments, only part of the data set is actually frequently accessed.  An example of this might be an order processing or inventory system that may go back years or even decades.  The old data isn’t accessed very frequently at all, while the newer data is getting hammered.

Using either manual or automated tiering, you put the older data on cheaper, denser, and more power efficient nl-SAS drives, while the most frequently accessed data can reside on faster, less dense (but still more power efficient) EFDs.

Here’s some supporting data.  Now, the performance was slightly increased, but the bulk of the savings came in the form of footprint – acquisition, power, management, and so forth.  Had the goal been to increased scale or reduce response time, then the deployment method would have differed.

  

source

Best Practices for Windows Mounts Points

Mount points – they’re understandably popular.  And although they’ve been around for quite a while, some people have questions around their implementation.  Yes, they’re really easy to set up, but you should follow some guidelines so you can take leverage advanced technologies down the road.

For the uninitiated, here’s an quick overview of mount points:

Traditionally, a windows volume (disk drive, LUN, etc) had to be mounted to a drive letter for Windows to access it.  This results in a limitation of 24 hard drives that can be mounted to a system.  To overcome this limitation, Microsoft introduced the ability to mount a drive on an empty directory within any NTFS filesystem.  You can try it yourself next time you format a USB stick.  It’s quite neat – now you can have a virtually unlimited number of drives attached to your system.  You know, like UNIX.  .  It also makes it easier to find stuff.

Here’s an example:  Let’s say I have a database with 2 data files, and 1 transaction log, and I want to keep them on separate drives for recovery and performance purposes.  Instead of mounting them to letters g:\, h:\ and i:\, I can do this:

• g:\dbname\dbfile01
• g:\dbname\dbfile02
• g:\dbname\tlog

Where dbfile01, dbfile02, and tlog are all empty directories on which a drive is mounted (mount point).  The directory structure is clear to anyone who looks at it, and if I need to add more drives to my database I just create a directory in dbname, and put a drive on it.  I try not to put the mount points on my system drive (c:\), although I’m allowed to.  The reason is that the mount point looks just like a directory – you have to know that it’s a mount point.  When I have it on another drive letter, it’s clear that there’s another drive there. 

So what’s a nested mount point?  As the name implies, it’s where you have a mount point within a mount point.  In our case, the dbname directory could be a mount point.

Are nested mount points categorically a bad idea?  Absolutely not – in fact it’s a pretty common practice.  The key is that there shouldn’t be unique data higher in the directory structure than a mount point.  For example, this is a bad idea:

• g:\dbname\dbfile
• g:\dbname\dbfile\tlog

Where both dbfile and tlog are both mount points.  First, it wouldn’t occur to another DBA that tlogs is actually a different drive.  It also has implications for backup and recovery if you’re trying to leverage a volume-based backup and recovery system.  The reason for this becomes clear when you start playing through a recovery scenario.  Let’s say I want to restore my database, but keep my transaction log around for replay.  VSS and the SQL Virtual Device Interface (VDI) backup and restore at the disk level rather than the file level.  So I’ve completely replaced not just g:\dbname\dbfile, but also g:\dbname\dbfile\tlog, where the transaction log lives.

I haven’t necessarily lost any data – I can just go find the tlog drive, mount it up, and continue my recovery.  But it clearly poses problems if I want to automate the process.

So this causes all sorts of confusion around whether nested mount points are supported.  In the case of Replication Manager, the answer is yes, nested mount points are supported, as long the volumes in the application set are not nested within each other.  To give examples, this is supported:

• g:\dbname\dbfile
• g:\dbname\tlog

Where dbname, dbfile, and tlog are all mount points, but only dbfile and tlog are part of the application set being protected.

On the other hand, this is not supported:

• g:\dbname\dbfile
• g:\dbname\dbfile\tlog

Where dbfile and tlog are mount points and both part of the application set being protected.

If you have any lessons learned about mount points you think are worth sharing, post a comment below!

Perfmon Counter of the Month–Disk Queue Length

OK, so maybe Perfmon Counter of the Week was a little optimistic.  Let’s say month, OK?

This one is another disk-related counter, but I like it because there’s so much myth around it.  It seems anytime I see a document around disk performance as it related to an application, I see references to “disk queue length”.  And it is either really vague, like “monitor the disk queue length – if it’s high, you may have problems with the disk subsystem,” or it’s really specific like “if your disk queue length is consistently higher than 2, you may have a problem with your disk”.

And neither piece of advice is quite wrong, nor is it quite right.

Here’s the thing with disk queue length, and any other queue-oriented counter:  Whether a queue length is “bad” or “good” depends on the number of resources servicing the queue. 

Imagine you’re in line at the grocery store.  There’s a single cashier, and there are nine people in front of you.  That’s a queue length of ten, and you’ll start wondering whether you really need the quinoa your wife told you to get.  Now imagine that there’s a single line with nine people in front of you, but there are 5 cashiers.  You still have a queue length of ten, but you’re going to be through it in a lot less time.  In fact, it’s better than having one person in front of you if there’s a single cashier, because that dude in front of you might have 63 coupons and a checkbook. If there are other cashiers, he’s only dominating one of multiple resources.

Unfortunately, that doesn’t make the quinoa look any more appetizing.

So does that mean that disk queue length is useless?  Not at all.  If you know how many resources there are servicing the LUN, then you can see whether it’s really a problem or not.  Typically the rule of thumb is 2 per disk.  So a queue length of 6 may indicate a bottleneck if you only have 2 disks in the RAID group servicing the LUN.  But it’s really not a problem if you have a 10-disk R1/0 RAID group.  The same goes for processor queue length.

The other aspect in which it is interesting is trending.  I’m a huge fan of measuring performance stats when application performance is good so that you have something to compare it to when things go downhill.  So it’s useful as a comparative value as well.  Even if you don’t have performance data from the “good times”, you can still visualize it with a time-series chart.

I/O Density and Exchange: A short history

Turns out that some people are dubious that Exchange 2010 and 7.2k drives are a good fit.   So I’m sometimes asked to justify spending less on storage for Exchange (a seemingly odd, but ultimately honest position to take).

Up until a short while ago, I just trotted out Microsoft’s graphs regarding the IO per “heavy use” mailbox they see in production.  It shows a 10x decrease in IO per mailbox between Exchange 2003 and Exchange 2010.

This can be unconvincing to some of the folks who fought Exchange 5.5, 2000, and 2003 in the trenches, where it was common knowledge that not only did you put Exchange on the fastest storage available, you often had to allocate more disks than you needed for capacity to meet the performance demands of the application.  A 10x reduction in IO workload doesn’t convincingly take you from “put it on the fastest storage you can afford” to “put it on the slowest storage you can buy”.

The story is actually two-fold – at the same time that IO per mailbox was being driven down, the size of the mailboxes were being driven up.  Since storage sizing is and always has been a balance between performance (how quickly you need to move data on and off the drives) and capacity (how much data you need to store on the drive), IO density (or IO per GB) is far more relevant than IO per mailbox.  In short, we don’t just look at how many IOs Exchange is driving per mailbox, we have to factor in how big those mailboxes are as well.  Typical mailboxes were only 50MB-100MB during the Exchange 2003 days.  Nowadays they’re between 10 and 100 times that.  When we do factor in mailbox sizes, it turns out that IO density isn’t 10 times lower with Exchange 2010, it’s orders of magnitude lower in most modern Exchange configurations.

I’ve come up with a graph to illustrate this notion. 

So while we saw “only” a 10x decrease in the IO per mailbox, we saw over a 400x decrease in IO per gigabyte.

Now let’s take a look at the IO density that spinning disks have offered throughout the years.

Clearly these are simple examples – the mailbox IO was far more variable in the 32-bit versions of Exchange, and the IO/GB for the disk drives doesn’t include RAID overheads, there’s no free space factored in and so forth.  But when we interpolate the IO/GB driven by mailboxes and the IO/GB provided by disks, it matches up pretty well with the typical disk topologies I recommend:

In the Exchange 5.5 to 2003 days, every IO counted, and you had allocate more disks to the Exchange workload than you needed for capacity.  Things got a lot better in Exchange 2007, where you could place your average mailboxes on either smaller 10k FC drives or larger 15k FC drives.  With Exchange 2010 and large mailboxes, the sweet spot for the typical deployment is somewhere between 1TB and 2TB drives.

Now this isn’t to say that 2TB or 1TB drives are appropriate for all Exchange deployments.  A large number of factors go into sizing storage for Exchange.  But if you’ve done your homework and factored in all the variables, and it looks like the slower drives are the way to go, then you should trust your sizing and save your organization some money.

Option 4: Third Party Replication (Or: How Stella Got Her Single Copy Cluster Back)

This is the fifth and final post in a series about the various options to achieve HA and DR with Exchange 2010.  In the first, I broke the DAG into its basic components (Active Manager and DAG replication).  In the second, I gave a quick overview of Native DAG.  In the third, I covered a hybrid approach that combined DAG replication and Active Manager for local HA, and array/SAN based replication for remote site recovery.  In the fourth, I described an option that deploys Exchange in a standalone configuration and leverages a hypervisor to achieve local high availability.

This one will cover Exchange’s Third Party Replication.  This one definitely has a lot of cool factor in it.  It actually leverages DAG (in the form of Active Manager) with array or SAN-based replication technology.  You get all the automatic failover, live patching, Exchange-aware coolness of DAG, zero data loss, and you only have to deploy one copy of the data at each site.

Although synchronous operation is possible with both Options 2 and 3, this is the only option shown that combines synchronous replication with automatic failover.

This option will use a plugin from EMC to coordinate the replication engine with Active Manager. This would be either Replication Enabler for Exchange 2010 (free!), or AutoStart 5.3 with the Exchange 2010 module. This option can also use a virtualization platform like Hyper-V or VMware, but a hypervisor is not necessary to leverage the benefits of this option.

Here are the cost factors:

• Storage: 2
• Network: 2

Architects and managers will typically consider this option when:

• Lossless, automatic failover is required
• Minimal hardware footprint is desired
• There is minimal latency between the sites
• A hardware VSS protection scheme is available for rapid recovery in the event of database corruption
• Live patching is required

Dissecting Database Availability Groups

I wanted to post a good picture of dissection at the beginning of this post, and it brought me back to middle school biology.  I was in eighth or ninth grade.  We are the World was a chart-topper, and the Asian Tiger Mosquito (awesome name for species, eh?) managed to migrate to Houston.  The important thing is that I wore an onion on my belt (which was the style of the time). 

Well, no, I guess that wasn't the most important thing.  The really important thing is that I couldn't find a picture of a real frog dissection that I want to post on this particular blog.  So I'll post this instead:

   

Onto what I'd really like to get at with this post:

DAG is not the only way to achieve high availability or remote site disaster recovery with Exchange 2010.  The reality is that there are quite a few options for replication and HA, and they run the gamut of requirements, from synchronous replication with automatic, lossless failover, to asynchronous replication with selectable recovery points, and from a single copy of the database(s) at each sites to whatever you choose.

So over the next few posts, I'll be going over the options that administrators and managers can consider when deploying Exchange 2010.  Most importantly, I'll be covering the factors that should be included in that consideration.

Before I start, I need to conceptually take Database Availability Groups (or DAG) apart, show a little bit about how the technology works, and define the terminology associated with it.

Most Exchange administrators have a good idea of what DAG is, and know enough about how it works to implement it.  But many administrators don't know that there are two components to DAG.  As with any true remote disaster recovery technology, there are two mechanisms:

1. There's a technology to replicate the data, known as the replication engine.  In native Exchange 2010, we'll call this "DAG Replication." It consists of asynchronous block-level replication of the transaction logs over an IP network between the members of a DAG.
2. There's a technique or technology that manages which copy of the data is "active."  A "technique" would be something a human does to decide which copy is active.  A "technology" is an automated process that determines which copy is active.  In Exchange 2010, this is a technology (in the form of a role) called Active Manager that runs on ALL Exchange mailbox servers (even standalone servers).

I'll take a few bullet points here to describe the various types of replication.  All replication I'll discuss in these posts involve taking the writes to storage or application, duplicating them, and sending them to another location.  If these locations are on the same server, disk, array, or site, it could be generally termed local replication.  If it's sent to another site, it can be called remote replication.  The different replication types are:

• Application replication:  This is where an application determines what is to be replicated at the application level.  In most cases, this is the application itself.  An example would be DAG replication, where writes to the Exchange transaction log are replicated to other members of the DAG.  Other examples would be SQL log shipping, Oracle DataGuard, or Active Directory replication.  An example where a middleware application determines what's to be replicated at the app level would be GoldenGate.
• Host replication:  This is where an agent running on the server, but outside the context of the application is configured to replicate data directed to one or more disks.  Examples of this type of replication would be Replistor, NSI Doubletake, and so forth.
• Array replication:  This is where the storage controller is configured to replicate writes destined for certain LUNs (or logical disks) configured on the array.  Examples of this type of replication would be EMC MirrorView and SRDF, NetApp SnapMirror and the like.
• SAN replication:  This is where the replication is handled by an appliance residing on a storage area network (SAN), but outside of the host or array.  Examples of this would include EMC RecoverPoint, NetApp ReplicatorX, IBM SVC, and technologies of that ilk.

Here's a little cartoon showing the types of replication:

Clearly, this is a simplification, and details differ between the implementation details the various vendors take.  Some third party products include both replication and application failover.  A book could be written on replication schemes, so I apologize for the brief treatment of the topic.  But all replication technologies I'm aware of can fit fairly comfortably in one of those four buckets, and it should suffice for this series of blog posts.

<whew>

Getting back to the topic at hand:

How can you replicate and fail over Exchange (other than DAG)?  Turns out there are a lot of possibilities.  The rules are pretty simple:

• You can't use DAG replication without Active Manager
• You should only use one high availability technique at a time

But you CAN use Active Manager without DAG replication.  This bears repeating:  You can get Exchange high availability without having multiple copies of your database.  And believe it or not, you don't have to use Active Manager at all.

I'm going to cover four different options for Exchange 2010 HA and replication, and this is not intended to be an exhaustive list: there are variants on each, and there might even be different methods.  Stay tuned for more...

Double Disk Failures

"The duration of this rebuild operation is important because a quick rebuild lowers the probability that data will be lost through a second drive failure before the rebuild completes."

That's a quote from one of those bought-and-paid-for "technical reports" that some vendors like to use as marketing tools.

The problem is that it's not true (or, at the very least, it's not as true as they'd like it to be).  The unfortunate fact is that improving RAID rebuild times does close to nothing to reduce the risk of double disk failures in parity RAID groups.

Here's the inaccurate theory:

You have a parity RAID group, and one disk fails.  The array immediately goes and picks out a hot spare, and starts rebuilding the data from the lost disk, but it's working without a net until that data can be rebuilt.  Murphy's law kicks in, and all of a sudden, a disk in the SAME RAID GROUP has a head crash, you can no longer rebuild your data.  You then go get your last backup tape and you go update your resume while the restore progresses.

The problem is that the math doesn't match the anecdotal frequency of the notorious double disk failure.  Basically, that kind of thing shouldn't happen very often at all.  Seagate advertises a 1.2 million hour MTBF on the Barracuda ES.2 SATA drives (they kind they sell to storage vendors like EMC, IBM, and others).  Let's assume that number is, well, optimistic and divide it by 3 for a 400,000 hour MTBF.  Then let's assume that it takes 48 hours to complete a 5-disk RAID group rebuild.  The chances of one of those 4 surviving members failing during 48 hours is:

(48*4)/400,000 = .00048, or .048%, or once in about 4800 rebuilds. 

Now that's a lot of rebuilds before you actually hit a problem.  Keep in mind that we're actually using 1/3 of Seagate's MTBF rating for its least expensive drive.   But I can't seem to go a week without meeting with someone that's had one of these bite them in the last two years.  I'll grant you that a lot of the people I talk to meet with me exactly because they're unhappy with their storage vendor, but it's still a huge discrepancy.

Some folks will associate the second failure with the increased duty cycle during a rebuild causing a second crash, or to use bathtub curves and "bad batch of disks" theories.  But that's all missing the point.

What's really happening:

The truth is actually pretty mundane - it has to do with undetected errors on the surviving disks in the RAID group.  When you do a rebuild from parity, part of that process means that you have to read every single bit on every single surviving disk in the RAID group.  If you can't read even one of those bits, you're not going to be able to determine what was on the failed disk.  And the array usually reports it as a double disk failure.  Which, in fact, it is.  You have one completely failed disk, followed by a read error on a surviving disk. 

Disk vendors measure this stuff, and publish it in the spec sheets for their drives.  It's expressed by Seagate, for example, as "Nonrecoverable Read Errors per Bits Read."  Basically, this is what appears to be a huge number that represents average number of bits you can expect to successfully read off the disk before before you encounter a bit you can't read.

This happens all the time - we just don't notice it until it happens during a rebuild.  Under normal operations, if an array sends a read request to a disk and doesn't get a response, it can always just rebuild the requested data from parity or the mirror.  It's just a little burp, but if you're rebuilding a parity RAID group, the array goes from a little burp to a full-on run for the bathroom.

And this brings the math closer to what we all would expect from experience.  For example, if we can expect that we'll encounter a read error with every 100 terabytes of data read, and we know that the rebuild of a 4+1 RAID group made up of 1TB drives is going to require 4 terabytes of data to be read, we'll see about one in 25 rebuilds will fail (unless the array vendor has some magic to help out).

The implications:

So, knowing this, what can we expect?  Well, first, we can expect that faster parity rebuilds doesn't translate into higher reliability - most of the time, the matter of a second drive failing is a matter of fate that's set before the first disk ever fails - a faster rebuild only means that you'll find out about the failure a little sooner.  Second, we know that our exposure to the problem directly correlates to the size of the disks involved - the bigger the disks, or more accurately, the more data in the RAID group, the higher the chance you'll hit one of these read errors.

What can be done to reduce the risk:

There are a number of techniques that can be employed to reduce the risk.  Some vendors implement them, some don't, but I'd certainly use them as evaluation criteria for any storage buy, whether direct-attached or SAN-attached.

• Background verification - This is a process that verifies the integrity on an ongoing basis, the  goal being to find the latent errors on the drive while you have the ability to recover from them
• Predictive disk failure - This is when the array pre-emptively fails a troubled disk before it actually fails.  Usually, the data are copied from the failing disk, rather than being rebuilt from parity.  This reduces the time to fail out the disk and reduces load on the array since parity is not re-calculated.  Most importantly, you can suffer a read error during the process and the RAID group can survive.
• RAID-6 - Since parity is calculated twice, you can survive a read error at any point in time.  The chances that the same block would be bad in two places is statistically insignificant. 

In most environments, a combination of verification and predictive disk failure makes double disk failure a very remote possibility.  RAID-6 becomes attractive when you have a need for very large RAID groups with very large disks with relatively high unrecoverable read rates (typically SATA, although the vendors have improved those numbers significantly in last few generations).

As for the white paper and vendor from which this came, I'm not sure what to think.  The vendor is trying to mislead customers into thinking that there is a linear relationship between rebuild rates and data availability in a parity RAID scheme.  Either this is a shell game to hide the fact they don't have enterprise class features on their arrays or they really don't know the true cause of double disk failures.  The first is pretty shameful, and the second is pretty scary.