Flipping Bits

Enterprise Flash Drives: not just performance

I often encounter the misconception that EFDs are not beneficial unless you need to either reduce latencies below what traditional disks can get you, or you’re short-stroking your disk in order to maintain performance.  So I figured I’d go through the three general use cases I talk about with EFDs:

1. Do more stuff
2. Do the same stuff faster
3. Do the same stuff, but with less gear

These are not mutually exclusive.  In most cases, EFDs allow people to do more stuff, faster, with less gear.  But your goals for EFDs will certainly flavor how to best deploy them.

Do more stuff (increase scale)

Let’s use an entirely contrived order processing system (like a trading desk). Let’s say this system can support 1,000 trades a minute. But during peak trading times, you’re getting more trades than you can process.

The business case for EFDs here would be that you can increase revenue by processing more orders.  Here’s an example where six EFDs supported seven times the transactions of six traditional fibre channel drives.  And this is a perfect example of how increased scale and reduced latency are not mutually exclusive – the response times on the EFD drives were 7 times lower than the response times on the spinning disks.

source

Note that both of the cases thus far really depend on how much your EFDs cost, and how much productivity improvement you’re going to see from their deployment.  That’s significantly different than this one:

Do the same stuff faster (reduce latency):

Let’s take a large manual order-entry system, where user wait time for a query is 5 seconds, and users do about a one query per minute. Let’s say the performance gate in this scenario is storage and it’s getting about 5-7 ms latency (about as good as you can get with a performance HDD at scale due to rotational latency).

The business case for EFDs here would be that employees in this role spend about 8% of their time waiting on the database. If you can reduce that to 1.6%, you realize massive productivity improvements.

Here’s some data: Note the graphs are not on the same scale.

source

Do the same stuff, but with less gear (decrease footprint): 

Let’s say that you’ve got an application that’s fat and happy residing on ninety 10k performance HDDs.  You’re not short-stroking them too badly, but it’s still taking about $6,000 a year to power them, $2,000 a year to cool them, about 18U of rack space to store them, not to mention the maintenance cost associated with them.  But the fact is that in most environments, only part of the data set is actually frequently accessed.  An example of this might be an order processing or inventory system that may go back years or even decades.  The old data isn’t accessed very frequently at all, while the newer data is getting hammered.

Using either manual or automated tiering, you put the older data on cheaper, denser, and more power efficient nl-SAS drives, while the most frequently accessed data can reside on faster, less dense (but still more power efficient) EFDs.

Here’s some supporting data.  Now, the performance was slightly increased, but the bulk of the savings came in the form of footprint – acquisition, power, management, and so forth.  Had the goal been to increased scale or reduce response time, then the deployment method would have differed.

  

source

Best Practices for Windows Mounts Points

Mount points – they’re understandably popular.  And although they’ve been around for quite a while, some people have questions around their implementation.  Yes, they’re really easy to set up, but you should follow some guidelines so you can take leverage advanced technologies down the road.

For the uninitiated, here’s an quick overview of mount points:

Traditionally, a windows volume (disk drive, LUN, etc) had to be mounted to a drive letter for Windows to access it.  This results in a limitation of 24 hard drives that can be mounted to a system.  To overcome this limitation, Microsoft introduced the ability to mount a drive on an empty directory within any NTFS filesystem.  You can try it yourself next time you format a USB stick.  It’s quite neat – now you can have a virtually unlimited number of drives attached to your system.  You know, like UNIX.  .  It also makes it easier to find stuff.

Here’s an example:  Let’s say I have a database with 2 data files, and 1 transaction log, and I want to keep them on separate drives for recovery and performance purposes.  Instead of mounting them to letters g:\, h:\ and i:\, I can do this:

• g:\dbname\dbfile01
• g:\dbname\dbfile02
• g:\dbname\tlog

Where dbfile01, dbfile02, and tlog are all empty directories on which a drive is mounted (mount point).  The directory structure is clear to anyone who looks at it, and if I need to add more drives to my database I just create a directory in dbname, and put a drive on it.  I try not to put the mount points on my system drive (c:\), although I’m allowed to.  The reason is that the mount point looks just like a directory – you have to know that it’s a mount point.  When I have it on another drive letter, it’s clear that there’s another drive there. 

So what’s a nested mount point?  As the name implies, it’s where you have a mount point within a mount point.  In our case, the dbname directory could be a mount point.

Are nested mount points categorically a bad idea?  Absolutely not – in fact it’s a pretty common practice.  The key is that there shouldn’t be unique data higher in the directory structure than a mount point.  For example, this is a bad idea:

• g:\dbname\dbfile
• g:\dbname\dbfile\tlog

Where both dbfile and tlog are both mount points.  First, it wouldn’t occur to another DBA that tlogs is actually a different drive.  It also has implications for backup and recovery if you’re trying to leverage a volume-based backup and recovery system.  The reason for this becomes clear when you start playing through a recovery scenario.  Let’s say I want to restore my database, but keep my transaction log around for replay.  VSS and the SQL Virtual Device Interface (VDI) backup and restore at the disk level rather than the file level.  So I’ve completely replaced not just g:\dbname\dbfile, but also g:\dbname\dbfile\tlog, where the transaction log lives.

I haven’t necessarily lost any data – I can just go find the tlog drive, mount it up, and continue my recovery.  But it clearly poses problems if I want to automate the process.

So this causes all sorts of confusion around whether nested mount points are supported.  In the case of Replication Manager, the answer is yes, nested mount points are supported, as long the volumes in the application set are not nested within each other.  To give examples, this is supported:

• g:\dbname\dbfile
• g:\dbname\tlog

Where dbname, dbfile, and tlog are all mount points, but only dbfile and tlog are part of the application set being protected.

On the other hand, this is not supported:

• g:\dbname\dbfile
• g:\dbname\dbfile\tlog

Where dbfile and tlog are mount points and both part of the application set being protected.

If you have any lessons learned about mount points you think are worth sharing, post a comment below!

Option 4: Third Party Replication (Or: How Stella Got Her Single Copy Cluster Back)

This is the fifth and final post in a series about the various options to achieve HA and DR with Exchange 2010.  In the first, I broke the DAG into its basic components (Active Manager and DAG replication).  In the second, I gave a quick overview of Native DAG.  In the third, I covered a hybrid approach that combined DAG replication and Active Manager for local HA, and array/SAN based replication for remote site recovery.  In the fourth, I described an option that deploys Exchange in a standalone configuration and leverages a hypervisor to achieve local high availability.

This one will cover Exchange’s Third Party Replication.  This one definitely has a lot of cool factor in it.  It actually leverages DAG (in the form of Active Manager) with array or SAN-based replication technology.  You get all the automatic failover, live patching, Exchange-aware coolness of DAG, zero data loss, and you only have to deploy one copy of the data at each site.

Although synchronous operation is possible with both Options 2 and 3, this is the only option shown that combines synchronous replication with automatic failover.

This option will use a plugin from EMC to coordinate the replication engine with Active Manager. This would be either Replication Enabler for Exchange 2010 (free!), or AutoStart 5.3 with the Exchange 2010 module. This option can also use a virtualization platform like Hyper-V or VMware, but a hypervisor is not necessary to leverage the benefits of this option.

Here are the cost factors:

• Storage: 2
• Network: 2

Architects and managers will typically consider this option when:

• Lossless, automatic failover is required
• Minimal hardware footprint is desired
• There is minimal latency between the sites
• A hardware VSS protection scheme is available for rapid recovery in the event of database corruption
• Live patching is required

Option 3: Virtualized Host Clustering

This is the fourth in a series of posts about the various options to achieve HA and DR with Exchange 2010.  In the first, I broke the DAG into its basic components (Active Manager and DAG replication).  In the second, I gave a quick overview of Native DAG.  In the third, I covered a hybrid approach that combined DAG replication and Active Manager for local HA, and array/SAN based replication for remote site recovery.

I call this option “Virtualized Host Clustering”, because like the Virtualized Local DAG option, this options leverages a hypervisor, but it leverages the HA capabilities of the hypervisor instead.  The Exchange mailbox role is deployed in a VM as a standalone server.  This is by far the least expensive option from all perspectives: acquisition, operation, complexity, footprint, and power.

As you can see, there are some clear cost benefits to this option. You are replicating only one copy of the database, and since the HA function is handled by the hypervisor, a second copy of the database is unnecessary. It’s also the most flexible option – the full suite of workload management tools provided by the hypervisor can be used, and we add a fourth potential replication engine – VPLEX.

I suppose it’s also worth noting that one does not necessarily need a virtualization layer to accomplish this. With the right operational recovery plan to replace the server and restore from backup, three nines (99.9% availability) could easily be achieved without any HA facility whatsoever (either at the application or hypervisor layer).

This is by far the least expensive option from all perspectives: acquisition, operation, complexity, footprint, and power.

Here are the cost factors broken down:

• Storage: 2
• Network: 2 (SRDF, MirrorView), .5 (RecoverPoint)

Administrators and managers will typically choose this option when want to:

• Minimize complexity of the deployment
• Use advanced virtualization features such as Live Migration/Vmotion, DRS, etc
• Achieve consistency with other line of business applications
• Control failover with scripts or tools like VMware Site Recovery Manager
• Control their RPO from zero data loss to minutes
• Have multiple recovery points at each site
• Control bandwidth utilized by replication
• Meet failover requirements not achievable with native Exchange’s Best Copy Selection

This solution is not without its drawbacks however.  Here are a couple of things to consider:

• This is the only option that where the administrator does not have the ability to do non-disruptive patching.  However, one should consider that boot times are pretty quick on virtual machines.  It’s very possible to achieve four 9’s (99.99% availability) with this solution despite the lack of live patching, and reboots can be scheduled for non-peak hours.
• Since only one copy of the data is available at each site, a rapid recovery mechanism for logical and physical failure modes is well advised.  This is usually achieved through hardware based snapshots or bookmarks at minimal cost.  It’s usually a good idea to have a rapid recovery scheme outside of the context of the application anyway, for a variety of reasons.

Option 2: Virtualized Local DAG

This is the third in a series of posts about the various options to achieve HA and DR with Exchange 2010.  In the first, I broke the DAG into its basic components (Active Manager and DAG replication).  In the second, I gave a quick overview of Native DAG.  In this post, I’m going to cover a pretty popular option for folks who’ve already made the decision to virtualize their Exchange environment.

I call it “Virtualized Local DAG” because it uses DAG replication and Active Manager for local HA, but third party technologies for remote replication and failover.

This configuration utilizes a hypervisor such as Hyper-V or VMware’s. A two member DAG group is created, and then both copies are replicated from one site to the other. It’s important to note that because HA is dealt with by Exchange natively, any HA or workload migration features offered by the hypervisor (such as VMHA, DRS, Live Migration and VMotion) should be disabled for the mailbox VMs (other Exchange roles can utilize those features).

As for the network cost, it’s going to be more expensive operationally – you’re replicating two copies of the databases and two copies of the logs, and the storage cost is identical to Native DAG Replication.  However, this can be mitigated through the use of interesting compression and data reduction techniques made available by products like RecoverPoint.

• Storage: 4
• Network: 4 (SRDF, MirrorView), 1 (RecoverPoint)

Why would anyone choose this option? Well first, the network cost isn’t much more than Native DAG replication when you need to make sure you can reseed your databases in a reasonable amount of time. And if you use a replication appliance like RecoverPoint, you can end up using even less bandwidth than Native DAG replication, owing to the very good compression and data reduction an appliance like that offers.

Basically, for customers who've decided to virtualize Exchange, and already have working replication technologies for other applications, this option offers a lot of flexibility, integration with their data center strategies, and costs nothing in terms of storage footprint when compared to traditional DAG.

Ultimately, people may choose this option when there’s a need for or concern about:

• Uptime while patching
• Simplified and coordinated failover of all Exchange roles and services
• The ability of the WAN to absorb re-seeding operations that will occur more frequently with log-based replication
• Better control of failover operations than what native Exchange’s Best Copy Selection can provide
• Full site failover of multiple applications (such as provided by PowerShell scripting with Hyper-V or VMware Site Recovery Manager)
• Consistency with other line of business applications
• Controllable RPO
• Synchronous replication (loss-less cross-site failover)
• Multiple recovery points are desired at each site
• Controllable bandwidth utilization
• Compliance and inclusion in a DR plan that includes other applications
• Business requirements that can’t be met by DAG replication or Active Manager