Flipping Bits

Perfmon Counter of the Week: Logins/Sec

For the second in the series, I’m going to take a look at Logins/Sec.  You get this from SQLServer:General Statistics\Logins\sec.  It has impact on the user experience, as it takes time to set up a connection, and also has impact on the CPU consumption, as documented in Linchi Shea’s excellent blog post on the topic.  It’s important to remember that the stats reported by logins/sec do not include pooled connections

As you might imagine, there’s a direct correlation between logins/sec and transactions hitting the Master database as shown in the graph below.  I scaled them differently to show correlation, because it turns out they overlap otherwise (one login equals one transaction to Master).

You can find some great information on how to tell if connections are pooled or not in James Rowland-Jones blog post here.

There are a number of resources out there to help you figure out and solve issues around connection pooling.    Mohammed Mawla has a good post about the subject, but I think the best succinct advice is from Microsoft:

"To make the best use of connection pooling, explicitly close database connections as soon as possible. By default, a connection terminates after your script finishes execution. However, by explicitly closing a connection in your script after it is no longer needed, you reduce demand on the database server and make the connection available to other users."

Option 4: Third Party Replication (Or: How Stella Got Her Single Copy Cluster Back)

This is the fifth and final post in a series about the various options to achieve HA and DR with Exchange 2010.  In the first, I broke the DAG into its basic components (Active Manager and DAG replication).  In the second, I gave a quick overview of Native DAG.  In the third, I covered a hybrid approach that combined DAG replication and Active Manager for local HA, and array/SAN based replication for remote site recovery.  In the fourth, I described an option that deploys Exchange in a standalone configuration and leverages a hypervisor to achieve local high availability.

This one will cover Exchange’s Third Party Replication.  This one definitely has a lot of cool factor in it.  It actually leverages DAG (in the form of Active Manager) with array or SAN-based replication technology.  You get all the automatic failover, live patching, Exchange-aware coolness of DAG, zero data loss, and you only have to deploy one copy of the data at each site.

Although synchronous operation is possible with both Options 2 and 3, this is the only option shown that combines synchronous replication with automatic failover.

This option will use a plugin from EMC to coordinate the replication engine with Active Manager. This would be either Replication Enabler for Exchange 2010 (free!), or AutoStart 5.3 with the Exchange 2010 module. This option can also use a virtualization platform like Hyper-V or VMware, but a hypervisor is not necessary to leverage the benefits of this option.

Here are the cost factors:

• Storage: 2
• Network: 2

Architects and managers will typically consider this option when:

• Lossless, automatic failover is required
• Minimal hardware footprint is desired
• There is minimal latency between the sites
• A hardware VSS protection scheme is available for rapid recovery in the event of database corruption
• Live patching is required

Option 3: Virtualized Host Clustering

This is the fourth in a series of posts about the various options to achieve HA and DR with Exchange 2010.  In the first, I broke the DAG into its basic components (Active Manager and DAG replication).  In the second, I gave a quick overview of Native DAG.  In the third, I covered a hybrid approach that combined DAG replication and Active Manager for local HA, and array/SAN based replication for remote site recovery.

I call this option “Virtualized Host Clustering”, because like the Virtualized Local DAG option, this options leverages a hypervisor, but it leverages the HA capabilities of the hypervisor instead.  The Exchange mailbox role is deployed in a VM as a standalone server.  This is by far the least expensive option from all perspectives: acquisition, operation, complexity, footprint, and power.

As you can see, there are some clear cost benefits to this option. You are replicating only one copy of the database, and since the HA function is handled by the hypervisor, a second copy of the database is unnecessary. It’s also the most flexible option – the full suite of workload management tools provided by the hypervisor can be used, and we add a fourth potential replication engine – VPLEX.

I suppose it’s also worth noting that one does not necessarily need a virtualization layer to accomplish this. With the right operational recovery plan to replace the server and restore from backup, three nines (99.9% availability) could easily be achieved without any HA facility whatsoever (either at the application or hypervisor layer).

This is by far the least expensive option from all perspectives: acquisition, operation, complexity, footprint, and power.

Here are the cost factors broken down:

• Storage: 2
• Network: 2 (SRDF, MirrorView), .5 (RecoverPoint)

Administrators and managers will typically choose this option when want to:

• Minimize complexity of the deployment
• Use advanced virtualization features such as Live Migration/Vmotion, DRS, etc
• Achieve consistency with other line of business applications
• Control failover with scripts or tools like VMware Site Recovery Manager
• Control their RPO from zero data loss to minutes
• Have multiple recovery points at each site
• Control bandwidth utilized by replication
• Meet failover requirements not achievable with native Exchange’s Best Copy Selection

This solution is not without its drawbacks however.  Here are a couple of things to consider:

• This is the only option that where the administrator does not have the ability to do non-disruptive patching.  However, one should consider that boot times are pretty quick on virtual machines.  It’s very possible to achieve four 9’s (99.99% availability) with this solution despite the lack of live patching, and reboots can be scheduled for non-peak hours.
• Since only one copy of the data is available at each site, a rapid recovery mechanism for logical and physical failure modes is well advised.  This is usually achieved through hardware based snapshots or bookmarks at minimal cost.  It’s usually a good idea to have a rapid recovery scheme outside of the context of the application anyway, for a variety of reasons.

User Interfaces in Cars

This isn’t a topic I normally write about (or even think about).  But it’s been near top of mind for me as Mrs. Bits and I have been shopping for a new car.  We traded in our 2005 Toyota Sienna.  The primary feature we wanted was all-wheel drive (the 2011 New England winter was a nightmare), and of course, it needs to fit our little family of four and all the stuff we need for vacations and so forth. 

Another thing we wanted was a built in technology package that included most of the stuff we like to have around, especially for long trips.  In 2004, cutting edge vehicle technology meant having a place to plug in your iPod.  So one of the little frustrations of life was all the management of all the wires for the GPS, phone charger, DVD system, etc, etc.  Not to mention the lack of integration between all this stuff.   So one of the other things we wanted to splurge on was a robust system that we could be reasonably happy with seven or eight years from now as we progressively drive the wheels off the vehicle.

Pretty much every feature we wanted was available on every car we evaluated:  Navigation with traffic, bluetooth, satellite radio, auxiliary inputs.  Of course, all this comes at a price.  On every one of these vehicles, the tech kit was the single most expensive option on the car – typically between 10-15% of the total price of the vehicle.  Pretty hefty.

What struck me is how different the implementations were between vehicles.  Some had touch screens, some didn’t.  A couple had a particularly baffling interface that required manipulation of a joystick.  One of them was nowhere near the screen, but behind the gear shift, where the user doesn’t even have a view of the joystick.

Seriously?  You’re trying to sell me a $3500 computer where I have to blindly use a frickin joystick to type in addresses?  One salesman told me it was “because touch screens get fingerprints on them”.  Please.  I have two small kids.  Any space they occupy for more than about 10 seconds is filled with goldfish crackers and smooshed fig newtons.  Fingerprints on a touch screen don’t really concern me, especially since I can, you know, WIPE THEM OFF.

Eventually we decided on a Hyundai.  It was comparable in most respects to other vehicles we tested, but the UI on the navigation/phone/radio interface was the most familiar.  Basically, they tried to mimic other computer interfaces and GPS devices in wide use, and for the most part they were successful.

So I got to thinking about why there’s so much variance between the interfaces of different manufacturers, why so many otherwise high quality manufacturers can put out such insane technology, and whether differentiating your car with its electronics UI is a good idea.

I think I know why they do it.  It’s to differentiate themselves from their competitors; I know what it’s like to compete with other formidable products, and I realize that the smallest things can make the biggest difference to the customer.  It’s very easy for a competitor to match your specs in a car (mileage, acceleration, turning radius, etc), but it’s hard to replicate the “feel” of a car, and the tech package is a big part of the “feel” for people who actually buy and use it.

They continue to do it because nobody calls them out on it.  My brother in law is a car nut; he gets at least a half dozen car magazines, and I flip through them when I visit.  The reviewers focus on the driving experience.  How it handles, road noise, acceleration, and the numbers.  They do this because they’re all metrics, and they’re easy to review.  They barely pay attention to the technology kit, and if it's unusable, it hardly warrants a sentence.

I think it’s a terrible idea.  A moving vehicle is probably the worst possible place to introduce novel user interface concepts.  In addition, people tend to want a familiar user interface that’s easy for them to both figure how to manipulate, and works easily and quickly in operation.  It’s like trying to differentiate your car by putting the brake pedal on the right and introducing the great new pentagular steering wheel that’s safer because it forces you to rest your hands at 10 and 2.

I’m clearly not the only person who feels this way.  A quick browse turned up this article from 2004, which covers some automotive electronics UIs, but also things like remote controls.  I’m with him on the remote controls, but I’ll tell you, people don’t generally try to manipulate them on the highway.  He says that poor UIs lead to bad reviews, but I don’t agree.  Reviews of these devices barely mention a poor UI, despite the fact that I couldn’t imagine getting used to them, and people I know who have these systems barely leverage them precisely because of the UI.  He brings up a good point that manufacturers have little incentive to emphasize usability because the user doesn’t realize how bad it is until after they’ve paid for it.  He expresses hope for the future of the consumer product UI.  Unfortunately, it’s been seven years now, and few manufacturers are heeding his advice.

Option 2: Virtualized Local DAG

This is the third in a series of posts about the various options to achieve HA and DR with Exchange 2010.  In the first, I broke the DAG into its basic components (Active Manager and DAG replication).  In the second, I gave a quick overview of Native DAG.  In this post, I’m going to cover a pretty popular option for folks who’ve already made the decision to virtualize their Exchange environment.

I call it “Virtualized Local DAG” because it uses DAG replication and Active Manager for local HA, but third party technologies for remote replication and failover.

This configuration utilizes a hypervisor such as Hyper-V or VMware’s. A two member DAG group is created, and then both copies are replicated from one site to the other. It’s important to note that because HA is dealt with by Exchange natively, any HA or workload migration features offered by the hypervisor (such as VMHA, DRS, Live Migration and VMotion) should be disabled for the mailbox VMs (other Exchange roles can utilize those features).

As for the network cost, it’s going to be more expensive operationally – you’re replicating two copies of the databases and two copies of the logs, and the storage cost is identical to Native DAG Replication.  However, this can be mitigated through the use of interesting compression and data reduction techniques made available by products like RecoverPoint.

• Storage: 4
• Network: 4 (SRDF, MirrorView), 1 (RecoverPoint)

Why would anyone choose this option? Well first, the network cost isn’t much more than Native DAG replication when you need to make sure you can reseed your databases in a reasonable amount of time. And if you use a replication appliance like RecoverPoint, you can end up using even less bandwidth than Native DAG replication, owing to the very good compression and data reduction an appliance like that offers.

Basically, for customers who've decided to virtualize Exchange, and already have working replication technologies for other applications, this option offers a lot of flexibility, integration with their data center strategies, and costs nothing in terms of storage footprint when compared to traditional DAG.

Ultimately, people may choose this option when there’s a need for or concern about:

• Uptime while patching
• Simplified and coordinated failover of all Exchange roles and services
• The ability of the WAN to absorb re-seeding operations that will occur more frequently with log-based replication
• Better control of failover operations than what native Exchange’s Best Copy Selection can provide
• Full site failover of multiple applications (such as provided by PowerShell scripting with Hyper-V or VMware Site Recovery Manager)
• Consistency with other line of business applications
• Controllable RPO
• Synchronous replication (loss-less cross-site failover)
• Multiple recovery points are desired at each site
• Controllable bandwidth utilization
• Compliance and inclusion in a DR plan that includes other applications
• Business requirements that can’t be met by DAG replication or Active Manager

Option 1: Native Exchange DAG – A Quick Overview

Last month I promised a series of blog posts on the replication and HA options for Exchange, because it seems everywhere I go, people think they’re stuck with Database Availability Groups.  I started off with a level set on replication options in general, and then ended with the notion that you don’t necessarily need DAG replication and multiple copies to achieve HA and remote disaster recovery with Exchange 2010.  As Chad Sakac says, I’m agreeing to disagree with Microsoft on this point.  Microsoft contends that native Exchange approaches for protection, HA, and DR covers all needs, at all scales, and always at the lowest TCO.  I contend that sometimes it does, but not always.

So this is Option 1 in the series.  I'll be presenting three more options over the coming weeks.

DAG is actually pretty cool kit when you get into it, but like all techniques and technologies, it has its advantages and disadvantages.  Let’s get into it:

So here’s an illustration of what we’d call Native DAG Replication:

As you can see, it’s characterized by the use of DAG as the replication engine, and Active Manager as the failover manager. You might be wondering why I distinguish between the DAG members and physical copies. That’s because when you use third party replication (enabled by the Third Party Replication API in Exchange 2010 in conjunction with EMC’s Replication Enabler for Exchange), you can have multiple DAG members accessing the same physical disk (very similar to single copy clusters in previous versions of Exchange). Although it’s not shown here, this option can be deployed in conjunction with a virtualization platform like VMware or Hyper-V.

As a point of comparison, I’ll assign cost factors for storage and network. It’s fairly expensive in terms of storage – you need at least two copies locally. You can have one or two, or more copies at the remote site, but if you have only one copy, you should have enough bandwidth to comfortably reseed a full copy of all the databases on the server, since any number of situations can cause database divergence. If you don’t have the bandwidth, you can reduce the number of situations where a WAN reseed would be needed by putting a second copy at the DR site (as illustrated here). But operationally, network cost is low – only the transaction logs are replicated.

So the factors:

• Storage: 4
• Network: 1

Typically people would use this option when there is:

• A need for uptime while patching
• Plenty of bandwidth between sites
• Enough budget to acquire and maintain disk for 3-4 copies of the databases
• No need to coordinate Exchange site to site failover with other applications
• No need for zero data loss disaster recovery between sites
• No need to maintain consistency between Exchange and other line of business apps[1]
• The Best Copy Selection (BCS) process meets the requirements of the business unit

 

---

[1] An example of this might be a procurement application that leverages email or public folders for workflow

Perfmon Counter of the Week: Disk Latency

So far, all my blog posts have been sort of event-driven.  Something I read or hear about, perhaps something I’m doing for a customer tickles my fancy and I start to think about a blog post.  That’s probably why my blog went dark for a year and a half.  A few months ago my colleague Chuck Chiambalero suggested a “Perfmon Counter of the Week”.  I think really slowly, so I’m getting cracking on it now. 

About as slow as this guy

Each week, I’ll go over a perfmon counter or two that I regularly use when I’m sizing out a customer’s environment, or evaluating performance bottlenecks they may be having.  All of the counters I’ll be covering are collected in perfcollect (of course only if they’re available on the host – a SQL Server counter isn’t going to be collected if you run perfcollect on an Exchange server).

Anyway, Perfcollect looks for about 600 different counters for various applications, so that’ll keep me busy until 2023, at which point I’ll be busy selling my organs to raise college money for my kids.

We’ll start off with a couple of counters that are near and dear to my heart:  Average Disk sec/Read and Average Disk sec/Write.  These counters indicate the response time of the disk – how long it takes for a read or write request to get acknowledged by the disk.  You can grab this off of PhysicalDisk and LogicalDisk.  I prefer to look at LogicalDisk unless there’s a specific reason to be looking at PhysicalDisk.   As a general rule for databases, the average read latency should be less than 20ms for database files.  The average write latency should be 10ms for log files.  And of course since perfmon collects the data in seconds, you’ll have to multiply by a thousand in order to get your milliseconds.

General guidelines indicate that spikes should not be more than 50ms, but that’s pretty meaningless unless you’ve defined a sample interval (a 15 second sample interval is usually going to be a hell of a lot more spiky than a 2 minute or ten minute interval).

So is high disk latency ALWAYS a bad thing?  Well, no.  The numbers are there as a rule of thumb.  If you’ve got performance problems and there are high or relatively high latencies on the disks, you’ll be advised by support to investigate and fix it.  But if the latencies have been steady over time and the performance problems are new, clearly fixing the disk isn’t going to yield much for you.  Which is why it’s important not to wait until you have problems to collect perfmon data.

Another factor to consider is the IO size.  Logic and physics dictate that if you have two requests, one with a large amount of data, and one with a small amount of data, the smaller amount of data is going to complete more quickly on average.  With spinning disks and random workloads, the vast majority of time is spent seeking data as opposed to moving the data, but it’s still noticeable in a number of workloads.  So the larger the IO, the more latency you should expect.  Larger IO usually implies sequential workloads, which are measured in megabytes per second rather than IOs per second, anyway.

Can everything (including latency) be healthy and performance still stink?  Yes.  Imagine a transaction system where the organization gets paid by the number of transactions it can complete.  If the storage is a bottleneck and latencies are at 10ms, then the storage is still a bottleneck, and investigating advanced drive technologies is well advised.

Finding busy SQL databases and files

Bet you though this blog was all about messaging and Exchange.  Well, it’s not really.  Seriously.  I’m not kidding.  I do think about other stuff.

Ever since I’ve started working with Enterprise Flash Drives (EFD), people have asked me how they can identify which databases are the best candidates for EFD.  The thinking is clear:  EFD’s are great, but they are more expensive than other more conventional Fibre Channel/SAS drives.  So they understandably want to make sure they’re getting the best bang for the buck.

Now, there are some very cool advanced technologies around extending array cache with EFD (like EMC’s FASTCache and NetApp’s FlashCache – then there’s the old tried and true “Use gobs of traditional cache” approach).  There are also some technologies that can automatically tier data for you, (such as EMC’s FAST and Dell’s Data Progression).  Both of these techniques rely on the frequency of use, which makes sense.  Put the data that the server is requesting most frequently on the fastest storage.

This is great Ron Popeil-style “Set it and forget it” storage tiering.  But it’s still prudent to do some analysis to make sure that you’ve got enough of the EFD, and on the flip side you want to make sure that you don’t buy too much.  Now if you’re already an EMC customer with our high end or mid range storage products, we can do some analysis of the data to determine the optimal amount of EFD for your workloads.  But even that won’t take into account specific workloads that you want to accelerate.  And of course if you’re using something that can’t tier for you automatically, then you’re stuck with the manual method anyway.

This post is about looking at the problem from a SQL server perspective.  There are three techniques I use, and they complement each other.  Sometimes I use all three, sometimes only one or two.  All of them use a measure of activity divided by the size of a dataset.  To coin a term, I'll call this "I/O density").  The relative ratio is what you’re looking for to identify the EFD candidates.

• Perfmon data (LogicalDisk)

With this type of analysis, you take the GB used on each logical drive and divide that by the Reads/Sec and Writes/Sec from LogicalDisk.  You might get something like this:

What’s good about this approach is that you get to see data over time, so you can see how spiky the workload is.  You can also evaluate read/write mix and IO size, both of which are factors in the decision.  The drawback is that it doesn’t have the resolution that the other techniques use.  This is something even a rudimentary automated tiering technology could handle for you, namely “This LUN is busy.  Let’s move it up a tier.”  If you have multiple databases per LUN, or multiple files per LUN, you can’t tell which is driving the IO, so you could be moving more data than you need.  Also, you need more than just perfmon data – you need the logical disk sizes.  Perfcollect will capture all the necessary metrics in a nice little package for you.  It’s up to you to extract the data to CSV and analyze it (I used Excel here).

• Perfmon data (SQL Counters)

This involves taking a key counter for your database (<instancename>:Databases\Data File(s) Size (KB)) and dividing it by another counter for the database (<instancename>:Databases\Transactions/sec).  (Perfcollect will grab this data for you as well).  You might get something like this:

Again with this, you get timeseries data.  This case you see a more or less steady green line plugging along all through the sample.  In this case, it’s not a critical data set and I don’t need snappy response times for it, so I’ve already done something automated tiering cannot.  The time series aspect of this is cool, because I can see when people start logging in about 60% of the way through the sample, and I clearly identify the database I’d want to move.  The drawbacks are that (1) it identifies only transactions – not transactions that result in storage IO.  If you’ve got enough server-side cache, many transactions won’t ever hit disk, (2) it doesn’t distinguish between read and write transactions or the size of the resulting IO, both of which are key metrics for evaluating the applicability of EFD, and most importantly (3) it operates at the database level.  If you have multiple files in a user database, in many cases you’ll have one file busier than the others (check out the next method for an example).  Of course you may know which files are busiest (for example, if you’ve partitioned your database with tiering in mind), but this doesn’t tell you if you’re in for a surprise.

• fn_VirtualFileStats

This involves querying fn_virtualfilestats, so you need rights to log into the database.

Here’s the getSQLDBstats script.  Just log into a SQL server with management tools and run it like this:

For a default instance, run:

sqlcmd –S [hostname] -i getDBstats.sql –o [output file.csv]  -s”,”

For a named instance, run

sqlcmd –S [hostname]\[instancename] -i getDBstats.sql –o [outputfile] -s","

You’ll get a file that looks a little like this after you’ve formatted it in Excel.

Now that you’ve got the data, you can do some interesting things with it.  Primarily, you’ll be taking FileSize and dividing it by NumberReads and NumberWrites.  Eliminate the log files and you end up with something like this:

The advantages here are numerous.  First, you have file-level resolution, so if there are busier files in a given database, you can avoid moving the “slacker” files.  In this case, the files to the left are tempdb, which is very evenly distributed (as you'd expect).  The spiky files are user databases, so you can imagine that there's some sort of partitioning in use here, or that they've added files over time.  Second, you can evaluate read/write ratio and if you do some more math with BytesRead and BytesWritten, you can get get an idea of average IO size, both of which are useful data points for evaluating the bang for the buck you’ll get out of EFD.  Third, it’s quick.  Instead of grabbing data over a day, week or month, you can get this data in a matter of seconds, and with an Excel macro, you can have the analysis done just as quickly.  The drawback is that it’s not timeseries data – this is aggregated over time since the moment the database was attached.  So if you have event-based activities like end of day/month batch processing, that could significantly skew the analysis.  So I always like to look at it in conjunction with the perfmon time series data to ensure the analysis is correct.

S#!t Happens (Or: What we can learn from the latest gmail outage)

So here's evidently what happened:

Some time around February 27, gmail was affected (mid-upgrade) by a bug that effectively deleted the mail data associated with about 40,000 email accounts.  Now, Google maintains multiple copies of users' data, so this bug affected all the available copies of the data for these users.  Google had the foresight to backup their data rather than relying on data replication as its sole protection against data loss, but that backup data resides on tape, which clearly takes time to restore.  Just to give you an idea of how much time, you need an idea of the scale of the data loss.  If each of those users had 5GB of data in their mailboxes, the restore operation requires about 200TB of data - not unmanageable, but clearly something that would take on the order of days to weeks to restore unless something really very cool is used.  One of the interesting aspects of the restore process is that users report having no access to the email services while their data is being restored.  An Exchange administrator would have the ability to spin up some dial tone databases and use something like recover-mailbox, recovery storage groups or a more robust tool like Ontrack PowerControls to merge data from the backup sets back into the dial tone databases.

Now, this is not a schadenfreude post.  I have a lot of respect for Google, especially around how they've transformed messaging, and provided consumers with a very viable and attractive alternative to what was a pretty miserable corner of IT when it was first introduced in 2004 (my, how time flies, huh?). They've delivered a remarkably reliable infrastructure for a massive number of users at an incredible price.

However, as a technologist, I'd like to look at what happened, as well as the users' reactions to get an idea of how I can architect messaging systems so that when stuff inevitably hits the fan, the impact can be minimized.

1. Avoid backups at your own risk.  It's tempting, especially with three, four, or five copies, to think to yourself "Well, how many copies do I need before I don't need to back up?"  The fact is that all of those copies are in a single failure domain.  As my friend and colleague Jim Cordes says, "This will work, up to the point where it won't."  In this case, a storage bug (likely associated with Google Filesystem) created data loss.  But it could as easily have been an application bug, administrator error, or a security breach.  In this case, the data also resides outside of the failure domain (on tape).  It's generally advisable that critical data be available outside the context of the application.
2. Users don't just care about service availability - they care about their data too.  Many people live in their email accounts.  Whether we like it or not, their email account is where they keep their most important data.  So make sure your SLAs (either internal or with a service provider) cover data availability and not just service availability.
3. Users don't just care about service and data availability - they care about metadata too. Complaints about the loss of starred emails and labels abound.  This shouldn't surpise us.  If people live in their email accounts, then they'll organize it.  Think of it like a filing cabinet.  If your "backup" of your filing cabinet entails copying everything and putting in all in a fireproof canister, then "restoring" those files to a usable state where you can actually find something is going to be a problem.  This could be a problem for folks who use a compliance archive as a last-ditch resort for data restore.
4. Set distinct SLAs for service and data/metadata availability.  "Distinct" doesn't mean "different" in this context.  With a robust email solution you can get service availability up quickly and cheaply after a disaster.  Getting the actual data back is the longer pole in the tent, and where the bulk of investment is required.  If you tier your data through the use of archives, cost can be mitgated by assigning different SLAs for the active and archive data.
5. Make sure your backup/restore solution meet SLAs for data/metadata availability.  As we've seen, even the best run organizations running top notch software can experience data loss, even when multiple copies of the data are deployed.  If the copy of the data outside of the application failure domain takes 100 hours to restore, then that's the SLA you can sign.  If a disaster requires that you restore many terabytes of data and you have a data availability SLA of under a day, then it's advisable to look at hardware-based snapshots or bookmarks as a solution.
6. Fast backups ≠ Fast restores.  There are many solutions out there that help people meet aggressive backup windows (incremental forever with synthetic fulls are widely available, for example).  Administrators and managers are well-advised to examine the restore speeds of these solutions.  Basically, if the solution calls for the full dataset to be moved from one place to another in order for it to be used, then you need to examine the restore speed.