« mbralign on Steroids | Main | Playing to Lose, Hoping to Win: EMCs Latest Guarantee (Part 1) »

May 01, 2010


Sounds nice, but I do have to say one thing. IMHO, anyone taking snapshots against a live database without going into backup mode is just asking for trouble. Crash recovery backup is called that because it's as good as a crash. I prefer my backups to be more stable than that.

Archivelog mode doesn't add any additional processing in Oracle. All it does is copy the previous archivelog out to the archived redolog directory when a log switch occurs. In testing I've never found this to have anything more than a 3% affect on performance. (Although I have to say I've never tested it on a 300 TB database.) To me, it's like saying the weight of the brakes and the spare tire will slow down the car, so you should take them off. They're there for a reason.

I had a customer years ago that had a 300 GB database Oracle (when that was huge). They hadn't EVER backed it up. They couldn't do a cold backup because they did database loads at night. They couldn't do a hot backup because they said archivelog mode slowed their database down. So they just didn't back up (snapshots weren't an option back then). I raised holy hell until they turned on archivelog mode and we got the first backup of the database ever. The next weekend they lost their primary array and my backup saved the day.


Thank you for your comments. I tend to agree that, even with additional load from log switches, a system can be designed to allow archive logging. In some cases, though, despite all efforts, you just have to take the customers word for it that they tried it and it didn't work well for them.

I think we all have a natural discomfort for any type of crash consistency, that is why both NetApp and Oracle recommend hot or cold backups if at all possible.

However, something I have to do frequently is test my fears against the technical merits of a solution. I'm not comfortable, nor would recommend, a customer cold power off a NetApp array twice a day. But, I would guarantee it would cleanly recovery ever single time. It is designed to do so. If it doesn't, there is a product defect that needs to be addressed.

Oracle crash recovery is one of the most tested mechanisms within Oracle, they have assured me of this. If you ask the Oracle Engineers about what success rate they should expect from crash recovery, they would say 100%. It must work, always, or there is a defect.

However, there are requirements to ensure this on the storage side, such as atomic processing of Oracle blocks. Block fracturing can cause a bad recovery, we ensure blocks, up to 64k, are never fractured.

All of this is important, but as you clearly pointed out, the only thing that matters at the end of the day is a recoverable database. No matter what snapshot method we implement, we recommend regular verifications (though flexclones) to ensure recoverability. Verifying crash recovery on a 300TB DB takes about 2 hours (most of the time in LUN discovery and database startup). I suspect a 1PB DB to take around 5 hrs.

Verifying crash recovery can be scripted to occur after every snapshot to ensure recoverability. I think a 5 hr verify on a 1PB db after a 10 second snapshot is quite acceptable.

It continually amazes me that some storage vendors believe their arrays will never, ever fail.

In your proposed solution above, what does RPO/RTO look like in the event of an array failure?

I shudder to think ..

-- Chuck

Hey Chuck,

I think you have NetApp confused with XIV. :)

If you keep asking softball questions like that, people are going to think we are working together on some kind of NetApp infomercial.

To answer your RPO/RTO question, the RPO/RTO is exactly the same in the above solution as, since all the snapshots are natively accessible, the secondary tier can be failed over to while the primary is repaired and resynced.

For an ultra-paranoid customer, tools like NetBackup, or even (your personal favorite) Legato can be used to backup the secondary tier snapshots directly to tape over NDMP. Of course, then you are looking at a longer RTO.

However, the parallel scale-out that allows the snapshots to scale to multi-petabytes also scales on the secondary tier for transfers to and from tape. Each secondary controller can sustain many tape transfers at once. As the solution grows and more controllers are added, the throughput capabilities to and from tape increase linearly. This would protect against the extremely unlikely event that 2 catastrophic failures occur on both the primary and secondary arrays in the same window.

Chuck is changing the subject. The question was how do you back up a 300TB database, not how to recover it. Nevertheless, it's a valid point.

If you've dumped your data to tape, you're limited to the abilities of your tape drive infrastructure. Maybe it's fast enough, maybe it's not. Depends on what your RPO/RTO is set at. Different customers have different requirements.

The simplest solution is a nice block-delta mirror like SnapMirror that can store multiple backups using the method described by Mike. If you need more, you can go with SnapVault. I've got a customer with one year of daily backups stored online. I haven't the faintest idea WHY they would do that, but it's not difficult and it's extremely space efficient.


As I read Mike's blog and subsequent comments, I don't see where he says controllers will NEVER fail. But let's take that off the table (we can discuss that later) because I really don't think that is at the heart of this blog post response.

In actuality, this is an EMC customer who is struggling to complete a backup of their 300TB DB.

Given your own customer's current situation, what is the likelihood they will complete a backup? Or did they not ask that question when they bought the V-Max so, its not up for discussion?

What Mike, myself and others have done is simply reply to the question

"How would you backup a 300TB DB?" that W. Curtis Preston posted on his blog at Backupcentral.com.

So Chuck, let's stay on point here, we've shared our answer.

What's your answer to how this customer can backup their 300TB DB?

Its a simple question.


The comments to this entry are closed.