NetApp - Dave's Blog

Dave Hitz on NetApp's Continued Leadership in Unified Storage - Part 2

 

Dave Hitz on NetApp's Continued Leadership in Unified Storage

 

Discussion Series Part 3: The Goal of Future-Ready

This is a continuation of the discussion with ESG senior analyst, Mark Peters about what’s going on in IT.  The first two posts in the discussion series can be found here and here.

Dave Hitz:    When you talk about consolidation and standardization, it seems to me the way IT is playing out is that whatever resources you have, or whatever technologies that you have, you want to be able to share them across multiple users.

That need to share drives the need for other things, as well.  If you’re going to be sharing, you need to be able to share securely.  You need to be able to share reliably. 

But in a sharing context, because you’ve got different users with external clouds, you may have different companies sharing the same resources.  It really brings a lot of other kinds of management requirements to the fore.

Mark Peters:  Sharing is another way of saying being efficient.  It’s another way of saying consolidating.

That aligns with everything I mentioned about using the minimum resources you can use to get a given piece of work, or set of works done, to the level that you need it done – still with the quality and all those other things you just mentioned, security, etc.  Sharing is absolutely the way ahead.

There are two things that are making this really crucial as you look forward.  One is growth.  The demands on IT have grown beyond anyone’s expectations.  At the same time we’ve had this massive economic malaise around the world.  The combination of those two things has made everyone go, whoa, hang on a second.  We are not running IT in a sensible fashion.  We’ve been doing it focused almost 100 percent on being effective without thinking about being efficient.

We can’t go back to that because the economic issues, and perhaps the emergence of certain technologies, has enabled us to turn, to focus on efficiency, whether that’s via long-term flexibility, sharing or some of the other things we’ve talked about. 

Dave Hitz:    What is the percentage of the overall economy that will be the IT industry?

For many years, that percentage has been going up and up and up.  People just keep spending more and more and more on IT. But you get to a certain size, and you say, we’re just not going to double our IT spend in our company from two percent to four percent, and then four percent to eight, and then eight to 16.  Then what?  We’re going to double it to 32 percent of our whole business is paying for IT?

We’ve got to stop.  It’s a little bit like health care in the US.  You look and ask what percentage of the total economy can be health care?

Are people, as part of their re-evaluation, saying, look, IT is not going to grow anymore.  In fact, maybe it should shrink a bit.

Mark Peters:    We’ve got to do more with less. What’s interesting about that is we can absolutely do that, even today.  I’m sure it’s only going to get better.  If you look in the big trends over the last few decades, we haven’t really worried about IT expense, because it’s produced so much more value for businesses than it was costing, it was easy to justify spending money there.

Whether it’s the economy or the unsustainable growth percentage of the companies that you discussed, whatever is causing this, we’ve realized it’s not that there are lower returns from IT, but like any other part of the business, it’s got to justify those, and it has to be producing results for the business.

IT has to do more – and whether it’s for less or the same, or whatever – but it has to do it in a more efficient manner.  The whole business, the whole nature, the whole architecture of IT – if you take a really big view of it over decades – has grown up in a very inefficient manner that got the job done really well.  And now we just need to tighten it on both sides of that equation.

Dave Hitz:    We’ve been talking a lot about the importance of flexibility and the importance of being ready for what comes with the future.  What is the business benefit of this in the end?

Is it really all about saving money?  Or are there other things that come out of this approach?

Mark Peters:    You’re right to raise that question.  It’s a great deal about saving money, but you remember early on when I said future flexibility, or being ready for the future is not about IT, it’s about the business.

The flip side to this is, if it’s done well, if you know where you started and you know the impact and potential impact of things you either do or might do, then what that drives to is producing money for the business.

Most people will cite that as faster deployment of applications.  Yes, that’s great.  But you need to go one stage beyond that, because that’s IT centric.

What did the faster deployment of that application buy the business in terms of bottom line - like competitive advantage or customer satisfaction, things that can be measured in terms of revenue?  That’s the flip side to this equation.


Dave Hitz:    So, what I hear you say is to do not just a good job, but a really good job at IT, you need to understand what an application means from a business perspective.  What does it do for customers?  What does it do for revenue?  What does it do for profitability?  If you as an IT person can’t make that linkage, you’re not operating at your full potential.

Thank you for your time Mark.

Mark Peters:    Thank you.  It was a great conversation.

Discussion Series Part 2: Key Issues in the Quest to be Future-Ready

This is a continuation of the discussion with ESG senior analyst, Mark Peters about what’s going on in IT.  The first post in the discussion series can be found here.

Dave Hitz:    In the quest to become Future Ready, what are the key issues that an IT organization ought to pay attention to?

Mark Peters:    The thinking and planning that I mentioned before are 1A and 1B on the list.

You have to know where you came from and where you want to go, to the best of your ability, but keep flexible so you can react.  For instance, we know there are going to be more virtual servers.  We know there’s going to be more storage.  But what are those things going to be asked to do?

The second issue is, how do you quantify what you’re getting from these various technologies or approaches? It’s easy to throw out things like speed and efficiency.  But you need to stop and think about those for just a second.

There is a discussion in the marketplace about driving cost out of the business.  It has been crucial for the last few years, and continues to be so.  Now you’re going to have this debate between using multiple vendors, the best of breed approach, if you like, and more of the collapsed stack or converged approach.

The question users need to ask themselves there is, does one make me lose as much or more as it makes me gain. Effectiveness, in my view, is getting the job done as well as you possibly can.  That’s great, and we should all do that in IT, whatever the future is, whatever the flexibility is. But efficiency is using the minimum number of resources you can to get that given work done – still to the same standards, still as effective, but doing it with fewer resources.  You can only do that if you know where you’ve come from, what you were doing, what the business benefits of all those things were, and therefore be able to quantify the improvement, because otherwise you’re just doing more of the same.

The only thing installed in every data center is inertia; unless you have a clear path that shows not only where you’re going but also where you started from.

Dave Hitz:    When you start to think about technologies, what technologies do you think are going to be involved?  Let’s imagine somebody who is just beginning the planning cycle and they’re saying, our current data center is full.  We know we’re going to have a new data center.  And we have an opportunity to think afresh about exactly what we’re going to do as we plan where we’re headed.

Mark Peters:    We talked about being future-ready, and you talked about knowing certain things that were going to happen, but we don’t know exactly how they’re going to play out.

If you’re talking about right now and the attributes that are important, you’ve got to look for things that have an ability to consolidate, an ability to use virtualization to the fullest extent possible and an ability to use standards wherever possible.

Standards are about ease.  Virtualization is about efficiency, which is what I just said is so important.  Consolidation is just about pure economics.  And overall, you’ve got to be able to manage this.

Because you’re making some decisions in the next few months, you’re going to want more of a unified platform.  That is not just a wave in the industry, but it’s a very logical approach.

You’re going to want to talk with people who are utilizing solid state and know how to do it, because, at the end of the day, solid state is all about serving (IO); in other words, getting back to that efficiency of minimum resources for given work.

The only reason you store things is because, ultimately, you may need them.  And therefore, the choice is, how frequently, how often and when am I likely to need it?  And that will determine where it goes.

If it’s about getting immediate work done, it should go on solid state.  And even though that’s a very bold statement, that’s true as you look for the medium future.  Right now we have data on different spinning disks as well because that’s the technology we have available.

The cloud is about how you consume IT and how you consume resources.  And whether that’s done behind your firewall or from a public source, it again comes back to this flexible use of resources; it comes back to consolidation, standardization, and so on.  It may not be suitable for all.  But it’s another way that, whether it’s internal or external, you can drive efficiency and leave yourself flexible.  Those are two things that are very important.

 

Stay tuned for Part 3 in the discussion series, which will cover the goals of future-ready.

 

 

Discussion Series Part 1: Thinking ‘Past-Ready’ to be Future-Ready

Dave Hitz:  I’m Dave Hitz and I’m here with ESG senior analyst, Mark Peters.  Over the course of the next three posts we’ll be talking about what’s going on in IT. What does it take for an IT organization to future-proof their environment?

Mark, good IT people are always worried about what they need to do to future-proof their environment.  That’s just part and parcel of what you do if you’re an IT person.

In the past few years it’s become especially important, given how trends are playing out.

Do you see things that way?

Mark Peters:    Yes, I do, Dave. 
       
I think it’s about planning.  That may sound too simple.  But we’re all good at thinking about growth in terms of capacity.  We’re good at thinking about the speed of our devices improving. 

But what we have to think about when we’re planning – and this gets to the future-ready aspect that you’re talking about – is about being able to change, being able to change fast, changing on the fly as business changes.

Dave, do you remember when IM came out, instant messaging?

Dave Hitz:    Yes.

Mark Peters:    I know that the Yahoo! IM team planned their maintenance between nine a.m. and five p.m., because, that was when the system wouldn’t be used much!  Now, in many businesses, they actually run on IM.  Things can change very fast.

The second point about this planning is to be future-ready.  It’s not just about the future happening to you as it evolves.  It’s about making it happen in the way that you want it to. 

In order to ‘do’ future-ready, you need to do ‘past-ready’ too, which means knowing where the heck you are now, understanding everything from utilization to power, to how many dedicated systems you’ve got.  How well are they used?  What are they producing for the business?

These are questions that, because they’re things that you’re already doing, you can answer.  And then you can talk about planning for the future.

Dave Hitz:    One of the things that’s so hard about planning for the future is you don’t know what’s going to happen.  Even though we don’t know exactly what’s going to happen, there are a few statements you can make that are straight-forward.  For instance, there are going to be a lot more virtual servers out there.

It’s not clear how many of them will be VMware and how many of them will be (hyper-V) or Xen.  There are dynamics in there that are hard to predict exactly.  But it isn’t hard to predict there will be a lot more virtualization, a lot more apps will be running in virtualized environments.

If you take that as a starting point you can do quite a bit of planning, even if you don’t know anything else about how it will play out.

Mark Peters:    Yes, I completely agree.  What you’re really talking about is the essence of what we know will happen.  But there’s the second layer, subtleties and specifics that we don’t know.

Therefore, what you need is something that is going to be flexible as you go forward.  You know you need more virtual servers.  You know pretty well how to deal with that.  But somehow within this, you’ve got to keep some flexibility.

And what’s interesting about the whole notion of flexibility, when you’re talking about IT is, in many respects – and this goes exactly to what you were just saying – it’s not about IT itself.  It’s about business.  It’s about opportunities.  It’s about applications.
   
Dave Hitz:    If you go back to the early ‘80s, when the PC was first introduced, people were saying, the PC is going to change everything, and it’s going to have a radical impact on business.  And that was true.  No question about it.

People were also saying, this will lead to the paperless office.  I haven’t noticed paperless offices in my organization.  How’s yours doing?

Mark Peters:    Exactly.  Although, the nature of the paper changes, doesn’t it?  We all print things off.  If you’re old enough like I am, sometimes you need it on a piece of paper to take it in, because you still don’t take it from the screen properly.

But, no, we haven’t gone paperless at all.

Dave Hitz:    The nature of the paper has changed.  That is what I find so interesting.  We were completely wrong about so many details.  And yet, big picture, we knew what was going to happen, and it did.

When people predict how cloud computing will play out, and how virtualization will play out, I suspect that our guesses – yours and mine as industry watchers – are about as accurate as the paperless office, which is to say, completely wrong and completely right, at the same time.

Mark Peters:    Absolutely.

There is a very interesting aspect to this I think we, as IT practitioners, need to think about.  The way we organize people and the way they’re rewarded and paid.  It’s one of those things I don’t think gets enough attention.

Now, you and I can drone on all day about flexibility, and that’s absolutely vital.  But there has to be flexibility within the way that people are measured and rewarded as well.  Because the lack of that is one of the reasons we’ve grown up with so many distributed, dedicated systems.

Everyone likes world peace.  Everyone will check the box for that, but no one wants to go and fix it.  It’s exactly the same in IT.  Everyone will check the box for ‘let’s share everything and let’s integrate everything and consolidate everything’. Oh, yes, well, except for the stuff I do.  I’d like my own server, and my own storage and my own application, thank you very much, because that’s the way I get paid, because I get measured on whether Exchange is up that day, or whatever else it might be.

It’s very simple to start a conversation about the future and flexibility.  But you really need to stop and think about what that means and how you do something about it.

Stay tuned for Part 2 in the discussion series, which will cover the key issues in the quest to be future-ready.

Three Ten-Year-Trends That Are Driving IT Change

A trick I use to predict the future is to identify ten-year-trends. These are really big trends that take place over a decade or more. At the moment, there are three ten-year-trends that interest me, but before I get into that, let me give some historical examples to clarify what I’m talking about.

In 1981, there were only an elite and nerdy few of us who had computers on our own desks. (I built my first computer in 1977, at age fourteen, but I am … different.) There were more people with some kind of remote terminal, but the majority of business people had no personal access to computers at all. Then, in 1982, IBM introduced the PC, and by 1992, ten years later, pretty much everybody in business had a computer on their desk. The ten-year-trend in IT during that decade was “put a computer in front of everybody’s face.”

During the 1990s, the ten-year-trend was “wire all of those computers into one big network.” At the beginning of the nineties, many computers weren’t networked at all, and networks that did exist were often small, connecting just a handful of computers in nearby offices. By 2000, the Internet was ubiquitous. IBM was running TV ads with nuns on camels in the middle of the desert sending e-mails to each other.

When I see lists of “this year’s top trends,” they often feel small and short-sighted to me. Any trend that only lasts a year is too insignificant to be important. It’s probably mostly over by the time you spot it. The really interesting trends, the ones that drive big change, last many years—a decade or more.

The three ten-year-trends that I see in IT today are: 1) Cloud/Outsourced Computing, 2) Server Virtualization, and 3) Flash Memory.

The big question behind cloud computing (or outsourced computing) is whether a company should build or expand its own data center, or whether it should access computing resources remotely, over the Internet. People are already using lots of cloud/outsourced computing today for things like blogging and web training, but I expect it to move up-market over time. This won’t happen overnight, but ten years from now I believe that many medium-sized businesses and a few large enterprises will have cloud-outsourced pretty much all of their computing infrastructure. (The term “cloud computing” is confusing because so many definitions are floating around. I’ll give mine in an up-coming blog.)

For people who do build data centers (either because they ignore the cloud/outsourcing trend or because they provide cloud computing to others), server virtualization will radically change how those data centers are built. Server virtualization is at the center of the trend, but it pulls a boatload of other trends along in its wake, including blade farms, network virtualization, and (my favorite) storage virtualization. Data centers built in ten years will look radically different than they look today.

Flash memory is amazing stuff. It is the first technology in decades with the potential to create a pervasive new layer in the storage hierarchy. Many technologies over the years, like bubble memory, have been touted as “disk-drive killers,” and none of them ever panned out, but flash memory is emerging as an enormous force, and there is no question that storage systems will look radically different in ten years. I’m not predicting that flash memory will eliminate disks any time soon, certainly not in five years and probably not in ten years, but I do think that it will relegate disks to an increasingly tape-like roll as flash absorbs more and more of the I/O intensive loads that disks handle today.

These are the three ten-year-trends that I see driving change in IT, but I’d love to hear from readers if they think I’ve missed some, or that the ones I’ve chosen won’t drive as much change as I think they will.

Disk Is The New Tape—Flash Is The New Disk

Almost every measure of computer performance increases exponentially. Here is one important exception: disk drives keep getting bigger, but they are not getting much faster. As a result, the number of seeks-per-second available for each gigabyte of data (seeks/second/GB) is plummeting.

Let me give a concrete example. Fifteen years ago, a typical one-gigabyte disk drive had a seek time of ten-milliseconds or so. Do that math, and that’s 100 seeks-per-second for that one gigabyte. Today, one-terabyte drives have maybe five-millisecond seek time. Do the math, and that’s 200 seeks-per-second for the whole terabyte, or just 0.2 seeks/second per gigabyte. Over the past fifteen years, your ability to access the data you own has gone down by a factor of 500.

That’s bad enough, but if you consider how many CPU instructions you can execute while you wait for the disk drive, things are even worse. Fifteen years ago, the Intel 486DX could do 54 MIPS (million instructions per second). Today Intel’s QX9770 can do 59,455 MIPS. For every millisecond you wait, today’s chips execute 1000 times more instructions.

Consider these facts together: From a human perspective, seeks/second/GB has gone down by a factor of a five-hundred, but from the CPU’s perspective, it is five-hundred-thousand times slower! Remember those old mainframe computers with tape drives on the front, jerking back and forth? To the CPU, tape drives back then seemed faster than disk drives today.

Disks are the new tape—worse than tape used to be—so we obviously need to find a new disk. That’s where flash comes in: ten times more expensive than disk, but for random access, a hundred times faster. For now, flash is mostly used in small portable devices, or for very high performance, but if you look forward five or ten years, I predict that people will think of flash the way they think of disks today—as the fast-access storage for everyday use. And they’ll think of disks as being more like tape—a slow-access media that’s useful for stuff you might want to look at someday, but not particularly useful for data you use all the time.

Flash is too expensive to replace disk right away, so first we’ll see a new generation of storage systems that combine the two: flash for performance and disk for capacity. I hesitate to compare this with the old HSM (hierarchical storage management) solutions that combined disk with tape, because those were so sucky—things that would take seconds on disk could sometimes take hours. They only worked for a limited subset of applications, and only with intricate and painful management. Fortunately, the performance ratio between flash and disk is much better than for disk and tape, so we will be able to automate the management and still get good performance. HSM without the M, if you will.

(This article is mostly about the industry in general. For more on what NetApp is doing, check here and here.)

Server Virtualization Trend: Just Starting or Almost Done?

Something has been bugging me about the market share numbers for server virtualization. Is the trend is just getting started, or is it almost finished? The numbers I’ve seen say that under 10% of all X86 servers have been virtualized – maybe 7-8%. By that measure, the trend of converting physical servers into virtual ones seems to be quite early.

Things look very different when I look at the percentage of total servers (physical plus virtual) that are virtual. Most customers seem to run at least 8-12 virtual servers per physical, and some are pushing past 30 towards 50. Let’s use 10 as a conservative number, and do the math: For every 100 physical servers, 7 are virtualized, for a total of 70 virtual servers. That makes a total of 163 total servers (70 + 93), and almost half are virtual. If we are half-way converted, then the virtualization trend must be very far along, because the second half will probably convert much faster than the first half.

It sure seems to me that looking at total servers is the right thing to do, as opposed to just counting how many physical servers are virtualized, because to a user, it shouldn’t make any difference whether their server is virtual of physical. (That’s the whole point!)

On the other hand, it doesn’t feel right that server virtualization is so far along. Most customers I talk with are just getting started. Only a few have seriously converted. My math must be busted, because there’s no way that we are half-way converted.

I think the problem is that the math assumes that there is a fixed-size pool of servers that people are converting from physical to virtual. It seems more likely that cheap and easy-to-provision virtual servers will lead to a massive increase in the total number of servers. That is always what has happened when a computing resource gets much less expensive. We didn’t just replace workstations with PCs, we gave PCs to everyone, instead of just development engineers. Likewise, the cost per gigabyte keeps dropping every year, but instead of buying fewer gigs, people keep storing more and more, and their budgets stay roughly flat.

Given the history of the computer industry, it seems unlikely that server virtualization will drive costs down in the way people think. Instead, it seems much more likely that costs will stay roughly flat, but there will be a radical proliferation in the number of virtual servers. They are just so fast, easy and cheap to deploy, it seems likely that most IT shops will hand out scads of them.

I’m not sure whether to think of this as a prediction, or a warning. I guess if you get value from all those virtual servers (just like we did from mini-computers, workstations, and then PCs), then there’s no problem. But if IT shops really want to use server virtualization to save money, then they had better be extraordinarily disciplined.

Controversy: NetApp Outperforms EMC in SAN Database Benchmark

We just released benchmark results showing that our FAS storage systems outperform EMC’s CLARiiON on SAN database workloads. For details, see Brian Pawlowski’s blog. The quick summary is that a NetApp FAS3040 beat an EMC CX3 Model 40 on SPC-1, which is an industry standard benchmark that measures OLTP (Online Transaction Processing) performance. Our system cost less, had fewer disks, and beat the EMC by 24%. With snapshots enabled on both systems, NetApp was three times faster. Here’s the chart:

Why the controversy? EMC has never posted any SPC-1 results, so we had to run the benchmark ourselves. We did follow EMC’s best practice document for the CLARiiON, and we did have the Storage Performance Council independently audit the results. But still, the fact that we ran the tests ourselves caused concern. For instance, Chuck Hollis, the VP of Technology Alliances at EMC, raised questions about the integrity of the auditor, calling him “an infamous part-time ‘administrator’ for the Storage Performance Council” and saying that “It seems to be a part time gig for him to make a few bucks.” Chuck also raised questions about why we would do this: “The only reason you'd spend the money to buy the equipment and run the tests is to put your competitor in a bad light. I think most reasonable customers would figure that part out.”

It is fair to ask why we did this, so let me share our thinking:

First, to continue improving our SAN and database credibility. Given our NAS origins in the early 1990s, many customers and analysts have been skeptical of NetApp’s SAN capabilities. Results like this show today’s reality. Steve Duplessie at the Enterprise Strategy Group commented that “Netapp appears to have legit block performance, and shouldn't be dismissed because people (like me) presume it can't be true.” Chuck Hollis is a vocal skeptic of NetApp’s ability to play in SAN and database environments, so it shouldn’t surprise him that we want to refute his claims. (See here for a brief history of our benchmarking efforts over time.)

Second, to showcase our snapshot performance. Snapshots help customers improve backups and archive old data, and writable snapshots (FlexClones) let customers completely rethink their database test and development strategy. Unfortunately, snapshots in most storage systems are unusably slow. With NetApp, performance dropped only 3% with snapshots on. With EMC, performance dropped by a factor of three. I freely admit that our goal in focusing on snapshot performance was to “put our competitor in a bad light.” I think that’s fair because EMC’s snapshots really are painfully slow in real-world use. On the other hand, we expected our non-snapshot performance to be about the same, or maybe even a bit lower, given that our system is less expensive, has fewer disks, and uses RAID-6 instead of mirroring. Winning there was a pleasant surprise.

Third, because Chuck asked us to. In his blog entry on SPC, Chuck said: “We've never done an SPC test, and probably will never do one. Anyone is free, however, to download the SPC code, lash it up to their CLARiiON, and have at it.” I don’t promise always to follow Chuck’s advice, but I think it’s important to recognize good ideas no matter where they come from!

One key take away from this result is that turning on a simple feature like snapshots can radically change performance. Don’t let a bad experience with EMC’s snapshots scare you away from NetApp’s.

Let me close with a final word on benchmarks. Any honest vendor will agree that benchmark results are sometimes misleading, and that you should examine the details carefully. We believe that SPC-1 effectively simulates OLTP workloads, and we used real-world configurations based on each company’s own best practices documentation. But despite our best efforts, I stand by my argument in a previous blog entry that you should admire and respect great benchmark results, but also be careful.

 

 

The following SPC-1 results have been posted at www.storageperformance.org:

NetApp FAS3040 (baseline):
http://www.storageperformance.org/results/benchmark_results_spc1#a00057

NetApp FAS3040 (with Snapshots):
http://www.storageperformance.org/results/benchmark_results_spc1#a00058

EMC CLARiiON CX3 Model 40 (baseline):
http://www.storageperformance.org/results/benchmark_results_spc1#a00059

EMC CLARiiON CX3 Model 40 (with SnapView):
http://www.storageperformance.org/results/benchmark_results_spc1#a00060

All comparisons are current as of January 29, 2008.

 

Test and Development Copy of an Oracle Database: 2400% Speed Up

Niel Armstrong is the CIO of Activision, and we gave a talk together at Oracle Open World. Niel uses pretty much the full suite of Oracle products, including the database itself, much of the middleware stack, the eBusiness Suite, and Hyperion. He also uses pretty much the full suite of NetApp products.

My favorite story was about the speed up Niel saw in making copies of his production database for test and development. Before he started working with NetApp, test and dev copies were a real pain. It took a “long week” to create a full copy. (A “long week” is when you start at close of business Friday, keep working for the whole week, and don’t finish till Sunday night the weekend after. Or maybe Monday morning.) It was so painful that they did a process-re-engineering project that drove the time from roughly nine days down to four. Niel said, “We felt great about cutting the time in half. This was a big project for us, and we worked really hard at it.”

When Niel switched to NetApp storage, he started using NetApp’s copy-on-write clones (FlexClone) to make test and dev copies, and he cut the time from 4 days to 4 hours. Actually, he said it’s usually 2 hours, but he says 4 just to give himself some buffer.

This completely changed their workflow model. Niel said, “Before NetApp, we used to really fight these clones. We just didn’t want to do them. No more than one or two per large Oracle project if possible. Now we do them all the time. I’ve got over a dozen clones right now just for the UK Oracle implementation. One guy wanted a clone for a training class, and we gave it to him. We never would have done that before. It would have taken too long and used up too much storage.”

Clones are handy during normal operations, but they are especially important when you are making a major change to your environment. “Without NetApp clones, we would never have finished our upgrade in time.”

What I love most about Niel’s story is that it is almost exactly the same thing I wrote about last year after Oracle Open World (read here), except then I focused on the demo we were giving in our booth, and now I’m describing a customer who is in production and who stood on stage next to me to describe his experiences.

Let me give you a sense of Niel’s environment. Activision has game design studios all around the world. A big problem was getting reliable backups at the studios, so they implemented remote disk-to-disk copies (SnapMirror) from each studio to their disaster recovery center in Burbank, as well as replicating all of the headquarters data. A few Sundays ago, disaster recovery was “tested” by two back-to-back power failures, courtesy of SoCal Edison. They recovered in less than 4 hours, with no business interruption, which earned them attaboys from both the Chairman and the CEO.

I asked Niel whether he mostly makes clones from his DR systems, or his production systems, or what. His response: “We make clones pretty much everywhere. It’s clones all over.”