Many readers won’t remember SpinMove since it was only on the market for about a year before NetApp bought Spinnaker. SpinMove enabled admins to non-disruptively move volumes between storage nodes in a Spinnaker cluster. Of all the innovation that Spinnaker brought to market, SpinMove generated the most customer interest. Many customers bought Spinnaker storage for this feature alone.

Avere’s FlashMove is a lot like SpinMove in that it allows data to be non-disruptively moved between NAS systems but has two significant advantages. First, FlashMove is integrated with Avere’s native tiering and this means you get transparent data migrations *and* performance acceleration in the same solution. Second, FlashMove works with heterogenous NAS solutions so there is no lock-in to a single brand of storage.

Check out the FlashMove data sheet for more info.

NetApp’s SnapMirror, on the other hand, has become the gold standard for asynchronous mirroring to provide a disaster recovery (DR) solution. NetApp has a lot of great products and this is probably their best. Unfortunately, not everyone gets to use SnapMirror since it only works on NetApp storage.

Avere’s FlashMirror has several significant advantages over SnapMirror. First, FlashMirror is more efficient and keeps replicated data more closely in sync by sending updates directly and in parallel to both the primary and secondary NAS filers. Second, FlashMirror offloads the replication-processing load from the storage and supports clustering to scale replication performance to any level required. Third, FlashMirror is simple to install in existing environments and works with heterogeneous NAS solutions.

Check out the FlashMirror data sheet for more info.

The Problem

First, a little background on why boot storms even exist. When sizing storage for both performance and capacity to serve in a VDI (Virtual Desktop Infrastructure) environment, the steady-state of  virtual machine operation dictates the sizing requirements. What happens when all of your users come in on a Monday morning and power on their VDI terminals?

It certainly won’t look like “steady-state” operation to your storage. Instead, you potentially end up with hundreds or thousands of virtual desktops running on a farm of Virtual Machine Hypervisor servers (i.e. VMware ESXi) all trying to read their operating systems from their Virtual Machine Disk (.vmdk) files all at once. Most modest NAS storage devices will leverage their RAM memory cache as much as possible and then rely on the number of storage spindles to serve the cache-miss IOPS. With enough cache-misses, the disks will start to run incredibly hot, leading to queuing, which leads to longer boot times, which leads to user complaints that their virtualized desktops take forever to boot.

One way to think about this problem is to consider the pre-virtualization landscape: Every user had their own desktop hardware with dedicated CPU, RAM, and storage. A desktop computer would generally take 2-5 minutes to boot, depending on the operating system environment and machine capabilities. In this scenario, most of the desktop boot time was spent by the CPU waiting for I/O from the hard-disk to provide the required data to load the kernel, device drivers, graphical user interface environment, etc. With a single SATA hard-disk capable of cranking out 250 IOPS while still delivering a satisfactory boot time, the machine ends up wasting a lot of CPU time “waiting.” After the desktop is booted and required applications are running, there are only occasional requests for disk I/O. This can be considered steady-state operation. The same principle behind virtualization applies here: why waste a physical resource by leaving it idle or unused?

As server virtualization technology has evolved, one can now leverage all that CPU time that was spent waiting for disk I/O to be utilized by another virtual machine that is hosted on the same piece of virtualized hardware. The beauty of this is that the end-user’s boot-time experience is no different, except now, you can have multiple virtual machines sharing to maximize the utilization of the available hardware resources by virtualizing RAM, storage and networking on an ESXi host. Sharing is wonderful and great, up until the point that everyone needs their fair-share of CPU/RAM/Disk resources, all at the same time.

This shouldn’t be news to anyone, as this is the foundation upon which the virtualization craze has so successfully built upon: maximizing resource utilization for efficiency. So all you need to do now is bring up an ESXi server that has 16 CPU cores, 128GB of memory, 10Gigabit ethernet, and you can host fifty or one-hundred virtual machines depending on their requirements. With 100 virtual machines running on a ESXi server, you’re going to need some storage for all these guys.

Shared SAN or NAS storage is the way to go here, to leverage cool features and tools hypervisors like VMware vCenter offer, to ease the pain of managing thousands of virtual machines. Thin-provisioning and sparse Datastores allow you to over-provision your storage (another big win,) so you end up buying a 20TB NAS with 20 spindles of 7200RPM SATA drives. That’ll give you a solid 5,000 IOPS to handle all of your I/O, should be fine right?

If you’re running 100 VMs against that, sure, you get 50 IOPS per VM which should be more than enough for shared steady-state. When you grow to 400 VMs, now you’re contending with 12.5 IOPS for each VM, which some may still consider acceptable for steady-state operation. When one of those ESXi servers hosting 100 VMs needs to be rebooted, all the guest VMs are going to be hungry for their own share of 250 IOPS to achieve boot times in the 2-minute range.

Let’s do the napkin math: 100 VMs X 250 IOPS/VM = 25,000 IOPS to achieve the status-quo boot time of 2 minutes. With a storage array that can handle 5,000 IOPS, you’re looking at boot times that are 5x longer, i.e. 10 minutes. The other 300 users that are on other ESXi servers sharing the same storage are now unable to get their ops through fairly either. So everyone ends up suffering.

Welcome to a boot storm. Those VMs that must reboot end up waiting 10 times longer, and those steady-state VMs that share the storage are going to be pretty much dead in the water for the next 10 minutes. Time? Aggravation? You bet!

The Test

Below, is a chart showing the results of up to 48 virtual machines booting up simultaneously. The test environment was pretty straightforward: Measure the amount of time that Windows 7 virtual machines, each with 2GB of memory and 16GB of virtual disk space, take to reach a point where they’re ready to launch an application. We gathered numbers as progressively greater numbers of  virtual machines were booting up in parallel.

We tested against against two different NetApp filers. An entry-level NetApp filer (FAS2050) with 12 spindles of SATA 7200rpm disk and a mid-range NetApp filer (FAS3240) with 12 spindles of SATA 7200rpm disk. The entry-level NetApp is typically the filer that a customer would choose when starting out small. However as one’s virtualized environment starts to grow, this platform will often prove to be inadequate.  The FAS2000 series is not upgradeable with a FlashCache card, so the next step up would be to go with a bigger filer that has more CPU and RAM like the FAS3240. Lastly, the option to add an Avere FXT2550 to an entry-level NAS environment was tested. The numbers speak for themselves:

The Results

A single virtual machine booting up, took about 38 seconds flat, across all storage platforms that were tested. The NetApp FAS2050 started to show signs of degrading boot times once 16 or more VMs were booting up simultaneously. At this load point of 16 VMs, boot times had increased by about 113% over the baseline boot time of 38-seconds provided by the FAS2050. Once simultaneously booting up 48 virtual machines, the boot times grow to 246-seconds, or about 550% higher.

With no option to add a NetApp FlashCache to the FAS2000 series, the next step is a forklift upgrade to a FAS3240, which is a mid-range platform that has more room for expansion. However, upgrading the Filer head alone will not yield a noticeable improvement: booting 16 VMs on the FAS3240 only took about 107% more time than the baseline single-VM boot. So now you’ve spent a good chunk of change on a brand-new filer that hasn’t really improved things significantly from the boot-storm mitigation perspective.

Instead, take that forklift-upgrade budget and roll in an Avere FXT 2550 in front of the FAS2050, and solve the boot storm problem. With an FXT 2550 in place for the 48 VM boot storm, testing revealed that boot times stay below 70-seconds. This is almost four times lower than the boot time for 48 VMs using the FAS3240 Filer. Additional testing has shown that the Avere FXT 2550 is capable of handling 145 VMs simultaneously booting before a reaching a saturation point where boot times grew larger than 90-seconds. Even then, once you’ve saturated a single FXT node, you can simply add more nodes to the FXT cluster and scale out your boot storm coverage with FXT nodes as incremental building blocks. The Avere architecture assures that the performance for this workload can scale out as you add more FXT nodes to the cluster, while maintaining the simplicity of managing a single filer.

Avere Systems shocked the storage world today and took the top NFS performance spot on the SPECsfs2008 benchmark, taking down the big dogs, NetApp and EMC/Isilon, in the process. Avere posted throughput of 1,564,404 ops/sec, which is the highest ever posted in the long history of the NFS benchmark. In addition, this throughput was achieved with an ORT (overall response time or latency) of just 0.99 msec, which is 35% better than NetApp’s best and 61% better than EMC’s best.

For more details on the performance tests by the three vendors, here are links to the posted SPECsfs2008 results from Avere, NetApp, and EMC/Isilon.

The performance that Avere demonstrated is impressive but it is only half the story. Even more impressive is the efficiency with which Avere delivered the results.

Avere delivered higher performance and lower latency with a system that costs dramatically less, both in terms of the capital expenses to purchase the system and operating expenses for space, power, cooling, etc.

I will take you through the numbers in a second but first let’s take a look at pictures that compare the storage systems that were tested by Avere, NetApp, and EMC/Isilon.

As you can see, Avere packs the highest performance and lowest latency into a package that is 79% smaller than NetApp and 65% smaller than EMC/Isilon.

Overall size is an approximate measure of the capital and operating expenses. Let’s dig deeper into the actual numbers. In the below table I have included the pertinent comparison data. As you can see from the below, Avere is 51-77% less cost, requires 56-78% fewer disks, and occupies 64-76% fewer rack units.

SPECsfs2008 does not measure power or cooling requirements. In a storage system, the disk drives are the largest consumers of power and dissipaters of heat. Therefore, a good estimate for the power and cooling savings is the disk savings, where Avere is 56-78% less.

Prior to Avere’s SPECsfs2008 posting, NetApp and EMC/Isilon waged a war of words contrasting their SPECsfs2008 results in the body and comments of this blog. It’s a highly recommended read. Make sure to read the comments. With the Avere results now out, the NetApp and EMC/Isilon battle is over 2nd and 3rd place, with Avere taking 1st in all the major categories.

Hope to see you at SC11.

SPEC® and the benchmark name SPECsfs®2008 are registered trademarks of the Standard Performance Evaluation Corporation. Competitive benchmark results stated above reflect results published on www.spec.org as of Nov 15, 2011. Above we compare all SPECsfs2008_nfs.v3 results that achieved 1,000,000 ops/sec throughput or higher. For the latest SPECsfs2008 benchmark results, visit http://www.spec.org/sfs2008.

Avere’s NAS Optimization enables using the cloud for primary applications. The Avere FXT Series uses intelligent tiering to automatically store active data near the client, eliminating the latency of the WAN. Customers are using our cloud solution in four data access scenarios:

1) Remote Office to Datacenter: Data storage is consolidated in a centralized datacenter and Avere is used at the edge to provide low-latency access to remote users.

2) Datacenter to Datacenter: Avere enables compute resources to be shared across multiple datacenters by automatically placing the data actively being processed near the compute nodes and eliminating the WAN latency between the datacenters.

3) Enterprise to Compute Cloud: Enterprises are deploying lower cost compute infrastructures, both in private cloud and public cloud models, by co-locating Avere clusters with the compute nodes to automatically tier and store the active data.

4) Enterprise to Storage Cloud: Avere enables storage clouds for primary applications by placing Avere clusters near the clients, whether in a datacenter, a remote office, or a cloud compute facility, to automatically tier and store the data that the client is actively using.

Check out our cloud solution brief for more.

Enterprise-wide NAS Cloud

• VMware is a write-heavy app and Avere’s write caching provides a huge performance boost.
• Avere block-level caching efficiently uses SSD/SAS tiers of FXT clusters, especially when using VMware Linked Clones.
• Avere’s NAS Optimization is faster than SAN and simpler to manage and scale.
• The Avere user interface provides great visibility into VMware operations, including ESX hosts, VMs, and VMDKs.
• Storage VMotion makes adding FXT clusters in front of existing NAS systems simple and non-disruptive.
• Storing VMDKs on inexpensive & high-density SATA and accelerating with Avere is much cheaper than storing all VMDKs on SAS/FC.

Check out our VMware solution brief for more.

Avere is VMware Ready certified.