SSD reliability in the real world: Google's experience

[Arthur]

Google's SSD reliability paper isn't available online until tomorrow, but in the interim ZDNet has an article containing a summary of the key points. It's quite interesting.

 

www.zdnet.com/article/ssd-reliability-in-the-real-world-googles-experience

 

SSD reliability in the real world: Google's experience

SSDs are a new phenomenon in the datacenter. We have theories about how they should perform, but until now, little data. That's just changed.

 

The FAST 2016 paper Flash Reliability in Production: The Expected and the Unexpected, (the paper is not available online until Friday) by Professor Bianca Schroeder of the University of Toronto, and Raghav Lagisetty and Arif Merchant of Google, covers:

 

• Millions of drive days over 6 years
  
• 10 different drive models
  
• 3 different flash types: MLC, eMLC and SLC
  
• Enterprise and consumer drives
  

KEY CONCLUSIONS

• Ignore Uncorrectable Bit Error Rate (UBER) specs. A meaningless number.
  
• Good news: Raw Bit Error Rate (RBER) increases slower than expected from wearout and is not correlated with UBER or other failures.
  
• High-end SLC drives are no more reliable that MLC drives.
  
• Bad news: SSDs fail at a lower rate than disks, but UBER rate is higher (see below for what this means).
  
• SSD age, not usage, affects reliability.
  
• Bad blocks in new SSDs are common, and drives with a large number of bad blocks are much more likely to lose hundreds of other blocks, most likely due to die or chip failure.
  
• 30-80 percent of SSDs develop at least one bad block and 2-7 percent develop at least one bad chip in the first four years of deployment.
  

THE STORAGE BITS TAKE

Two standout conclusions from the study. First, that MLC drives are as reliable as the more costly SLC "enterprise" drives. This mirrors hard drive experience, where consumer SATA drives have been found to be as reliable as expensive SAS and Fibre Channel drives.

 

One of the major reasons that "enterprise" SSDs are more expensive is due to greater over-provisioning. SSDs are over-provisioned for two main reasons: to allow for ample bad block replacement caused by flash wearout; and, to ensure that garbage collection does not cause write slowdowns.

 

The paper's second major conclusion, that age, not use, correlates with increasing error rates, means that over-provisioning for fear of flash wearout is not needed. None of the drives in the study came anywhere near their write limits, even the 3,000 writes specified for the MLC drives.

 

But it isn't all good news. SSD UBER rates are higher than disk rates, which means that backing up SSDs is even more important than it is with disks. The SSD is less likely to fail during its normal life, but more likely to lose data.

 

I'll be digging deeper into the data this weekend. Stay tuned!

[dhicks]

ZDNet has an article containing a summary of the key points.

 

Hacker News' discussion raises some valid points:

 

https://news.ycombinator.com/item?id=11188445

 

When Google say "SSD drive" they don't mean something they bought from Amazon, they buy memory chips direct from the factory and make their own drives with their own firmware. These aren't drives inside laptops and similar, these are in datacentres, typically running at around 35 degrees celcius. I also doubt most of those drives are in a RAID array, Google designes its own high-availability storage architecture, and has moved on rather from RAID arrays.

 

Reading that article and various comments, I think we're doing okay with our array of "consumer" Sandisk Ultra II SSDs currently in one of our servers. This is in an HP server with an HP Smart Array RAID card, updated to the latest firmware version. In particular, that firmware update has features to let the RAID array use SSDs properly. Regarding the article above, the comment about over-provisioning seems relevant. Over-provisioning in case of data loss doesn't seem to be worth the bother (SSD drives seem to fail by age rather than by usage), but having over-provisioning in place to avoid write slowdown when the drive itself does garbage collection might be handy, depending on application. Our RAID controller seems to let you specify over-provisioning controlled by the RAID controller, so you don't have to worry about the feature on the drives.

[Arthur]

I also doubt most of those drives are in a RAID array

I highly doubt it too, especially since Google wants hard drive manufacturers to redesign 3.5" disks for their datacentres.

 

Google doesn't care if the next generation of hard drives is less reliable

 

It's pretty terrible when a hard drive crashes and loses your data, but Google says that manufacturers should stop worrying about it so much in the future. That's because most spinning hard drives will soon be used for cloud storage — not personal storage in laptops or mobile devices — so a hard drive's reliability matters less than its price and storage capacity. "This shift has a range of interesting consequences including the counter-intuitive goal of having disks that are actually a little more likely to lose data, as we already have to have that data somewhere else anyway," Eric Brewer, Google's infrastructure VP, writes in a blog post.

 

Brewer writes that it's becoming more important to design hard drives that are meant to be used in large groups, rather than on their own. Reliability was important when the key use for a hard drive was storage in desktop and laptop computers. But most personal devices are shifting over to solid state drives now, which are typically faster and more reliable. They're much more expensive, though, which is why Google is still interested in optimizing traditional hard drives for use in its data centers.

 

Google is releasing a paper today with suggestions on how to improve future hard drives. Brewer says they range from "physical changes, such as taller drives and grouping of disks" to "shorter-term" firmware changes. "We hope this is the beginning of a new era of 'data center' disks," the paper's authors write.

[Arthur]

Here's the actual paper...

 

www.usenix.org/conference/fast16/technical-sessions/presentation/schroeder