Facebook SSD Reliability Study

[Arthur]

Source: Facebook (via ZDNet / The Tech Report)

 

A team comprised of Facebook engineers and Carnegie Mellon researchers has published a study covering flash memory failure rates over a four-year period, with data collected from SSDs in Facebook's servers. The paper doesn't tie the statistics gathered to particular makes and models of drives, but the results are still informative.

 

The team collected data straight from the drives' hardware rather than relying on OS reporting, thus measuring the amount of data that was actually written to the flash cells. Somewhat parallel to the findings in Google's 2007 paper on mechanical hard drives, the SSD study reveals that failures tend to occur early, followed by a period of smooth sailing up until the end of the drive's useful life.

 

However, there's a key difference: the team identified what they called an "early detection period." When SSDs are still young, their initial usage leads the controller to immediately identify which flash cells are unreliable. This is further corroborated by the fact that higher-capacity SSDs see relatively the same percentage of failed flash cells in this initial period—and that the window for this early detection is shorter in the smaller drives.

 

The study also shows that temperature has a direct impact on flash memory reliability, with drives running at lower temperatures or using more aggressive throttling mechanisms displaying comparatively fewer cell failures. Interestingly, the amount of data read seems to play no part at all when determining a cell's lifetime.

 

What They Found

The good news: some issues that worry people, aren't issues. The bad news: there's other stuff to worry about.

 

Temperature

SSDs are sensitive to temperature - more so than hard drives. When they get hot, the SSD may throttle back performance. Unexplained slowdowns on some servers: check temperature.

 

The first gen SSDs failed more often as temp rose, possibly due to a lack of throttling. Some second gen SSDs throttle aggressively enough to reduce failure rates, while others kept the failure curve flat.

 

Bus power

SSDs are thirsty. PCIe v2 SSDs ran anywhere from 8 to 14.5 watts, a high and surprisingly wide range. The team found that as power consumption rose, so did failure rates.

 

Write fatigue

The team found that the level of system write activity correlated with SSD failure, probably because flash writes require a lot of power. Disks could be a better choice for heavy write applications such as logging.

 

SSD failures

SSD failures - i.e. UREs - are relatively common: 4.2 to 34.1 percent of the SSDs reported uncorrectable errors. In fact, 99.8 percent of the SSDs reporting an error in one week reported another error in the next week.

 

Life cycle and failures

The SSD failure profile differs from disks, where the latter exhibit infant mortality, then a few years of good reliability, before age catches up with them. SSDs have an early period of UREs as faulty cells are identified, increasing reliability, until cell wear-out leads to increasing read failures.

 

The data layout surprise

Disk drives aren't much affected by data layout - unless it involves lots of random seeks. But SSDs, very different.

 

Sparse logical data layouts - non-contiguous data - lead to higher SSD failure rates as do very dense data structures. My reading: problems in the logical-to-physical address logic in SSD controllers. Update: Alert reader Wilback noted that the paper theorized that "Such behavior is potentially due to the fact that sparse data allocation can correspond to access patterns that write small amounts of non-contiguous data, causing the SSD controller to more frequently erase and copy data compared to writing contiguous data."

 

http://vgy.me/Y0jyoo.jpg

 

http://vgy.me/2GOxGv.jpg

Edited June 25, 2015 by Arthur

[mats]

Finally - a use for Faceache! The worlds biggest drive endurance test system.