Abstract

<p> Component failure in large-scale IT installations is be- coming an ever larger problem as the number of compo- nents in a single cluster approaches a million. <p> In this paper, we present and analyze field-gathered disk replacement data from a number of large production systems, including high-performance computing sites and internet services sites. About 100,000 disks are cov- ered by this data, some for an entire lifetime of five years. The data include drives with SCSI and FC, as well as SATA interfaces. The mean time to failure (MTTF) of those drives, as specified in their datasheets, ranges from 1,000,000 to 1,500,000 hours, suggesting a nominal an- nual failure rate of at most 0.88%. <br> <p> We find that in the field, annual disk replacement rates typically exceed 1%, with 2-4% common and up to 13% observed on some systems. This suggests that field re- placement is a fairly different process than one might predict based on datasheet MTTF. <br> <p> We also find evidence, based on records of disk re- placements in the field, that failure rate is not constant with age, and that, rather than a significant infant mor- tality effect, we see a significant early onset of wear-out degradation. That is, replacement rates in our data grew constantly with age, an effect often assumed not to set in until after a nominal lifetime of 5 years. <br> <p> Interestingly, we observe little difference in replace- ment rates between SCSI, FC and SATA drives, poten- tially an indication that disk-independent factors, such as operating conditions, affect replacement rates more than component specific factors. On the other hand, we see only one instance of a customer rejecting an entire pop- ulation of disks as a bad batch, in this case because of media error rates, and this instance involved SATA disks. <br> <p> Time between replacement, a proxy for time between failure, is not well modeled by an exponential distribu- tion and exhibits significant levels of correlation, includ- ing autocorrelation and long-range dependence.

 

Manuscript

 

Bibtex