File corruption on my web server

Got a bug report for windhistory.com that I traced back to the main JSON database I’m serving being corrupted. The file hasn’t been touched since September 2011 and I know it was working correctly as of December 1. Needless to say this is very disturbing. Files shouldn’t go bad like this. The pattern of corruption doesn’t suggest malice, just a bug. It makes me wonder what else is corrupted. None of the other datafiles for that project have changed but I can’t easily audit the entire system.

My backups didn’t capture the corruption. But I got clever and realized I could pinpoint when the file changed by looking at the compressed size that Apache was serving; even though the file itself wasn’t truncated, the data changed so the compressed size changed too. It’s typically 369275 or 369276 bytes (why the inconsistency?) but starting at 11/Dec/2012 18:25:58 (UTC) the length 370464 or 370465 started showing up. At first it was really inconsistent, it didn’t stabilize on this new length until 12/Dec/2012 01:05:25.

Nothing obvious in the syslog around that time. Munin graphs show some other weirdness; Munin processing time went up to timeout levels for awhile, HTTP loadtime stopped being reported, NTP briefly went from a 40ms delay to a 90ms delay from true time. A disturbance in the force. But no temperature or voltage spikes, no heavy load, nothing major.

I flushed my filesystem caches via echo 3 >| /proc/sys/vm/drop_caches and now the file is correct. So at least the disk didn’t get corrupted, just the cached copy in memory. Still disturbing, this is way more than a single cosmic ray bit flip. But it does look like a systemic modification of the old bits, not completely overwritten with  new ones. Very weird.

Here’s the output of cmp -b -l; index, then good data on the left, corrupted data on the right. There’s some mild pattern to it, it looks in general like bits are being flipped to 0. But not entirely. And why is the corruption scattered around the file? Very weird.

 188486  71 9     41 !
 188494  54 ,     44 $
 188502  63 3     41 !
 188510  54 ,     44 $
 188518 135 ]      5 ^E
 188526  62 2     40
 188534  54 ,     44 $
 188542  62 2     40
 188614 133 [      1 ^A
 188622  62 2     40
 188630  62 2     40
 188638  54 ,     44 $
 188646 133 [      1 ^A
 188654  61 1     41 !
 188662  54 ,     44 $
 188670  60 0     40
 188742  61 1     41 !
 188750  54 ,     44 $
 188758  61 1     41 !
 188766  54 ,     44 $
 188774  54 ,    144 d
 188782  60 0     40
 188790  64 4     44 $
 188798 135 ]      5 ^E
 188870  54 ,     44 $
 188878  54 ,     44 $
 188886  63 3     41 !
 188894  63 3     41 !
 188902  62 2     40
 188910  54 ,     44 $
 188926  63 3     41 !
 188998  54 ,     44 $
 189006  54 ,     44 $
 189014  65 5     45 %
 189022  60 0     40
 189030  66 6     44 $
 189038  70 8     40
 189046  54 ,     44 $
 189054  71 9     41 !
 189126  54 ,     44 $
 189134  54 ,     44 $
 189142  63 3     41 !
 189150  63 3     41 !
 189158  61 1     41 !
 189166  62 2     40
 189174  54 ,     44 $
 189182  54 ,     44 $
 189254  66 6     44 $
 189262 120 P      0 ^@
 189270 122 R      0 ^@
 189278 105 E      5 ^E
 189286 122 R      0 ^@
 189294  60 0     40
 189302  66 6    144 d
 189310  61 1     41 !
 189382  54 ,     44 $
 189390  54 ,     44 $
 189398  54 ,     44 $
 189406  54 ,     44 $
 189414  54 ,     44 $
 189422  54 ,     44 $
 189430  54 ,     44 $
 189438  63 3     41 !
 189510  64 4     44 $
 189518  54 ,     44 $
 189526  61 1     41 !
 189534  66 6     44 $
 189542  54 ,     44 $
 189550  54 ,     44 $
 189558  54 ,     44 $
 189566  63 3     41 !
 189638  54 ,     44 $
 189646  54 ,     44 $
 189654  54 ,     44 $
 189662  54 ,     44 $
 189670  54 ,     44 $
 189678 113 K      1 ^A
 189686  61 1     41 !
 189694  54 ,     44 $
 189766  71 9     41 !
 189774  63 3     41 !
 189782  54 ,     44 $
 189790  54 ,     44 $
 189798  66 6     44 $
 189806  64 4     44 $
 189814  61 1     41 !
 189822  54 ,     44 $
 189894  70 8     40
 189902  67 7     45 %
 189910  61 1     41 !
 189918  54 ,     44 $
 189926  54 ,     44 $
 189934  54 ,     44 $
 189942  67 7     45 %
 189950  64 4     44 $
 190022  63 3     41 !
 190030  61 1     41 !
 190038  67 7     45 %
 190046  60 0     40
 190054  62 2     40
 190062  61 1     41 !
 190070  54 ,     44 $
 190078  54 ,     44 $
 190150 117 O      5 ^E
 190158 125 U      5 ^E
 190166 101 A      1 ^A
 190174  54 ,     44 $
 190182  60 0     40
 190190  54 ,     44 $
 190198  63 3     41 !
 190206  54 ,     44 $
 190278  64 4     44 $
 190286  63 3     41 !
 190294  66 6     44 $
 190302  54 ,     44 $
 190310  54 ,     44 $
 190318  54 ,     44 $
 190326 135 ]      5 ^E
 190334  62 2     40
 190406  60 0     40
 190414  62 2     40
 190422  64 4     44 $
 190430  54 ,     44 $
 190438  54 ,     44 $
 190446  61 1     41 !
 190454  67 7     45 %
 190462  64 4     44 $
 190534  63 3     41 !
 190542 135 ]      5 ^E
 190550  42 "     40
 190558  65 5     45 %
 190566  67 7     45 %
 190574 141 a     41 !
 190582 145 e     45 %
 190590  54 ,     44 $
 190662  54 ,     44 $
 190670  54 ,     44 $
 190678  65 5     45 %
 190686 133 [      1 ^A
 190694  61 1     41 !
 190702  61 1     41 !
 190710  64 4     44 $
 190718  63 3     41 !
 190790  63 3     41 !
 190798  54 ,     44 $
 190806 135 ]      5 ^E
 190814  54 ,     44 $
 190822  54 ,     44 $
 190830  64 4     44 $
 190838  54 ,     44 $
 190846  61 1     41 !
 190918  67 7     45 %
 190926  67 7     45 %
 190934  54 ,     44 $
 190942  54 ,     44 $
 190950  54 ,     44 $
 190958 135 ]      5 ^E
 190966  42 "     40
 190974  65 5     45 %
 191046  60 0     40
 191054  60 0     40
 191062  60 0     40
 191070  60 0     40
 191078  60 0     40
 191086  60 0     40
 191094  60 0     40
 191102  60 0     40
 191174  54 ,     44 $
 191182  54 ,     44 $
 191190  54 ,     44 $
 191198  63 3     41 !
 191206  54 ,     44 $
 191214  54 ,     44 $
 191222  54 ,     44 $
 191230  64 4     44 $
 191302  61 1     41 !
 191310  61 1     41 !
 191318  54 ,     44 $
 191326  62 2     40
 191334  54 ,     44 $
 191342  54 ,     44 $
 191350  54 ,     44 $
 191358  62 2     40
 191430 101 A      1 ^A
 191438  42 "     40
 191446  64 4     44 $
 191454  61 1     41 !
 191462  65 5     45 %
 191470 135 ]      5 ^E
 191478  54 ,     44 $
 191486  54 ,     44 $
 191558  61 1     41 !
 191566  61 1     41 !
 191574  54 ,     44 $
 191582  70 8     40
 191590  67 7     45 %
 191598  54 ,     44 $
 191606  54 ,     44 $
 191614  54 ,     44 $
 191686  54 ,     44 $
 191694  54 ,     44 $
 191702  60 0     40
 191710  70 8     40
 191718  61 1     41 !
 191726  54 ,     44 $
 191734  70 8     40
 191742  66 6     44 $
 191814 130 X      0 ^@
 191822  71 9     41 !
 191830  64 4     44 $
 191838 117 O      5 ^E
 191846 111 I      1 ^A
 191854 133 [      1 ^A
 191862  62 2     40
 191870  63 3     41 !
 191942  61 1     41 !
 191950 135 ]      5 ^E
 191958  71 9     41 !
 191966  63 3     41 !
 191974  54 ,     44 $
 191982  54 ,     44 $
 191990  54 ,     44 $
 191998  54 ,     44 $
 192070  61 1     41 !
 192078 133 [      1 ^A
 192086  61 1     41 !
 192094  62 2     40
 192102  60 0     40
 192110  63 3     41 !
 192118  54 ,     44 $
 192126  54 ,     44 $
 192198  61 1     41 !
 192206  62 2     40
 192214  54 ,     44 $
 192222  54 ,     44 $
 192230  71 9     41 !
 192238  54 ,     44 $
 192246 133 [      1 ^A
 192254  64 4     44 $
 192326  54 ,     44 $
 192334  54 ,     44 $
 192342  64 4     44 $
 192350  54 ,     44 $
 192358  61 1     41 !
 192366 135 ]      5 ^E
 192374  54 ,     44 $
 192382  60 0     40
 192454  54 ,     44 $
 192462 133 [      1 ^A
 192470  60 0     40
 192478  71 9     41 !
 192486  62 2     40
 192494  62 2     40
 192502  54 ,     44 $
 192510  54 ,     44 $
 356422  54 ,     44 $
 356430  61 1     43 #
 356438  54 ,     44 $
 356446  54 ,    144 d
 356454  60 0     40
 356462  71 9    141 a
 356470  54 ,     44 $
 356478  54 ,     44 $
 356550  54 ,     44 $
 356558 135 ]      5 ^E
 356566  61 1     41 !
 356574  61 1     41 !
 356582  54 ,     44 $
 356590 133 [      3 ^C
 356598  61 1     41 !
 356606  54 ,     44 $
 356678 135 ]      5 ^E
 356686  61 1     41 !
 356694  61 1     41 !
 356702  54 ,     44 $
 356710 133 [      1 ^A
 356718  60 0     40
 356726  54 ,     44 $
 356734  54 ,     44 $
 356806  54 ,     44 $
 356814  54 ,     44 $
 356822  62 2     40
 356830  54 ,     44 $
 356838  54 ,     44 $
 356846  64 4     44 $
 356854  64 4     44 $
 356862  65 5     45 %
 356934 105 E      5 ^E
 356942 116 N      4 ^D
 356958  54 ,     44 $
 356966  54 ,     44 $
 356974  54 ,     44 $
 356982  54 ,     44 $
 356990  54 ,     44 $
 357062  54 ,     44 $
 357070  54 ,     44 $
 357078 133 [      1 ^A
 357086  64 4     44 $
 357094  61 1     41 !
 357102  54 ,     44 $
 357110  54 ,     44 $
 357118  54 ,     44 $
 357190  66 6     44 $
 357198  61 1     41 !
 357206 135 ]      5 ^E
 357214  65 5     45 %
 357222  66 6     44 $
 357230  61 1     41 !
 357238 135 ]      5 ^E
 357246  63 3     41 !
 357318  54 ,     44 $
 357326  63 3     41 !
 357334 135 ]      5 ^E
 357342 117 O      5 ^E
 357350  60 0     40
 357358  64 4     44 $
 357366 105 E      5 ^E
 357374 114 L      4 ^D
 357446  54 ,     44 $
 357454 135 ]      5 ^E
 357462  66 6     44 $
 357470  61 1     41 !
 357478  54 ,     44 $
 357486 133 [      1 ^A
 357494  64 4     44 $
 357502  54 ,     44 $
 357574  64 4     44 $
 357582  54 ,     44 $
 357590  54 ,     44 $
 357598  54 ,     44 $
 357606  54 ,     44 $
 357614  70 8     40
 357622  65 5     45 %
 357630  54 ,     44 $
 357702  54 ,     44 $
 357710  54 ,     44 $
 357718  63 3     41 !
 357726  70 8     40
 357734  54 ,     44 $
 357742  62 2     40
 357750  54 ,     44 $
 357758  60 0     40
 357830  67 7     45 %
 357838  62 2     40
 357846  54 ,     44 $
 357854  71 9     41 !
 357862  65 5     45 %
 357870  61 1     41 !
 357878  63 3     41 !
 357886  54 ,     44 $
 357958  54 ,     44 $
 357966 135 ]      5 ^E
 357974  62 2     40
 357982  62 2     40
 357990  63 3     41 !
 357998  67 7     45 %
 358006  54 ,     44 $
 358014  54 ,     44 $
 358086  70 8     40
 358094  62 2     40
 358102  61 1     41 !
 358110  63 3     41 !
 358118  54 ,     44 $
 358126  54 ,     44 $
 358134  54 ,     44 $
 358142 133 [      1 ^A
 358214 154 l     44 $
 358222 164 t     44 $
 358230  54 ,     44 $
 358238  54 ,     44 $
 358246  54 ,     44 $
 358254 133 [      1 ^A
 358262  54 ,     44 $
 358270  54 ,     44 $
 358342  54 ,     44 $
 358350  54 ,     44 $
 358358  54 ,     44 $
 358366 133 [      1 ^A
 358374  54 ,     44 $
 358382  54 ,     44 $
 358390  54 ,     44 $
 358398  54 ,     44 $
 358470  54 ,     44 $
 358478 133 [      1 ^A
 358486  54 ,     44 $
 358494  54 ,     44 $
 358502  54 ,     44 $
 358510  54 ,     44 $
 358518  67 7     45 %
 358526  71 9     41 !
 358598 163 s     41 !
 358606 141 a     41 !
 358614  61 1     41 !
 358622  54 ,     44 $
 358630  61 1     41 !
 358638  54 ,     44 $
 358646 133 [      1 ^A
 358654  62 2     40
 358726  54 ,     44 $
 358734  54 ,     44 $
 358742 133 [      1 ^A
 358750  62 2     40
 358758  61 1     41 !
 358766  54 ,     44 $
 358774 133 [      1 ^A
 358782  63 3     41 !
 358854  60 0     40
 358862  64 4     44 $
 358870  54 ,     44 $
 358878  65 5     45 %
 358886  54 ,     44 $
 358894  54 ,     44 $
 358902  64 4     44 $
 358910  62 2     40
 358982  54 ,     44 $
 358990  64 4     44 $
 358998 125 U      5 ^E
 359006  56 .     44 $
 359014  54 ,     44 $
 359030 164 t     44 $
 359038 156 n     44 $
 359110  54 ,     44 $
 359118  54 ,     44 $
 359126  54 ,     44 $
 359134  54 ,     44 $
 359142 135 ]      5 ^E
 359150  60 0     40
 359158  54 ,     44 $
 359166  54 ,     44 $
 359238  54 ,     44 $
 359246  54 ,     44 $
 359254  60 0     40
 359262  60 0     40
 359270  54 ,     44 $
 359278  54 ,     44 $
 359286  54 ,     44 $
 359294  54 ,     44 $
 359366  63 3     41 !
 359374  60 0     40
 359382  54 ,     44 $
 359390  54 ,     44 $
 359398  54 ,     44 $
 359406  54 ,     44 $
 359414  60 0     40
 359422  61 1     41 !
 359494  54 ,     44 $
 359502  54 ,     44 $
 359510  67 7     45 %
 359518  62 2     40
 359526  54 ,     44 $
 359534  63 3     41 !
 359542  66 6     44 $
 359550  54 ,     44 $
 359622  54 ,     44 $
 359630  62 2     40
 359638  64 4     44 $
 359646  54 ,     44 $
 359654  60 0     40
 359662  64 4     44 $
 359670  54 ,     44 $
 359678  54 ,     44 $
 359750  54 ,     44 $
 359758  67 7     45 %
 359766  54 ,     44 $
 359774  54 ,     44 $
 359782  54 ,     44 $
 359790  61 1     41 !
 359798  54 ,     44 $
 359806  54 ,     44 $
 359878 164 t     44 $
 359886 101 A      1 ^A
 359894 106 F      4 ^D
 359902 162 r     60 0
 359910 133 [      1 ^A
 359918  54 ,     44 $
 359926  54 ,     44 $
 359934  63 3     41 !
 360006  66 6     44 $
 360014  62 2     40
 360022  66 6     44 $
 360030  66 6     44 $
 360038  67 7     45 %
 360046  65 5     45 %
 360054  54 ,     64 4
 360062 135 ]      5 ^E
 360142  62 2     40
 360158 133 [      1 ^A
 360166  54 ,     44 $
 360174  66 6     44 $
 360182  62 2     40
 360190  66 6     44 $
 360262  54 ,     44 $
 360270  71 9     41 !
 360278  65 5     45 %
 360286 113 K      1 ^A
 360294  67 7     45 %
 360302  56 .     44 $
 360310 165 u     47 '
 360318 122 R      0 ^@
 360390  54 ,     44 $
 360398  63 3     41 !
 360406  54 ,     44 $
 360414  54 ,     44 $
 360422  54 ,     44 $
 360430  54 ,     44 $
 360438  60 0     40
 360446  54 ,     44 $
 548934  61 1    163 s
 548942  61 1    163 s
 548950 133 [      1 ^A
 548958  65 5     45 %
 548966  54 ,    166 v
 548974  54 ,    166 v
 548982  54 ,     64 4
 548990  54 ,    144 d
 549062  60 0    172 z
 549070 135 ]    127 W
 549078  54 ,     44 $
 549086  61 1     41 !
 549094  65 5    167 w
 549102 135 ]    127 W
 549110 124 T      4 ^D
 549118  61 1     41 !
 549190  61 1    163 s
 549198  54 ,    166 v
 549206  60 0     40
 549214  63 3     41 !
 549222  54 ,    166 v
 549230  54 ,    166 v
 549238  60 0    140 `
 549246  54 ,     44 $
 549318  54 ,    166 v
 549326  66 6    166 v
 549334  54 ,     44 $
 549342  54 ,     44 $
 549350  54 ,    166 v
 549358  61 1    163 s
 549366  54 ,     46 &
 549374  54 ,     44 $
 549446  54 ,    166 v
 549454  61 1    163 s
 549462  71 9     43 #
 549470  63 3     43 #
 549478  62 2    172 z
 549486  61 1    163 s
 549494  61 1     41 !
 549502  64 4     44 $
 549574  62 2    162 r
 549582  40      162 r
 549590  54 ,     44 $
 549598 124 T      4 ^D
 549606  70 8    162 r
 549614  62 2    162 r
 549622  63 3     43 #
 549630  63 3     43 #
 549702  67 7    167 w
 549710  54 ,    166 v
 549718  54 ,     44 $
 549726 135 ]      7 ^G
 549734  61 1    163 s
 549742  54 ,    166 v
 549750  54 ,     66 6
 549758  63 3     41 !
 549830  54 ,    166 v
 549838  54 ,    166 v
 549846  54 ,     44 $
 549854 135 ]      5 ^E
 549862  54 ,    166 v
 549870  54 ,    166 v
 549878  62 2     40
 549886  54 ,     44 $
 549958  65 5    167 w
 549966  62 2    162 r
 549974  63 3     41 !
 549982  54 ,     44 $
 549990 133 [    123 S
 549998  67 7    167 w
 550006  67 7     45 %
 550014 156 n     44 $
 550086  54 ,    166 v
 550094  54 ,    166 v
 550102  54 ,     44 $
 550110  54 ,     44 $
 550118  62 2    162 r
 550126 133 [    123 S
 550134  66 6     44 $
 550142  65 5     45 %
 550214  54 ,    166 v
 550222  61 1    163 s
 550230  62 2     40
 550238  71 9     41 !
 550246 135 ]    127 W
 550254  62 2    162 r
 550262  61 1     43 #
 550270  61 1     41 !
 550342  61 1    163 s
 550350  61 1    163 s
 550358  61 1     41 !
 550366  61 1     41 !
 550374  54 ,    166 v
 550382  54 ,    166 v
 550390  65 5     45 %
 550398 135 ]      5 ^E
 550470  54 ,    166 v
 550478  61 1    163 s
 550486  54 ,     44 $
 550494 135 ]      5 ^E
 550502  65 5    167 w
 550510  54 ,    176 ~
 550518  62 2     40
 550526  60 0     40
 550598  54 ,     66 6
 550606  61 1    163 s
 550614  54 ,     44 $
 550622 133 [      1 ^A
 550630  64 4    166 v
 550638  71 9    163 s
 550646  61 1     41 !
 550654  63 3     41 !
 550726  65 5    167 w
 550734  54 ,    166 v
 550742  54 ,     44 $
 550750  64 4     44 $
 550758  54 ,    166 v
 550766  64 4    166 v
 550774  65 5     45 %
 550782  54 ,     44 $
 550854  67 7    167 w
 550862  64 4    166 v
 550870 162 r     40
 550886 133 [    123 S
 550894  62 2    162 r
 550902  62 2     42 "
 550910  61 1     41 !
 550982  62 2    162 r
 550990  61 1    163 s
 550998  54 ,     44 $
 551006  54 ,     44 $
 551014  54 ,    166 v
 551022  54 ,    166 v
 551030  60 0     40
 551038  61 1     63 3
 551110  54 ,    166 v
 551118  70 8    162 r
 551126  54 ,     64 4
 551134  54 ,     44 $
 551142  54 ,    176 ~
 551150  60 0    162 r
 551158  61 1     41 !
 551166  64 4     44 $
 551238  54 ,    166 v
 551246  61 1    163 s
 551254  70 8     42 "
 551262  66 6     44 $
 551270 103 C    123 S
 551278  62 2    162 r
 551286  67 7     45 %
 551294 102 B      0 ^@
 551366  54 ,    166 v
 551374  61 1    163 s
 551382 133 [      1 ^A
 551390  61 1     41 !
 551398  54 ,    166 v
 551406  54 ,    166 v
 551414  54 ,     44 $
 551422  54 ,     46 &
 551494  66 6    166 v
 551502  63 3    163 s
 551510 135 ]      5 ^E
 551518  63 3     41 !
 551526  63 3    163 s
 551534  65 5    167 w
 551542 135 ]      5 ^E
 551622  65 5    167 w
 551630  54 ,    166 v
 551638  54 ,     44 $
 551646  54 ,     44 $
 551654  63 3    163 s
 551662  71 9    163 s
 551670 135 ]      5 ^E
 551678  61 1     41 !
 551750 151 i    163 s
 551758 141 a    163 s
 551774  63 3     41 !
 551782  66 6    166 v
 551790  54 ,    166 v
 551798  62 2     40
 551806  64 4     44 $
 551878  54 ,    166 v
 551886  63 3    173 {
 551894  65 5     45 %
 551902  67 7     45 %
 551910  63 3    173 {
 551918  54 ,    166 v
 551926  54 ,    144 d
 551934  54 ,     44 $
 552006  61 1    173 {
 552014  65 5    167 w
 552022  63 3     43 #
 552030  54 ,     46 &
 552038  61 1    163 s
 552046  63 3    163 s
 552054  70 8     40
 552062  64 4     44 $
 552134  65 5    167 w
 552142  71 9    173 {
 552150 154 l     66 6
 552158  40       60 0
 552166 154 l    166 v
 552174 151 i    163 s
 552182 160 p     40
 552190  61 1     43 #
 552262  54 ,    166 v
 552270  54 ,    166 v
 552278  54 ,     44 $
 552286  63 3     61 1
 552294  54 ,    166 v
 552302  54 ,    166 v
 552310  64 4     44 $
 552318  54 ,     44 $
 552390  61 1    163 s
 552398  54 ,    166 v
 552406 133 [      1 ^A
 552414  65 5     45 %
 552422  70 8    162 r
 552430  66 6    166 v
 552438  54 ,     44 $
 552446  54 ,     44 $
 552518  61 1    163 s
 552526 135 ]    137 _
 552534  62 2     42 "
 552542  61 1     41 !
 552550  61 1    163 s
 552558 135 ]    127 W
 552566 117 O      5 ^E
 552574  65 5     45 %
 552646  61 1    163 s
 552654  67 7    167 w
 552662  54 ,     44 $
 552670 135 ]      5 ^E
 552678  62 2    162 r
 552686  61 1    163 s
 552694  54 ,     46 &
 552702 135 ]      5 ^E
 552774  61 1    163 s
 552782  67 7    167 w
 552790  54 ,     46 &
 552798  63 3     61 1
 552806  54 ,    166 v
 552814  70 8    172 z
 552822  54 ,     44 $
 552830  60 0     40
 552902  71 9    163 s
 552910  61 1    163 s
 552918 133 [      1 ^A
 552926  61 1     41 !
 552934  62 2    162 r
 552942  54 ,    166 v
 552950 133 [      3 ^C
 552958  62 2     40

One thought on “File corruption on my web server

Comments are closed.