Classic Cameras Reviewed Up Close

  Photography Help Pages
Home  -  Email
 About me  CLASSIC CAMERAS REVIEWED UP CLOSE  Camera shopping in Hong Kong  Camera choice  Camera repairs in HK
 Photography essays  Photo printing  Photo preservation  Corrupted JPEGs  Photo framing shops
 External links  Other camera review web sites  Lens reviews  Helpful software

Many years ago I copied a folder of jpeg files from my PC's desktop to an internal back-up hard disk in my computer.  After the copy was complete, I opened the newly made folder on the back-up disk, and browsed the jpeg file copies in the Windows Picture and Fax Viewer utitlity.  I was surprised to find that about 5 to 10 per cent of the copied files had a strange corruption.  A thin light grey horizontal line was visible at random heights and starting points in each of the corrupted images.  A sample is shown below:
 
 
 
Photoshop complains that the file is damaged when I try to open it:
 
 
 
And here's what you get if you proceed with opening the file in Photoshop:
 
 
 
I checked the original images in the source folder on the desktop computer and found that the files were all okay - no corruptions.  So I did a test.  I copied the folder again to a different partition on the back-up hard disk.  The same problem occurred - some of the images were corrupted with the thin grey horizontal line.  But they were not the same images that became corrupted during the first copy I made of the folder.
 
I was becoming worried.  It seemed that I couldn't trust my computer to reliably make copies of my JPEG files.  After a lot of trials and investigations, I concluded that there was something wrong with my PC's motherboard.  So I built a new PC based on a motherboard that used a different chipset, and then the problem seemed to go away.  I was relieved to think that I had solved the problem.
 
A few months later, I browsed some of the old folders from the old back up hard disk that was now living in my new PC.  I was shocked and disappointed to find that not only were there corrupted files, but the degree of corruption was now far worse than what I had seen a few years earlier.  It seemed that the JPEG files on my hard disk were deteriorating.  There are multiple lines of random colour now, not just simple grey lines.  And some files are so badly corrupted that they are now a mish-mash of horizontal slices, or they can't be opened at all.
 
Here are some examples:
 
 
 
 
 
 
 
 
 
The corrupted JPEG files are now not only more numerous than they were two years earlier, but the degree of corruption had become much worse.  Many of the corrupted JPEG files cannot be opened at all by Windows Picture & Fax Viewer, nor can that program make thumbnails of the very badly corrupted images.
 
Where Windows Picture & Fax Viewer can open the file at all, I'm now seeing some very serious damage to the images - far beyond simple grey lines.  Here are some samples:
 
 
 
 
 This is a very serious problem, and it will just get worse with time, because two things are inevitable:
 
Firstly, my hard disks are ageing, and with that age comes an increased "bit error rate"  (BER).  The JPEGS are corrupted because the BER for the hard disk is too high for the usual error correction mechanism built into the disk drive.  When one bit becomes corrupted in a JPEG file, it can cause the loss of a whole section of the image (the horizontal grey stripe, as a minimum).
 
Secondly, I am accumulating more and more JPEG files every month, as I pursue my photography business and hobby.  There are now over one million jpeg files on my workstation, and the really scary thing is that I don't know how many of them are corrupted.  It's impractical for me to check them all manually by trying to view all of them individually with my own eyes.
 
I wondered if the problem was unique to JPEG files, or whether any type of file is prone to corruption.  Investigation found that any file is prone to the same type of corruption.  But JPEG files show the corruption in a more dramatic way, due to the fact that small bit errors can destroy the 'key' to decoding larger sections of a photograph.
 
Here are some examples of what the bit errors look like in a TIFF file:
 
 
 
 
The bit errors in a TIFF file only damage a small number of pixels. There's no gross multiplier effect like there is with bit errors in a JPEG file.  The small corruptions in the TIFF file, shown as short, narrow "drop-outs" (a video term) could be repaired easily enough in Photoshop.
 
So it seems that TIFF files are more robust as an archival format for photographs.
 
Photoshop .PSD files are also victim to the bit error corruptions, and Photoshop can detect the fact of corrupted pixels in a .PSD file when opening a corrupted file.  The following message will appear:
 
 
 
I figure that there must be other people in the world who are having the same problem, so perhaps the solution that I come up with might be useful to others too, not just me.  For that reason, I am investing in the development of a software solution, and I hope I might be able to recover some of that investment by sharing the solution with other photographers who are suffering from the same difficulty.
 
Here's the basic overview of the technical problem, and the approach I am taking to solve it forever:
 
1. Consumer-oriented operating systems like Windows and the typical hardware that they run on do not have the "high integrity" features of the more professional operating systems like one would find on say, an IBM server running z/OS or a Sun Microsystems workstation running the Solaris operating system.  A simple example is how the Windows XP CD-burning utility offered no option to verify the data on the CD after it was burned.  ( I don't use Apple's Mac OS so I can't compare it to Windows or know if it provides better data integrity or not.)
 
2.  Due to the ubiquitous accumulation of rich-media data such as photographs, digital video, digital audio and the like, consumers are relentlessly using up hundreds of gigabytes of data storage on their home PC systems.  The standards for consumer PC data integrity have not kept up with the growth of consumer data accumulation.
 
3.  Many professionals, such as photographers and musicians, are trying to run their business on unprofessional operating systems like Windows. (I am one of them.)  The business risk associated with lost data is very high.
 
4. Data storage media like hard disk drives and recordable CD's and DVD's and even digital videotape, are not perfect.  Any storage medium has a native bit error rate (BER) and is equipped with Error Checking and Correction (ECC) to improve the native bit error rate from something like one bad bit in a million (1 x 10e-6) to a net error rate after correction of something like 1 bad bit in 100 quadrillion (1 x 10e-14).
 
5.  After just a few years, typically soon after the warranty has expired, hard disk performance and reliability will noticeably degrade.  Replacing the hard disk with a new one is recommended.
 
And here is the overview of my proposed solution:
 
Step 1:  find out if your PC has the problem or not, and to what degree it may have the problem.
 
Step 2:  Locate every corrupted JPEG file.  See if you can find a good copy of it elsewere.
 
Step 3:  Determine the 'fitness' of your hard disk drive and make a judgment as to whether you should continue using it or replace it.  If in doubt, replace it just to be safe.
 
Step 4:  To prevent the problem from occurring again, do the following:
 
4.1  employ a 'verify after copy' method to confirm that newly copied files and folders are not corrupted
 
4.2  employ an ECC wrapper around JPEG files so that errors can be detected and corrected when they are read from degrading media some time later
 
4.3  upgrade your computer hardware to "server class" equipment, which has ECC on the memory and robust ECC on the hard disc storage subsystem
 
I am currently working on all the steps above.  It's work in progress, so email me if this work is of interest to you.  I might be able to help you find a solution if you've also got this problem.
 
You can email me via craig [at] alkiracamera [dot] com.

© Copyright 2008 Craig Norris.  All rights reserved.
Powered by ABCHK.com