Nikon D800 file recovery strips thumbnails from main jpg

Using PhotoRec to recover lost data
Forum rules
When asking for technical support:
- Search for posts on the same topic before posting a new question.
- Give clear, specific information in the title of your post.
- Include as many details as you can, MOST POSTS WILL GET ONLY ONE OR TWO ANSWERS.
- Post a follow up with a "Thank you" or "This worked!"
- When you learn something, use that knowledge to HELP ANOTHER USER LATER.
Before posting, please read https://www.cgsecurity.org/testdisk.pdf
Message
Author
thefool
Posts: 10
Joined: 07 Mar 2014, 18:18

Nikon D800 file recovery strips thumbnails from main jpg

#1 Post by thefool »

Hi,

I recently had an SD card die on me while I was copying the files from it.
On it were about 6-8GB of photos and videos taken with a Nikon D800.

I was able to recover all the files, I think, but noticed something strange:
The thumbnail is saved as a separate file (yes, I know that's normal)
But the main jpg is stripped of this thumbnail jpg.

Since the problem occurred during copying, I have the first few images as they were supposed to be, and can compare some original files with the recovered ones directly. The difference in file size is almost exactly the size of the thumbnail that's stored separately.

Any idea why this happens? Of course I'd like to restore the files as originally as possible, which means including the thumbnail. Who knows what else is missing... maybe the jpg gets truncated right where the header for the thumbnail jpg is found or something like that...

I could send you both an original and a recovered file (jpg+thumbnail) if anyone wants to have a closer look...

Thanks already,

Oliver

thefool
Posts: 10
Joined: 07 Mar 2014, 18:18

Re: Nikon D800 file recovery strips thumbnails from main jpg

#2 Post by thefool »

OK, I just found some more info on this:

the original files are recovered in 3 parts:
txxxxx.jpg - about 10KB
fxxxxx.jpg - about 800KB
fxxxxx.jpg - about 20MB

(original - about 21MB)

I noticed that the size of the second preview image added to the size of the main recovered image is pretty much exactly the size of the original jpg. I therefore did some searching, and found that some JPGs come with JPG trailers, which in some cases contain Preview images in an "MPF" Segment. It looks like the Nikon D800 does exactly this, but photorec does not recover these JPG trailers.

More info here:
http://search.cpan.org/dist/Image-ExifT ... d#MPF_Tags
http://www.cipa.jp/std/documents/e/DC-007_E.pdf
http://cpansearch.perl.org/src/EXIFTOOL ... ool/MPF.pm

I read that many applications strip this trailer too, and the JPG is perfectly usable as it is anyway, so I assume the files that photorec recovers are useable and not corrupted. However, for the sake of completeness and originality, I'd love to see photorec be able to recover the file as it was originally, including this (and possibly other) "trailer" data. It seems a lot more than just this single preview image can be stored in the JPG trailer, so in other cases it might be much more valuable data that is lost when the trailer is not recovered along with the jpg.

I'd be happy to provide an original + the 3 recovered parts to anyone that wants to have a closer look. I have about 70 such sets.
I also have the SD card and an image.dd of it that I made with testdisk. but it's 32GB...

I would also be happy to recreate the scenario on a smaller card - for example a 512MB card, that I could completely zero out, then take a couple of pictures and a video with the D800. I could then make an image of the card, and make it available for testing with photorec. The images could then be copied while the filesystem is intact, and then recovered using photorec (ignoring the filesystem).

User avatar
cgrenier
Site Admin
Posts: 5432
Joined: 18 Feb 2012, 15:08
Location: Le Perreux Sur Marne, France
Contact:

Re: Nikon D800 file recovery strips thumbnails from main jpg

#3 Post by cgrenier »

Can you reproduce this problem when using PhotoRec 7.0-WIP ? If it's the case, please send me by email an unmodified jpg taken with your camera.

thefool
Posts: 10
Joined: 07 Mar 2014, 18:18

Re: Nikon D800 file recovery strips thumbnails from main jpg

#4 Post by thefool »

I'll try right now. But it will probably take a while... last time, photorec recovered about 11GB.
Thanks already.

thefool
Posts: 10
Joined: 07 Mar 2014, 18:18

Re: Nikon D800 file recovery strips thumbnails from main jpg

#5 Post by thefool »

OK, yes, I can confirm that the 7.0-WIP version does the same problem.

I've made a zip file of one original, and the 3 corresponding recovered parts (recovered by the 7.0-WIP version) and uploaded it here:
http://www.foolography.com/d800photorec.zip

If you want, I can take some more sample photos for you - maybe some smaller ones would be easier to work with - I'd either set the camera to the lowest resolution, or take a completely black photo with the cap on or something like that (or both, for smallest possible file size...)

I can also take a raw photo if you want to use that for comparison too...

Thank you!

thefool
Posts: 10
Joined: 07 Mar 2014, 18:18

Re: Nikon D800 file recovery strips thumbnails from main jpg

#6 Post by thefool »

One more thing:
while renaming the files using exiftool, I get this warning:
Warning: [minor] Error reading PreviewImage from file
When I run the exact same command on the original file, I don't get that warning. It seems exiftool knows that there should be a PreviewImage in the JPG trailer, but can't find it - I guess you could use that same information to look for the trailer, and recover it along with the main JPG...

User avatar
cgrenier
Site Admin
Posts: 5432
Joined: 18 Feb 2012, 15:08
Location: Le Perreux Sur Marne, France
Contact:

Re: Nikon D800 file recovery strips thumbnails from main jpg

#7 Post by cgrenier »

I have uploaded a new 7.0-WIP version. Can you try it ?

thefool
Posts: 10
Joined: 07 Mar 2014, 18:18

Re: Nikon D800 file recovery strips thumbnails from main jpg

#8 Post by thefool »

Wow! It seems to be working!
I'm just running a test, and the first files (also the one that I sent you) are being recovered fully.
exiftool is not giving a warning anymore either, when I try to rename the files with

Code: Select all

exiftool '-filename<DSC_${FileNumber;}.%e' -r -ext JPG .
very cool. Thank you very much!
Looks like you found the bug, or managed to add this new feature, if it wasn't a bug...
What was the exact problem in the end?

I'm glad I could help with my problem :-)
And of course I'm even more glad you were able to help me (and so many others) with photorec in General, but also with this specific fix!

By the way - I found out that all newer Nikon Cameras handle their files similarly - e.g. the D600, D610, D4, etc... So this is not just D800 specific (if you want to add that to your change log or so).

Thanks again so much!

Oliver

thefool
Posts: 10
Joined: 07 Mar 2014, 18:18

Re: Nikon D800 file recovery strips thumbnails from main jpg

#9 Post by thefool »

I just got back to my laptop, photorec has been recovering since 2:17 hours... now it's filling up my hard drive with thousands of .gz files, which it didn't do before your last update!

Code: Select all

Disk /dev/disk3 - 31 GB / 29 GiB (RO)
     Partition                  Start        End    Size in sectors
     No partition             0   0  1 62333950   0  1   62333951 [Whole disk]


Pass 2 - Reading sector   25931008/62333951, 581466 files found
Elapsed time 2h21m44s - Estimated time to completion 3h18m58
gz: 580851 recovered
jpg: 601 recovered
mov: 8 recovered
tx?: 3 recovered
apple: 3 recovered
It's currently in the empty parts of the SD card. (It's a pretty new SD card, and I've only used a max of 11GB of the 32 available. it's already at recup_dir.1000, creating more and more. (the last hundred in less than a minute)

each folder contains one .gz file that's 512 bytes in size

thefool
Posts: 10
Joined: 07 Mar 2014, 18:18

Re: Nikon D800 file recovery strips thumbnails from main jpg

#10 Post by thefool »

I went through the recovered files today, renamed them all, and just to be sure, used "diff" to compare the recovered files with the originals I still had - all the files I tested were identical! Thank you again!

Locked