DNG workflow ideas

When I come back from a wedding, I've got hundreds and hundreds of RAW files that have to be dealt with. It's not uncommon to have 20-30 gigabytes of data on several CompactFlash (CF) cards. Here's what I do to manage all those photos.

When I put a CF card in to my reader I like to copy the files directly to an external hard drive rather than having some program (Lightroom or Photo Mechanic) do the transferring for me. Call me paranoid, 'cuz that's what I am. I like to have my original image source (the CF card) plugged in to my cranky computer for as little time as possible.

Here's my secret recipe (how corny is that?)..

First I create a folder with a unique filename for the event (like "2008/02/29_Karen & Paul") so that I can keep everything organized. As I work with the files in the days ahead, I always keep that folder structure. For the past couple years I’ve been starting the name with the year, because after several years of doing weddings I found that I actually had some redundant names and dates: in fact I’ve been called “The Official Photographer of Jennifers and Mikes.” This folder sits on an external firewire hard drive so that if I have any sort of system problem I'll have a set of files that are, again, independent of the computer.



Then I use the Adobe DNG Converter to process all those proprietary RAW files on the first external hard drive to another external drive.

The first option you see when you open the Adobe Digital Negative Converter is the select the images you want to convert. Here's where I pick that first folder where I brought in the files from the CF cards.



Just down from there you have the area where you can choose where you're going to put the images once you've converted them. In my case they are going in to a second folder (but with that same name as the first folder, “2008/02/29_Karen & Paul) on a different drive where I'm going to end up importing them later in to Lightroom. I have numerous external drives, and this second one is an eSATA drive that I have set up as a RAID array.)

The renaming option is next, and many photographers may want at this point to name their files with something that is unique to the event or assignment. I hold off on renaming until the last step of my workflow, because I may end up editing out lots of photos from the final set that goes to a client and I like all the filenames to be continuous. (They may never notice, but I don't like to have a gap in my numbering in files that go to clients.. "I've got number 16, and number 19, but where are 17 & 18?" is a question I want to avoid.)

Lastly you find a place where you can set some preferences for how you want the files handled during the process. What I've chosen to do is to leave the files uncompressed (more on this in the next post) and to preserve the original raw file rather than convert it to linear data. This allows me to refer to the original raw data later if I need to. And I choose not to embed the original raw file in with the DNG, because I already have a backup of those files anyway and for now I get as much out of the DNGs as I need. Also, if you choose to embed the original raw files you end up with a MUCH larger set of files, because you've got a "box" with the original file PLUS the converted file. If you do choose to embed the file, you can also later extract that with the "extract" button options at the bottom of the converter.



Now the converter purrs away, transmogrifying all those files at breakneck speed.

So, now I have my dngs all sitting on the second drive, and it's at this point that I launch Lightroom and import the images in place. This happens very quickly. In fact, bringing in 1100 images to Lightroom from last week's wedding took less than 8 minutes, including building the standard previews and writing the keywords and iptc information to each file. That first drive, where I brought everything in from the CF cards, really just serves as a backup at this point, and I leave the files there until the job is delivered. Once that’s done, weeks from now, I’ll clear them off to make room for new images.

Once all the images are safely off the CF cards, I'll do a quick pass through the images in Lightroom and delete any obvious screw-ups like super-underexposed photos, out-of-focus shots and photos I accidentally shot of my feet. This eliminates, oh, half of the pictures right off the bat. I select all the images and choose SAVE, which writes any Develop or Metadata changes in to the files in case I’ve played with tweaking some of them (and I always have). Then I burn DVD's of the remaining images and put the discs in my safe in the basement, which is supposed to be fireproof. I'm not testing that anytime soon.

Now I'm free to do my major editing in Lightroom or, if I need to do some detailed work I can take the images in to Photoshop. I've got the original RAW files if I need them, and a set of DNG's from those RAW files on another drive and a set of those files backed up to DVD. At this stage, I can erase and format those CF cards and stick them back in my camera for the next shoot, knowing that short of a major lightning strike that sets fire to my house I should be able to work with last week’s pictures until well in to the 21st Century.

Why convert RAW files to DNG?

A 2007 survey of photographers using Photohop and Lightroom found that something like 40% of those photographers are now converting their RAW images to Adobe's DNG format. I do this too, and I thought it might be helpful to tell some about what I know about the format and also describe my workflow.

When I began shooting digitally I didn't work with RAW files at all. Mainly that choice was driven by the equipment I was using. Even though my Fuji S3 could capture images in their RAW format (.RAF), the camera took almost 12 seconds for the data to clear from the buffer as it was being written to my CF card. During that time, you could not make another image. Perhaps if I was shooting a landscape or was in a situation where I didn't need to worry about shooting more rapidly it would have been okay to shoot in RAW. But shooting news assignments and events and weddings that just wasn't an option. So for a long time I set the camera to make JPEGs of all my images, and I was fortunate that the camera produced some beautiful images in that format. I still use the S3, even though I have added several newer bodies to my arsenal.

After I got my Nikon D200 I started shooting RAW files (Nikon calls them .NEF files) and I quickly found that it opened up some interesting possibilities in my workflow, and that it also created some new headaches for me. Programs like Adobe Lightroom seemed to work a little more smoothly and quickly with RAW files, in part because it was designed in large part to do just that. And of course it integrates with the Camera Raw processor for Bridge and Photoshop, letting me apply or undo settings across the various programs. But my JPEG files had only been something like 4 or 5 megabytes in size (when unopened), so I could shoot a whole wedding with just a couple of 1-gig flash cards. The .RAF files from my Fuji S5 clock in at a whopping 25 megabytes per image, and I quickly found myself having to buy 4- and 8-gig CF cards as well as budgeting for a number additional hard drives for my computer each year. Suddenly, having a few terabytes didn't seem like such an extravagance anymore.

My collection of CompactFlash cards grew once I started shooting RAW instead of JPEG.

Fortunately, you can go a long way toward solving this storage problem by converting your files to DNGs, and you can pick up some other benefits along the way.

If I could cite no other benefit to using the conversion, I would still be pleased with this part of it. You can see this illustrated in the screenshots below, showing files that I shot at a wedding late last year. I've renamed the files, but I am being honest about what the information reveals.. the DNG files created from the Fuji S5 are nearly 40% larger than the original .RAF files. DNG's made from the Nikon D200 show a similar space-saving effect.

The .raf files from my Fuji S5 are 24.5 MB each,
but converting them to .dng files means they take up much less space.
Even though I renamed these files, they are each from that camera.

What accounts for the difference in size? Well, that's not entirely easy to explain. NEF and RAF files, as well as Canon's CR2 and many other RAW formats, are called Proprietary Files, meaning they belong to those manufacturers. They create these formats to work with their specific camera models, and they want you to use their specific software to work with those files once you get them in to your computer. That way, your Canon images that you made on your Canon camera all stay in the Canon family, and you end up buying more stuff from Canon. (Substitute your own brand anywhere in that last sentence.) In order to keep them "proprietary," they don't tell other companies (like, say, Adobe) all the stuff that's in them.

There is some good speculation for some elements to account for the size change. For one, it seems common that there are several levels of preview files (generally JPEGs) of the main image that are included along with the data. That's how you can see the photo right away on the back of your camera and once you start loading them in to various other programs on your computer. Because software like Lightroom builds new previews from the image data and stores it in a separate library file, you can get away without having that stuff embedded in the original.

Another possibility (and again, all we can do is guess, 'cuz the companies ain't sayin' nothin') is that there is extra data in the file that allows specific functionality with the maker's own software. For example, Nikon Capture lets you work with specific areas of an image to lighten a subjects' face without having to adjust the entire image. At present that functionality doesn't exist in Adobe Camera Raw or Lightroom. It's a cool feature, frankly, and I wish that Adobe would figure out a way to add this to the already impressive lineup of features that they afford.

So that could be an argument for not converting your RAW files to DNG and just using, say, Nikon Capture or Fuji's HyperUtility for their files. But to my mind there is a slight flaw in that logic, and it's based on my experience with software. Even if they say they will support software and cameras down the road, it's not always the case that companies actually do that. To cite extreme cases, I sometimes like to say that no one is listening to their eight-track tapes anymore, except for the creepy guy you stand next to in line at the Taco Bell late at night.

A few years down the road, that proprietary file you have sitting on your hard drive could quite possibly be just a useless mass of 1's and 0's that you can't access. This is where conversion to DNG once again saves the day.

Adobe developed the DNG format to convert RAW files and has made the coding openly available to anyone that would like to take advantage of it. Even though they thought it up, they aren't trying to "own" it. And the recent history of open source software has shown that this is a pretty good model for sustainability, because as new situations arise there is a large field of determined geeks willing to write new code to deal with them. It will not surprise me even if some of the secret proprietary information is eventually decoded by some of these same geeks (don't get me wrong; I love geeks, and depend on them, and secretly long to be accepted by them) so that you might someday be able to take advantage of those specific features. The chance of that (decoding the secret stuff) happening is probably small, whereas the chance of the first part (geeks constantly enhancing and supporting open-source software) is pretty good. So down the road I'm confident that I will be able to open, view and work with my DNG files, and enjoy making adjustments to them in future programs while I listen to my 8-Tracks and eat Gorditas.

Next up: some workflow thoughts on handling and storing DNG's and RAW files.

Organizing your photos.. it's about time

My wife and mother-in-law recently went on a three-week trip to Morocco. They visited a lot of cool places, and they each took a lot of photos. When they came back we went through each of their pictures as they told me all about the trip. But we ran in to an interesting problem trying to match up their stories because her mom's photos were all taken five years, three months and 11 days before Edna's.

"Did we visit the market in Fes before we saw the mosque in Rabat, or was it after we drove to Essaouira?" It may seem like you'll remember every detail of that dream vacation for ever and ever, but in my experience that's not how it works. Sometimes I can't remember when we did what even as I'm sitting on the plane coming home.

This isn't just a problem for hobbyists. Real pros in the field very often have to figure out just when they shot something: one of the problems I regularly face is having to mix together photos taken with several different cameras to put them in order.

When I would come back from a wedding with 35 rolls of film I sometimes spent hours trying to sort the prints in chronological order,because I thought it would be a bad thing (by which I mean, "I would look stupid") for my clients to be going through the pictures from the ceremony and find a few prints of the cake-cutting interspersed with them. This was made more difficult by the fact that I shoot with multiple cameras at an event: I may make a few pictures with one body and then use another camera for 15 minutes before using the first one again. What a pain it was untying that knot later on.

Fortunately, with digital cameras, there's a very simple solution for this.

SET THE DATE AND TIME ON YOUR CAMERA!

Duh.

All digital cameras embed the date and time that you clicked the shutter in to the image data that acompanies the pixels themselves, whether you're aware of it or not. In the "old" days of film cameras some models had the HORRIBLE option of actually burning the date and time in to the image itself. The result? You ruined every shot you took, much in the same way I have here by inserting the copyright information on each picture you see. (Fortunately, the originals are pristine.) The timestamp information is tucked away in each digital picture, and it can be very useful because many photo management programs let you view and sort your photos by the date they were shot. For me I sometimes need to sort images down to the second they were made, and this option lets me do that.

Adobe Bridge and other programs let you sort by date created.

So even if you don't do all that fancy keywording and captioning of every images like the big movie stars do, you could still locate groups of images based on when you pressed the shutter button. This would have helped my friend Gordon find the photos of the superhot babe from two years ago at the racetrack that he was trying to show me last weekend, which we finally got to after first wading through hundreds of birthday and holiday photos in Picasa. At least it was worth the wait!

If you've got a giant pile of digital pictures in your computer and you didn't bother to set the date on your camera before you shot them, it's not too late to fix that.

In Adobe Lightroom, where I do most of my work with images, there's a fairly elegant way to edit the capture time of your photos by pulling down the "Metadata" menu to "Edit Capture Time." First, select all the photos that were taken within the time range you want to adjust (say, everything shot on the 4th of July.)

Lightroom's pull-down menu

In the menu that pops up you can set everything to the 4th of July or whatever day you want. Or if you only need to adjust things by a few hours (Spring ahead, Fall back, or maybe some of your photos were shot in Fresno and some in Bangor) you can do that as well.

Adjust the date & time on many images at once by selecting them first.

Photo Mechanic, a very nice browsing software from CameraBits gives you even a bit more control in this process, letting you adjust by the minute or second if you choose.

The adjustment options in Photo Mechanic

If you use Adobe's Bridge, you can still make the change but it's a bit more cumbersome. If you select a photo or a series of photos, you can manually type in a new date on the "Date Created" line in the IPTC fields—look for the little pencil that indicates you can edit this field—but you can't change that same information in the "File Properties" or "Camera Data (EXIF)" fields. There's just no little pencil. In fact this function in Bridge is so inelegant that I haven't even included a photo of it here, because I'd just be making things worse. I wish Adobe would make this easier in Bridge, and given that they've gone a fair way in Lightroom it seems strange that they don't in Bridge.

Some cameras will maintain the correct date and time information if you have to change batteries, but it's not a bad idea to check this anytime your batteries die or you have to swap them out. And before you wave bye-bye to go on your big trip, taking this little step can save you some headaches later on.

Wave bye-bye!