Lightroom as a DAM Tool and Metadata Stats
Everyone loves stats! I built a giant Lightroom catalogue of all my photos since 2004 — about 20,500 odd. Much to my surprise, it worked superbly well. This is one area in which Lightroom 2 is a vast improvement over version 1 which started slowing down noticeably when my catalogs reached a size of about 10k photos (although I have also upgraded my machine since last I tried that). Having so many photos lets you get a birdseye view of your own shooting habits and equipment. So here is a summary from the metadata view:
Top cameras:
- Pentax K100D - 11,972
- Pentax K10D - 3,636
- Pentax *ist DL - 1,938
- Canon EOS 350D - 1,471
- Fuji FinePix 4700z - 560
- various other point-and-shoots, film scans, edits
Of those, I still have the K10D and the K100D. Although I've had the K10D for about the same time that I had the K100D before, I took all my photos in Namibia and Europe last year with the K100D hence the much higher picture count from with body.
Top Pentax lenses:
- Pentax DA 50-200mm F4-5.6 ED - 6,713
- Pentax DA 16-45mm F4 ED AL - 5,069
- Pentax FA 50mm F1.4 - 2,911
- Pentax DA 18-55mm - 2,204
- various K/M/A lenses - 628
- various others
The 16-45/4 is the lens most likely to be found on the front of my K10D these days but I've shot all the fashion weeks with the 50-200 for lack of a better lens and those generate tons of shots in a short space time. This accounts for the puny 50-200 having so many pics against it! Otherwise the 16-45 is my favourite lens - great optical quality, light, and almost-but-not-quite wide enough standard zoom (equivalent to 24-70mm in 35mm terms).
*DAM = Digital Asset Management, i.e. handling very large media libraries.
AnandTech: Intel’s Larrabee Architecture Disclosure
AnandTech has a great write-up with some early information on Intel's upcoming Larrabee graphics architecture. While the details are mostly sketchy, the article does an excellent job of analysing what the future might hold and drawing parallels with existing AMD and NVIDIA products. It sounds quite promising in theory, and the HPC/GPGPU crowd in particular should be quite excited about the total programmability. The CPU has an extended x64 instruction set and 4-way SMT (aka HyperThreading), so how soon before mainstream OS's can make use of the forthcoming hybrid architectures?
I also wonder if Apple knows something that the rest of us don't, given their OpenCL proposal and claimed multi-threaded improvements on a massive scale for Snow Leopard. It's not like they don't have a history of working closely with Intel on upcoming products. Their installed base is just about manageable enough to try something really crazy and pull it off.
With 8-core Nehalem on the horizon, and massively data-parallel processors like Larrabee, we're quickly moving towards a heterogeneous NUMA architecture on your average desktop! (IBM already offers QS22 server blade based on the Cell BE.) Exciting times, no? 
Source: AnandTech: Intel's Larrabee Architecture Disclosure: A Calculated First Move
Windows Vista Chronicles
I recently upgraded my desktop PC at home to Vista and the results were rather surprising. I found XP to be getting a bit old and dabbling in some hackintosh experiments only made its age more apparent. While Apple's OS is excellent, my old graphics card died and I could never get the replacement to play nicely with Leopard. There were several other small niggles (all due to the nature of the hack) with that installation but I liked OS X well enough to consider buying a Mac.
The trouble with the current Apple Mac line-up is that I wanted a desktop machine which only leaves the Mac Pro, and that's frankly way out of my reach. The iMac is nice but using laptop internals in an all-in-one design means it is a bit underpowered for what you pay, and totally lacking in the extensibility dept. For photography in particular the 24" model is a far better bet as it uses a fancy H-IPS LCD panel. These do not suffer from contrast and colour shifts related to viewing angle changes and are better than the common 6-bit TN panels. At that price I'd rather buy a separate LCD that I can connect to other things too, thank you very much. So while waiting for the mythical midrange Mac minitower, Vista SP1 got released and my curiosity got the better of me.
Naturally, there were a few issues along the way. One of the first was a documented problem - my screen saver was not kicking in, no matter what. Turns out that this is caused by none other than a piece of Microsoft hardware! Oh the irony. Certain types of Microsoft wireless keyboard/mouse USB receivers cause this and the solution is to install the "Microsoft - Other hardware - HID Non-User Input Data Filter" update from Windows Update. Bizarrely, this is an optional update so you have to explicitly get it.
Another issue is the well publicized problem with monitor calibration settings being lost after sleep, UAC prompts, or after starting full-screen games. It seems as though the video card Look-Up Tables (LUT) get reset on any of the above mentioned events. No real solution in sight - my workaround is to place a shortcut to the spyder2express start-up item on my Start Menu and run it manually if I notice that the calibration is lost. (If you're using a ColorVision spyder product, you will find an item called ColorVisionStartup under All Programs > Startup. Just drag it somewhere convenient or assign a keyboard shortcut via its properties panel. Other calibration solutions will probably have a similar loader program.) Windows really needs better support for this - Mac OS X does it brilliantly via System Preferences, no need for proprietary utilities to load the LUT.
I do pretty much all my digital photography in Pentax’s proprietary raw format called PEF. Although my primary camera is one of the few that support Adobe’s DNG file format, the PEF format as implemented in it offers lossless compression and thus better shooting capacity. I import my files with Adobe Lightroom, and it has the option to convert them into DNG on the fly, this time compressed. I also have some Canon CR2 files from shooting with friends’ cameras which I have also converted into Adobe’s format for long term storage. So I have quite a few DNG files lying around and I was very excited to read that Adobe has released a DNG codec for Windows Vista. Such codecs allow you to see thumbnails in Windows Explorer and browse images just like any old JPEG in Windows Photo Gallery. It works well and it even reflects Lightroom's develop settings although it’s a touch slow – the rendering speed is comparable to the delay you'd get in Bridge to get a full preview.
Besides messing with your LUTs, colour management in Vista appears to work well. I tried exporting the same file from Lightroom into sRGB, Adobe RGB, and ProPhoto RGB JPEGs. All three as well as the original DNG’s colours look identical in the built-in picture viewer! On a related note - do remember to turn on colour management in Firefox 3 to get the same consistency in there as well. Of course the proof of all this will be in the printed results that I can get, but I am not there yet.
Unfortunately, I had to uninstall the DNG codec because it causes indexing to slow down to a crawl. It would appear as if the indexing service asks it to provide the metadata for each file, and the codec renders the full image in the background before returning. Took me a while to figure this one out but with a good few gigabytes of DNG files, my index was taking weeks and still wasn’t finished. I tried excluding various folders of the index until I found it was photos that were causing it. Uninstalling the DNG codec and the full index got rebuilt in a few hours. The Adobe Labs page suggested reporting any issues on their user forums which I have done – but there hasn't been any response as yet. Hopefully this will be addressed in a future release because Windows Search is proving to be quite a useful addition to the OS.
Another minor annoyance is the inability to change he menu bar colour. In previous versions if Windows, you could change the individual widgets' default colour scheme. In Vista, the menu bar is a funky shade of blue. Change the Aero colours only affects the window frames, and fiddling with the Advanced settings does not have an effect on the menu bars. (The menus themselves are gray - mmmkay?) This is especially annoying in imaging programs such as Lightroom where the UI designers specifically only chose neutral grey hues to minimize interference with your photos' colours.
I am using AVG Free 8.0 for now but the whole Link Scanner mess is very annoying. Why can't it just scan only what I actually visit? I have disabled the Firefox AVG extension which inserts yucky green ticks into your Google Search results until they back down on this silliness, or something better comes along.
All in all, I've been very impressed with Vista and the upgrade was not at all the bumpy ride that I was expecting. Its ability to auto-locate and install certified drivers off of Windows Update is just great. It saved me from installing all the crapware that comes on our HP all-in-one OfficeJet driver CD, just plugged it in and it was ready. It even found a driver for my PCI Wi-Fi card which I'd given up searching for on SMC's website. Sleep also works fine - something I could never get right with XP. It has also been rock solid which is probably the most important thing.
JetBrains’ Dmitry Jemerov on Scala
I can rave about JetBrains' IntelliJ IDEA until the cows come home - it is simply a superbly executed and very well focused product. What I found especially interesting is the following quote on Scala becoming a dominant language for the JVM:
I don't believe that, however: Scala is very complicated, it's tricky, and has a lot of surprises and edge-cases. I would say that Scala is at least as complicated as C++, and with C++ you need a hundred-page style guide before you even start writing C++ code, otherwise you'll end up writing C++ code that nobody will understand.
Scala gives an impression of great elegance and simplicity at first glance but the same goes for Perl as well. The one feature of Scala I really like is the language-level support for traits which allow reusing multiple concrete implementations similarly to multiple inheritance, but with fewer gotchas.
Via Artima: JetBrains' Dmitry Jemerov on IntelliJ 8, Flex, and Scala.
SABS Files Official Complaint About OOXML Approval Process
The South African national standards body has become the first to file a complaint with ISO about the fast-tracked OOXML voting process, despite numerous outstanding issues. This is obviously great news and hopefully more will follow.
South Africa launches formal objection at OOXML | The Register
Donald Knuth Interview
I came across this fascinating Donald Knuth interview via Artima. A couple of snippets to pique your interest, one on the subject of open source software:
[...] I think that a few programs, such as Adobe Photoshop, will always be superior to competitors like the Gimp—for some reason, I really don’t know why! I’m quite willing to pay good money for really good software [...]
And his opinon of reuse, with which I wholeheartedly agree:
I also must confess to a strong bias against the fashion for reusable code. To me, "re-editable code" is much, much better than an untouchable black box or toolkit. I could go on and on about this. If you’re totally convinced that reusable code is wonderful, I probably won’t be able to sway you anyway, but you’ll never convince me that reusable code isn’t mostly a menace.
Other interesting topics he touches on include multi-core processor architectures and XP. Happy reading!
Fresh Xbox 360 Blu-ray Rumors
Various news sites have been reporting on new information about a possible Blu-ray console from Microsoft. Quote from Tom's Hardware (my emphasis):
The PS3 had the ability to draw both gamers and audio/video enthusiast, and this was proving to be a major advantage over its XBOX rival. It is estimated that 85% of Blu-ray players in use are PS3s.
Many expected the death of HD-DVD would trigger a growth in Blu-ray hardware sales. However, to the dismay of the industry, acceptance and adoption of Blu-ray has been dismal.
In a previous post I speculated that it is unlikely for the Xbox to get a Blu-ray add-on. But the recent spike in PS3 sales might have convinced Microsoft to update their optical drives.
The trouble with Blu-ray right now is that it is still of marginal benefit to the majority of consumers out there, thus prices of both hardware and software (well, physical media really) will remain high for some time. The only serious buyers right now are the home theater enthusiasts and Microsoft will have to address quite a few issues to please this group of people.
The biggest issue with the current Xbox 360 in the context of home theater is the amount of noise generated. A transition to a newer chip fabrication process could do a lot to reduce the need for active cooling; a potential Blu-ray transport mechanism would also have to be quieter than the current DVD drives. Another issue I see with the HT crowd is the Xbox remote - it is nowhere near as slick as the Sony Blu-ray remote. And that's before we even consider the substantially bigger pull of the Sony brand when it comes to home electronics.
It remains to be seen whether the entire range will be updated or if only a high-end model will get the Blu-ray treatment. Xbox 360 games are released on regular DVD, so I suspect Microsoft will want to keep their price advantage at the low end and stick to DVD drives for the Arcade/Core consoles.
The big question, is are there enough HD afficionados out there who have been holding out for a Blu-ray Xbox that don't already have a PS3? (Or at least would be keen to trade their existing 360 for a Blu-ray model?)
Update: Taiwanese trade press reports on an updated 65nm Xbox GPU being manufactured on behalf of Microsoft. With last year's 65nm CPU revision, this should finally make the Xbox 360 a cooler (and hopefully quieter) beast.
Uncle Bob’s upcoming book: Clean Code
Awesome - finally something to bash people on the head with when they write ugly code! 
(Ok, the "code smells" from Refactoring are a good start too but this seems more pedagogic in nature.) There are a lot of technical books out there, but few focus on good general practices. One of my favourites is Effective Java by Joshua Bloch, but it's focus is slightly more architectural and less on day-to-day coding. I love Bob Martin's writing so I didn't have to look long at the table of contents to order my copy as soon as it's available!
Office Open XML Standardised; Does It Matter Anymore?
Many might wish that this was a bad April Fools joke, but Microsoft has at last managed to push OOXML through as an ISO standard. The New York Times reports that an overwhelming majority of voters supported the notion with only 10 countries' standards bodies voting against (Brazil, Canada, China, Cuba, Ecuador, India, Iran, New Zealand, South Africa, and Venezuela).
The main problem with OOXML is that it is a hugely complicated specification (the document is nearly 10 times bigger than ODF and there are doubts as to whether it is complete); aside from the fact that it is riddled with bugs, legacy workarounds and potential patent traps for competitors, not even Microsoft currently ships a working implementation of the standard. So it is hard to imagine anyone writing new OOXML-compatible software just for the fun of it.
As Joel Spolsky explains, the reason why Microsoft's binary Office formats are so complicated to start with is that they are just binary serialized forms of Microsoft's proprietary OLE object model. The OOXML standard describes how to store the same structures but this time serialized in XML instead. Sometimes a fresh start is the best course of action.
The benefit of open interfaces in computer systems is much the same the same as in real life. Take the common electrical plug for instance - it allows for a multitude of devices to be plugged into a multitude of sockets to draw power. You can be fairly certain that any one device will operate correctly plugged into any one of the vast numbers of sockets out there. And this is where the plug analogy becomes relevant - your valuable data becomes locked in, and only available via Microsoft's proprietary and commercial sockets.
From a developer's point of view, I think the standard is largely irrelevant. There are few incentives to support the format besides the creation of conversion tools. Simplicity has always been key in ICT and there is no reason to select OOXML in places where previously a custom format might have been used.
The danger of having OOXML accepted as an ISO standard is that Microsoft can and will use it as an argument in favor of selling Office into governments and companies concerned about the portability of their data. Of course in practice, with Microsoft being the only supplier of OOXML tools, the data is no better off than it was stored in their undocumented binary formats. Undoubtedly, Microsoft will also try to use its new-found standards compliance as a defense against monopoly accusations.
Adobe Photoshop Lightroom 2.0 Beta Now Publically Available
It was only a matter of time - Adobe Systems has countered Apple's release of Aperture 2 by offering a beta version of its upcoming competitor for anyone to download. The original beta testing program leading up to the release of Photoshop Lightroom 1.0 was a huge success by gathering useful user feedback while introducing digital photographers to a new breed of image processing tools.
Although it carries the Photoshop branding, Lightroom is mostly clean-room a implementation borrowing some concepts from Adobe Bridge and reusing the Camera Raw engine. The original beta release of Photoshop Lightroom (called just Lightroom at the time) also trailed the first release of Apple's Aperture. Both have completely redefined the "digital darkroom" by offering a streamlined solution for digital photographers, although arguably Lightroom has had a bigger impact by reaching both Mac and Windows users.
Notably, the Lightroom 2.0 beta adds two much-requested capabilities: multi-monitor support and localized adjustments. The former has always been present in Apple's offering, while the latter was recently introduced in Aperture 2.1. Apple still has an important advantage in that it offers 3rd party plugin SDK which allows independent developers to offer their image processing add-ons. For virtually all photographers, this would mean that the need to occasionally switch to a more powerful program such as Adobe's Photoshop CS3 for a small subset of retouching features is now eliminated.
So have a look at the release notes, and get downloading! More impressions to follow soon.