Imaging Products
  1. Home
  2. Products & Support
  3. Imaging Products
  4. History & Technology
  5. Behind the scenes
  6. Scene Recognition System

Scene Recognition System

Before shooting a scene, a professional photographer adjusts focus, exposure and white balance based on his or her experience. Nikon's Scene Recognition System represents a completely new concept whereby the camera makes this series of judgments as if it were the photographer's brain. We would like to share with you some of the views of the developer of the Scene Recognition System, which has been introduced in the Nikon D3 and D300 digital SLR cameras.

Hiroshi Takeuchi
1st Development Section, 3rd Development Department, Development Headquarters, Imaging Company
PROFILE:
Joining Nikon in 1995, Hiroshi Takeuchi was first engaged in developing light metering technology for film cameras. He then became involved in the development of metering for digital SLR cameras and TTL flash control technology. Mr. Takeuchi is the primary force behind the development of the Scene Recognition System incorporated in the D3 and D300. On his days off, Mr. Takeuchi enjoys taking photos and videos of his young daughter with his COOLPIX and playing samba music with friends.

Simulation of judgments made by the photographer using a combination of advanced optical technology and data processing technology

The D3 and D300 are equipped with the Scene Recognition System, which has been attracting much interest since its announcement to the media. First of all, please tell us about the system.

“Technically, the Scene Recognition System uses color, brightness and other information obtained from the 1,005-pixel RGB sensor to analyze the subject prior to capture and applies the results to achieving greater accuracy in AF (autofocus), AE (auto exposure), and AWB (auto white balance) control. The 1,005-pixel RGB sensor that was first introduced in the F5 has been improved in the area of exposure accuracy, mainly as a light meter. By adding a diffraction grating between the prism and the lens for light metering, the 1,005-pixel RGB sensor is able to detect the color and brightness of the subject more accurately, significantly improving accuracy in AE and AWB. AF benefits as well, because the sensor even recognizes changes of the subject's position within the viewfinder.”

Is information obtained from the 1,005-pixel RGB sensor really applied to all three of these areas of AF, AE, and AWB? If so, this is a completely new concept in the ever-evolving performance of camera technology, isn't it?

Mr. Takeuchi explains the 1,005-pixel RGB sensor by removing it from the D3's finder unit.

The diffraction grating forms a clear image on the sensor.

“Yes. It is correct to say that this is an epoch-making technology that has changed in a fundamental way the usual thinking about cameras. While 3D color matrix metering has vastly improved exposure accuracy with information obtained from the sensor, in the Scene Recognition System the recognition of various other elements beyond brightness has been improved to a level close to that of the human brain. The idea with the Scene Recognition System is that complex judgments that photographers make while viewing the subject through the viewfinder, such as “What is coming into the photo here?” or “How is the subject moving?”, can now to some extent be left up to the camera. “Until now, we have been improving AE, AF, and AWB technologies independently, but when these technologies matured, there was no technology capable of linking them. The Scene Recognition System is thus a core technology that links the superior aspects of these three technologies. It is second nature for professional photographers in particular to make detailed calculations in their head concerning exposure, focusing, and white balance based on experience, but color and brightness information captured by the 1,005-pixel RGB sensor provides support for these complex calculations.”

Isn't the diffraction grating a major point in the hardware that supports the ability of the Scene Recognition System to make judgments closer to those made by the brain of a professional photographer?

“Yes, at first glance, the diffraction grating only looks like a small plastic plate, but the zigzags you see here are actually very important. The diffraction grating is a special filter with a minute, nanometer-order structure. It diffracts light into red, green and blue, and radiates it more appropriately to the 1,005-pixel RGB sensor. By more fully bringing out the capabilities of the sensor, the light that has been separated into red, green and blue is reproduced in more vivid colors, allowing high-grade calculation of information for AE, AF, and AWB. Since the calculation process itself is a technology that combines various types of information, the calculations are of an extremely large scale. The development behind the Scene Recognition System could thus be described as a synthesis of advanced optical technologies and data processing technologies.”

Dramatically improved performance and functionality with the application of scene recognition to AF, AE, and AWB

Specifically, what sorts of functions is Scene Recognition being applied to?

“First of all, the primary features of Scene Recognition are Subject Identification and Subject Tracking. Initially, background and foreground are identified in the information obtained from the 1,005-pixel RGB sensor, and then the color of the skin of people comprising the subject is identified. A judgment is also made on the size of that section. This subject identification information is used in Auto-area AF mode to automatically focus on people in the scene. It is also now possible for planar movements of the subject to be detected and reflected in Dynamic AF. Even when the subject moves out of the focal area, it is always possible for the subject to be photographed in focus. Subject Tracking makes this 3D-Tracking mode possible.”

That’s amazing. The difference is immediately visible when we look through the viewfinder. This would be especially useful in areas like sports photography.

“Yes, exactly. The 51-point AF system employed in the D3 and D300 has significantly improved AF performance, but now with the advent of 3D-Tracking, rather than constantly tracking the subject while keeping it in the center, it's possible to freely create a composition and shoot while tracking the subject. At a tennis match, for instance, you might like to change the space to the left or right depending on whether the player is using a forehand or a backhand swing. I think a great attraction of 3D-Tracking is the ability to change angles in line with rapid movements.”

A simple question concerning skin color recognition, since there is such a wide range of skin colors: Does the Scene Recognition System also recognize differences in skin color?

“Differences in brightness make skin colors look different to the human eye, but actually all types of skin have a tendency to look very similar. When a subject is recognized, the judgment made is based on the tendency of skin color.”

What about the case of AE or AWB?

The Scene Recognition system enables
the identification of the source of light,
dramatically improving white balance
performance.

“Highlights within the frame are analyzed in detail and applied to matrix metering and i-TTL balanced fill-flash to achieve greater accuracy in exposure control. With AWB, in the past, light meter data was applied to white balance, but now more complex analyses are used to identify the appropriate light source even under any difficult light conditions. For example, the green inside a forest and the green light of a fluorescent lamp look the same to a camera; it's difficult to see from a single image what sort of green it is. The Scene Recognition System uses not only information from the scene for the photo but also information from the 1,005-pixel RGB sensor that it matches against a database to identify the light source, thus improving accuracy in determining whether it is a green seen outdoors or from a fluorescent lamp indoors.”