Digital Cameras

New software dramatically simplifies addition of objects to photos

New software dramatically simp...
Shadows from set geometry are accurately depicted on added objects
Shadows from set geometry are accurately depicted on added objects
View 14 Images
Multiple added objects can interact with one another
1/14
Multiple added objects can interact with one another
Reflections from added objects can alter original set lighting
2/14
Reflections from added objects can alter original set lighting
Schematic of work flow
3/14
Schematic of work flow
The various steps leading up to successful object addition
4/14
The various steps leading up to successful object addition
Reflections from added objects can alter original set lighting
5/14
Reflections from added objects can alter original set lighting
Example of various types of user input to refine results
6/14
Example of various types of user input to refine results
The algorithm does an admirable job with mirrors and reflections
7/14
The algorithm does an admirable job with mirrors and reflections
Virtual furnishing is one possible use of the algorithm
8/14
Virtual furnishing is one possible use of the algorithm
Shadows from complex shapes are accurately cast on set geometry
9/14
Shadows from complex shapes are accurately cast on set geometry
Added objects throw believable shadows on set geometry
10/14
Added objects throw believable shadows on set geometry
Reflections from added objects show up on set geometry
11/14
Reflections from added objects show up on set geometry
Reflections from adjacent objects show up on the added item
12/14
Reflections from adjacent objects show up on the added item
Translucent objects are handled as easily as opaque
13/14
Translucent objects are handled as easily as opaque
Shadows from set geometry are accurately depicted on added objects
14/14
Shadows from set geometry are accurately depicted on added objects

For more than twenty years, the software program Photoshop has been the industry standard for seamlessly manipulating images, especially the removal of unwanted items like blemishes, wires and telephone poles. When it comes to adding something to a photo, however, the process is still rather involved. Now, a team from the University of Illinois at Urbana-Champaign (UIUC), led by PhD candidate Kevin Karsch, is presenting a new algorithm at Siggraph Asia 2011 that promises to radically simplify the procedure of believably incorporating new or "synthetic" objects in still shots and the results are quite impressive.

To appreciate the elegance of the UIUC team's contribution, it's helpful to understand just what an improvement it is over current techniques. Look at any effects-heavy feature film and you'll see instances where computer-generated objects, be they giant robots, dinosaurs or futuristic vehicles, were added to previously-shot footage (also called a background plate in the Biz). In order to sell the illusion that the added objects were there in the first place, technicians typically need to gather copious amounts of on-set data (e.g. lens height and focal length, size of on-set objects and direction/color temperature of visible light sources, etc.) from which they recreate a virtual environment. When this is done poorly, it's a dead giveaway that the added object is fake and was inserted after the original image or footage was shot.

Multiple added objects can interact with one another
Multiple added objects can interact with one another

The genius of the Karsch team's approach lies in what the user doesn't have to provide in order to create believable additions. (Currently, the software only handles still images, but plans are in the works to tackle film and video sequences before long). Starting with a still photo about which nothing is known, novice users have been able to add virtual subjects in just minutes with results good enough to fool experts. That's great news for graphic artists, but maybe not so thrilling for folks like crime scene investigators who'll soon have to doubt the veracity of every object in frame!

Schematic of work flow
Schematic of work flow

To pull off this digital sleight-of-hand, the process is broken up into intuitive stages, the first being the creation of 3D geometry from 2D data. "The main idea is we want to take our picture and we want to get a 3D model of the scene from it," Karsch explained. "We first apply an automatic estimation procedure which tries to find the boundaries of the scene - it tries to find the walls, the ceiling, the floor." Since that's occasionally inaccurate, the software also includes an interface that allows the user to refine the estimate.

"With two vanishing points, we can recover the focal length of the camera, and then using the rest of the input we can also recover a rough representation of the 3D geometry of the scene: where the floors are at, where the walls are at and where the ceiling is at," Karsch added.

Example of various types of user input to refine results
Example of various types of user input to refine results

Next, the Lighting Stage requires the user to identify light sources in the image, whether in frame or not. In the traditional computer graphics approach, lighting sources and characteristics are identified by placing a chrome reference ball in front of the camera. The UIUC team's approach eliminates the need for that and a host of other steps as it calculates a decent representation of the actual lighting conditions.

If the object to be added will land behind another opaque item, the Occluding Object Stage (in which the user basically scribbles on the objects that will block part of the object to be added) comes next. Also called a spectral matting procedure, this generates an object mask which, when applied, creates the illusion that the added object is behind the masked one.

Enabling beginners to generate professional results with minimal time and effort is sure to make the Karsch team's algorithm a big hit in the graphics world. Judging from the sample images provided from the team's paper, the approach adeptly handles just about every lighting condition conceivable, even for transparent and shadowed subjects. For those of you itching to get your hands on it, plans are in the works to create an online version that anyone can use. In the meantime, it seems we'll just have to make do with run-of-the-mill reality!

Source: Kevin Karsch

Check out the video below for a look at the software in action:

Rendering Synthetic Objects into Legacy Photographs

10 comments
Janet Haines
Looks amazing but what is the name of this new software? We have the exhibition details and the developers name, but not the product name! And price? (Holding my breath here)
Mark Penver
Not gonna lie, but it does look kinda fake. Each image put into the stills is a 3d model. Of which you can alter easily for lighting and shading and drop ontop of photos. Not to mention it opens with a video despite the article mentioning they haven\'t been done yet.... Look keyshot up ;)
Krister Knutars
I don\'t need this for a second, but it\'s so amazing that I will definitely buy one as soon as it is released, just for the hell of having fun with it.
TheRogue1000
Hmmm....so now we can\'t believe ANYTHING we see. Amazing tech. I\'m sure it will be put to nefarious use immediately! LOL
Dave Andrews
Pretty awesome. Now they just need to make it work for video and make a plug in for Premiere and/or After Effects... Or simply sell the tech to Adobe so they can add it.
William H Lanteigne
...thus making it easier for criminals to fake documents or photographic evidence. Before this is released, they should develop the algorithms to detect inserted objects just as easily.
Leong Hee Chan
Guys in future beware what you see in any picture... digital or not!
Gregg Eshelman
First step on the road to criminals producing fake incriminating videos of influential and wealthy people to use for blackmail. Which of course the gullible sorts who\'ll believe absolutely anything - except for the truth - will believe in spite of proof the videos are fake.
tkj
this was demo\'d (? leaked?) months ago .. at least now we know the team responsible for it ..
Donald Thomas
I have been doing this for quite a while now using two picture editing programmes. See egsample in videos at youtube at https://www.youtube.com/watch?v=6GTrIyvbcK8