Camera Tracking/3-D Tracking/Match-Moving?
Say what?

Although SynthEyes is being used on major Hollywood features, we expect it to interest many local and regional edit/effects studios and independent feature producers that might be new to the technology, but wish to "move up" to offer more sophisticated effects to their clients or audience — who increasingly expect Hollywood excitement even from local productions.  

Here, we've tried to look at what match-moving is about, how you can do it, and how to use it effectively with clients. For technical details such as import and export formats, see the feature list.

What is it?

"Camera tracking" or "Match-moving" or "3-D Tracking" is the process of analyzing a video clip or film shot to determine where in 3-D the camera went, what its field of view was, and where parts of the set were. The 3-D path of a large moving object can be determined as well.

Although SynthEyes is being used on major Hollywood features, we expect it to interest all kinds of folks who might be new to the technology, but wish to "move up" to offer more sophisticated effects to their clients or audience — who increasingly expect Hollywood excitement even from local productions.  

Why bother? What can I do with it?

The tracking information lets you add 3-D animated effects into live-action footage, such as

• Animated 3-D character insertion (mascots, beasts, flying pixies, you name it). A favorite for commercials.
• Vehicle insertion. Aircraft, boats, cars, trucks, spacecraft, things you wish you could get live, but couldn't afford or don't exist.
• Architectural Previsualization (ground level or helicopter)
• Crash reconstructions (from a boom, or vehicle P.O.V.)
• Virtual set extensions (boy, that's a fancy newsroom!)
• Matte paintings (ah, so that's what sunset on Mars looks like)
• Transparent effects. A broad catch-all for effects you really can't see, often fix-its: insert snow, puddles, volumetric smoke, a big tree to cover a phone pole, a vase to replace a microphone left in-camera.

Can't I do some of these already?

You can do some 2-D effects using a compositor, if the camera is locked off, on a tripod, or if amount of perspective shift is small.

Match-moving is used with the much more kinetic moving-camera shots that today's audiences crave. Even on simple shots you may want to use match-moving, as it can offer better speed and accuracy.

Describe the workflow for using SynthEyes.

Shoot. In SynthEyes, track features using the automated system or supervised trackers, solve for the camera and environment information, export to your existing 3-D animation or compositing system (Lightwave, 3ds max, Maya, Hash A:M, Inferno...). Using your 3-D system, create effects to insert. Render. Composite. Edit.

How do I shoot?

Most match-moving shots are shot from a Steadi-Cam, dolly (or wheel-chair cam), boom or crane, camera truck, or aircraft. SynthEyes works fine with hand-held shots, though you'll have to decide if your camera stability is adequate for your production. See the tutorials on the main web site.

What shots will it work on? 

To determine 3-D information, a shot must exhibit perspective changes as it evolves. Some features must be closer to the camera than others, and the relationship must change over time. If you could paint your shot onto a big cyclorama and re-shoot it from a camera on a tripod, there is no perspective. SynthEyes offers a special mode for tripod shots, but the result is 2.5-D, not 3-D.

There must be enough features to track (no empty white rooms or green backdrops). The features must be in the field of view long enough for their 3-D position to be determined. They can't be all bunched up in one portion of the image.

What's this about features?

A feature is a small spot that can be tracked from frame to frame, including scuff marks on floors, clumps of weeds, light switches, the corner of a box, etc. 

What shots won't work?

One of the common novice shooting problems is to use a hand-held videocamera and wave it around while primarily standing still.. This results in a non-3-D shot, where the camera is effectively on a tripod. It does not produce true perspective shifts. Furthermore, as the camera whips around, trackers fly on- and off-screen rapidly. True 3-D can not be determined.

Any "rules of thumb?"

Make the camera focus on a single area as it moves to approach or circle this focus of attention.

Make sure the shot is busy enough that there are features to track. Make sure that they extend over most of the frame (especially when tracking an object).

The client wants to shoot an actor against blue-screen with some markers, and insert him/her into a 3-D environment. Anything to look out for?

Beware of the client zooming in for the head shot, and taking most/all the trackers out of the frame. The pure blue screen shots are dangerous just because the background is so featureless that if you mess up and have trackers out of frame, you can wind up in big trouble. And there's a tendency to put some marks on a back wall way behind the actor, focus on the actor, and wind up with a very flat shot with little/no perspective on the tracking marks. When this is the case a 2-D track is the right/only thing (using SynthEyes's tripod mode).

For a higher probability of success, you want the shot to encompass the presenter from a distance, so that tracking marks can be seen not only on the back wall, but on the floor and possibly side walls, depending on camera motion. Having trackers visible at a variety of depths makes tracking much more sane. And there's no such thing as too many tracking marks. Odds are that a moving-camera shot will move the field of view around so that only a fraction of them are visible at a time, unless it has been knowledgably planned. If there's a desk or other props in the scene, that can be helpful, as long as it has some trackable features. 

What's this about tracking objects?

If there's a large object moving around in the camera field of view, you can track the object in 3-D, even if the camera was locked off. In your animation package, you'll see the object moving relative to the camera. With the right scene, the camera can be moving too, carrying the object along.

Can I track an actor's head as an object?

Yes! It takes a bit of care and experience and some advance preparation as well. Use a tight shot of the head, with a shorter focal length lens --- by shooting up close there will be more perspective to get 3-D. There have to be plenty of trackable points on the head at all times. They have to be on parts of the head that don't move relative one another---trackers on the jaw, mouth, eyebrow are bad with the actor talking. Usually the tracking features are hand tracked off tiny details on the face and head. Sometimes drawing some dots on the actor helps too. Use high-resolution source such as an HDTV camera or film ---- VHS camcorder source is probably a non-starter. Progressive-scan DV is a good idea.

If SynthEyes can track and solve automatically, why does it have supervised and manual capabilities also?

It's like cruise control on the car: handles the boring stuff, but switch to manual for heavy traffic. SynthEyes lets you mix and match, even in the same shot.

What do I do in my 3-D animation package?

You'll open the scene as exported by SynthEyes, to find a camera moving along the computed path, with the correct field of view, plus a collection of tracker points, which are nulls or points or axes, depending on the package. Your shot will appear as the camera's background.

You'll add computer-generated objects into this scene, using the tracker points and original shot as guides. You'll add matching lights (or use ones located and imported from SynthEyes), and generally adjust your materials and texture maps to match the scene.

Often, you'll add stand-in geometry (planes, boxes, cylinders) to act as shadow catchers to help sell the integration of the effects with the scene. Sometimes you may use camera mapping to achieve more complex matching of perspective.

Can I really do this?

Yes! Like using any sophisticated program, such as an animation package, compositor, or Photoshop, more experience helps.

Start with the tutorials in the free demo, then move on to some of your own footage. Learn to recognize shots that can be tracked easily. Then you're ready to try some projects with clients.

If possible, offer on-set supervision as a service: a little cost up front may save the client much more later (and you head-aches). You can run SynthEyes on a laptop on-set, so if you're shooting DV, you may be able to track shots immediately before the shoot is wrapped. (SynthEyes reads DV in AVI or QT movies).

Will SynthEyes make my project look like a $150M budget?

Not by itself, that takes skill and practice too. But SynthEyes can help you take you there, and stay there. 

Did this help?

Let us know!

©2003 Andersson Technologies LLC

SynthEyes: Integrating Imagination with Reality™