The Treachery of Sanctuary

We teamed up with Chris Milk and Ben Tricklebank to create the software behind their latest installation for the San Francisco Creators Project event.

Check out the Creators Project video of the story behind the installation.

Case Study

The Treachery of Sanctuary is three monolithic white frames towering above a still reflecting pool. Entering the space in front of the pool you notice your shadow appears within the frame as if you'd stepped before a bright light. A flock of birds swarms at the top of the panel. Reaching up to them you are surprised as your shadow begins to dissolve, transforming into hundreds of small birds fluttering upwards to join the flock. Within just a few moments your silhouette has completely disintegrated leaving no trace of you left in the image.

Moving to the second panel the flock above you is larger and menacing than the last. When you enter the birds begin to swoop down, attacking your shadow and taking chunks of you away in their claws. The onslaught continues until you've been devoured to just a pair of legs.

Entering the third panel your silhouette has returned. Upon swinging your arms upwards you are bestowed with a massive pair of wings unfurling from your shadow. Your wings follow your gestures, swaying as you move your arms.

Technical Approach

The implementation of Treachery stitched together several different technologies. We needed a way to visualize the viewers as silhouettes in front of the display such that we could augment the shadows, selectively removing parts or attaching wings. This meant we needed both the outline of the viewers visually as well as data points describing their actual posture in terms of torsos, arms and legs. In order to create a flock of birds we needed a way of efficiently animating hundreds of 3D models flying together and interacting directly with the silhouettes.

We chose to use both a 3D game development environment called Unity and an creative coding platform openFrameworks. OpenFrameworks was used to access a Kinect camera sensing the presence of people, which was passed to the game engine to show silhouettes interacting with the bird flocks.

Both the environments have unique strengths that made them work together well. OpenFrameworks has powerful libraries for processing data from cameras. It was easy to interface to Kinect cameras with openFrameworks. Specifically we used the Kinect for Windows SDK since it provides the most advanced pose tracking out there. For displaying the visuals Unity3d worked well as it's made for working with the lots of animated models.

Video Tracking

The system begins when a viewer steps in front of the screen. The Kinect camera discreetly built into the wall behind them recognizes the outline of the person and infers their posture. The raw image from the Kinect camera is quite noisy and requires several image filtering techniques to smooth its edges before its displayed as a shadow. We attempted several techniques to produce cleaner shadows, including Delaunay triangulation, contour finding, and edge detection. While many of those techniques generated interesting effects, they didn't result in the pure shadow look we were after. The solution we settled on used a specific combination of image filtering techniques. We used an erosion pass to remove small pixel glitches, followed by a dilation pass to fill any holes and finally a median blur to smooth things out. We averaged consecutive frames in time together to further stabilize the silhouette edges and remove noise.

Unity and openFrameworks integration

Since the data from the Kinect was received and processed in openFrameworks we needed a way to visualize the interaction in the game engine Unity. In the past we had used OSC network communication to connect the two applications, but that was way too slow for this installation. We decided to experiment with the more direct approach of a native C++ plugin for Unity. A plugin opened up the option of low level shared memory access between the openFrameworks and Unity processes. That way both applications could share the actual RAM location where the skeletons and shadows were stored. This was super fast.

Silhouette Masking Magic Trick

Once the silhouettes and skeletons were visible in Unity we needed a way to make them dynamic and interactive. The design for the first two panels required us to be able to progressively hide each viewer's silhouette as they interacted with the birds. To accomplish this we built a "particle suit" of small images and attached it to the skeleton data per viewer. The particles were not visualized directly but served as a reveal mask to let the silhouette image show. As the birds would emerge or rip apart the silhouette, the particles were progressively removed and those parts of the body would disappear. Because the particles were attached to the Kinect's skeleton data, it didn't matter where you waved your arm, if it had been ripped off it was gone!

Bird Animation and Flocking

Birds moved through the scenes using a custom flocking algorithm simulating bird flight. Flocking works by applying the same simple set of movement rules to each bird in the flock. The rules sound like "move towards the average center of all the birds" and "head in the same direction as the birds near you", or "move away from the birds if they are too close". By applying these simple rules to all birds every frame organic flocking animations emerge in the entire group. In order to have fine control of where the flocks swarmed we positioned a series of invisible points around the 3D scene. The birds were assigned points to fly towards which let us create paths for the flocks to follow.

The birds were created in Maya, a program for 3D animation traditionally used to create films and video games. Each bird had a series of animations done by hand as short loops or gestures, such as a peck, a flutter, or a dive bomb. By mixing code based animations like the flocking with hand done animations we achieved a dynamic and scriptable yet still organic looking behavior.

Once all the elements are in place the technology can sink into the background and let a seamless experience emerge.

Credits

Installation at The Creators Project, Ft. Mason San Francisco, March 2012
Project Concept and Direction: Chris Milk
Executive Producers: The Creators Project
Production: Radical Media
Executive Producer: Jennifer Heath @ radical.media, LLC
Design & Creative Direction: Ben Tricklebank @ Punk and Butler
Interactive Creative Production: Fake Love
Creative Director: Layne Braunstein, Josh Horowitz @ Fake Love
Technical Director: James George
Creative Software Development: Aaron Meyers, Brian Chasalow, Charlie Whitney
Producer: Catherine Badger
3-D Animators: Chelsea Hash, Sean Thorpe, David Larkin
2-D Design: Caitlin Morris
Audio Design: David Horowitz
Special Thanks: Eyebeam

Media