Audiences flocked to to the futuristic thriller Minority Report when it debuted in 2002, impressed not just with thefilm noir mystery, but also the visually stunning futuristic world depicted onscreen. So naturally there was a packed house at the Hammer Museum on April 22 to hear a talk called “Beautiful Tools” by artist/scientist John Underkoffler of Oblong Industries — part of a series of lectures sponsored by 5D on the future of immersive design. Underkoffler (who is an advisory board member of the Science & Entertainment Exchange) consulted on Minority Report, and drew on some of his own groundbreaking research at MIT while doing so. (He’s also consulted on The Hulk, Aeon Flux, Stranger Than Fiction, and Iron Man.)
The 5D conference “explores the profound impact of the convergence of art and science across all narrative media: film, game, animation and architecture,” and the organization sponsors an ongoing lecture series to particularly explore the new emerging relationship “between artists and scientists, designers and engineers, and the pervasive effect of this new collaboration not only on design and science process but as a fundamental change in the relationship between artifact and audience.”
Underkoffler opened with a still of the very first Apple computer introduced in 1979, at a time when the “computer” was all about creativity, or “making stuff.” You had to program in everything; there were no handy operating systems or cute little icons, no mouse, no “drag and drop” feature, and so forth. Fast forward to 2010, and we have ingenious handled deices like the iPod and the iPad — which are impressive pieces of industrial design but are essentially about media consumption. Underkoffler thinks it was the Web that changed things: “stuff is no longer just on your computer, it’s distributed, on a server, within cloud computing.”
He prefers to view the machine as an extension of the human: “a gift, an act of generosity, and an aesthetics of agency, if you will.” His vision was to build “a distinct ecosystem in which devices work as digital exoskeletons: amplifiers for human creative intent… to aid in making, in building, in design.” If you’re Underkoffler, and you’re at MIT, you start with a simple light bulb. It’s a useful medium with no particular “message,” but turn on a light and it enables us to work in the dark, for instance. Underkoffler decided to take this one-way illumination and transform it into a two-way conduit of information. Per this Website:
The I/O Bulb is based on a traditional light bulb but is able to not only project light, but also collect live video of the objects and surfaces it projects light onto. The Luminous Room may then be used as an optics simulator for a variety of purposes; Underkoffler sees the I/O Bulb as having applications for urban planning and architectural modeling, where planners and designers would be able to observe light patterns and reflections resulting from various arrangements of structures and buildings.
While working on his PhD, Underkoffler also designed a gestural interface system (now known as G-Speak), which allows users to navigate and interact with data by interpreting a user’s motion so the user move through datasets with no need for a computer mouse or any other physical object to do so.
Hollywood art director Alex McDowell (also an Exchange board member) heard about his work, and asked Underkoffler to consult on Minority Report to help director Steven Spielberh create a believable world 50 years into the future.
Most notably, the filmmakers decided to use Underkoffler’s work on the gestural interface system to build a forensic analysis display. They needed a cool new technology to help Cruise’s character sift through all the images collected from the “pre-cogs” and match them to information on file within their massive database. It would all be showcased on a gigantic curved display, using their hands to “conduct” the information — with no voice technology, no keyboards, and no mice.
Drawing on his prior work, Underkoffler literally invented a sign language for the film, drawing on bona fide sign language for the deaf, SWAT signals, and musical systems to synthesize into a new language. The actors had to study the “thesaurus” of gestures and practice them using training videos. When they put it all together — Lights! Camera! Action! — it looked like this:
It’s almost performance art. With G-Speak, “Physicality comes back into computing space,” said Underkoffler. “We built a real program with real language and trained real people to use it — even though it was fictional.” Oblong has since built a prototype G-Speak system: a new machine that can be used to design rather than just tedious tasks like file management, with an operating system that is not predicated on the use of a mouse. Even better, more than one user can collaborate in the virtual space at a time: it’s no longer just “one user, one screen.” Check out this video of Underkoffler and colleagues working their magic with the real-world G-Speak:
“The tools’ form ensures that acts of construction are often indistinguishable from acts of exhibition. Inherently, working means performing, whether anyone is watching or not,” says Underkoffler. “And underlying it all is a digital architecture that acknowledges space — the real-world geometry that structures the rest of existence — for the first time.”
Ideally, the Hollywood/science interaction should benefit both the entertainment and research communities, and Underkoffler’s work is a prime example of that. His research informed Minority Report, which in turn inspired him to develop his rudimentary system into a viable real-world prototype. And we have no doubt whatever he’s working on now will end up informing another film at some point in the future.
