Bringing the Physical to the Digital

This dissertation describes an exploration of digital tabletop
interaction styles, with the ultimate goal of informing the design
of a new model for tabletop interaction. In the context of this
thesis the term digital tabletop refers to an emerging class of
devices that afford many novel ways of interaction with the
digital. Allowing users to directly touch information presented on
large, horizontal displays. Being a relatively young field, many
developments are in flux; hardware and software change at a fast
pace and many interesting alternative approaches are available at
the same time. In our research we are especially interested in
systems that are capable of sensing multiple contacts (e.g.,
fingers) and richer information such as the outline of whole hands
or other physical objects. New sensor hardware enable new ways to
interact with the digital. When embarking into the research for
this thesis, the question which interaction styles could be
appropriate for this new class of devices was a open question, with
many equally promising answers. Many everyday activities rely on
our hands ability to skillfully control and manipulate physical
objects. We seek to open up different possibilities to exploit our
manual dexterity and provide users with richer interaction
possibilities. This could be achieved through the use of physical
objects as input mediators or through virtual interfaces that
behave in a more realistic fashion. In order to gain a better
understanding of the underlying design space we choose an approach
organized into two phases. First, two different prototypes, each
representing a specific interaction style – namely gesture-based
interaction and tangible interaction – have been implemented. The
flexibility of use afforded by the interface and the level of
physicality afforded by the interface elements are introduced as
criteria for evaluation. Each approaches’ suitability to support
the highly dynamic and often unstructured interactions typical for
digital tabletops is analyzed based on these criteria. In a second
stage the learnings from these initial explorations are applied to
inform the design of a novel model for digital tabletop
interaction. This model is based on the combination of rich
multi-touch sensing and a three dimensional environment enriched by
a gaming physics simulation. The proposed approach enables users to
interact with the virtual through richer quantities such as
collision and friction. Enabling a variety of fine-grained
interactions using multiple fingers, whole hands and physical
objects. Our model makes digital tabletop interaction even more
“natural”. However, because the interaction – the sensed input and
the displayed output – is still bound to the surface, there is a
fundamental limitation in manipulating objects using the third
dimension. To address this issue, we present a technique that
allows users to – conceptually – pick objects off the surface and
control their position in 3D. Our goal has been to define a
technique that completes our model for on-surface interaction and
allows for “as-direct-as possible” interactions. We also present
two hardware prototypes capable of sensing the users’ interactions
beyond the table’s surface. Finally, we present visual feedback
mechanisms to give the users the sense that they are actually
lifting the objects off the surface. This thesis contributes on
various levels. We present several novel prototypes that we built
and evaluated. We use these prototypes to systematically explore
the design space of digital tabletop interaction. The flexibility
of use afforded by the interaction style is introduced as criterion
alongside the user interface elements’ physicality. Each
approaches’ suitability to support the highly dynamic and often
unstructured interactions typical for digital tabletops are
analyzed. We present a new model for tabletop interaction that
increases the fidelity of interaction possible in such settings.
Finally, we extend this model so to enable as direct as possible
interactions with 3D data, interacting from above the table’s
surface.