Synlab: Papers

In this paper, we will introduce the ROSS framework, an integrated application development toolkit that extends across different tangible platforms such as multi-user interactive tabletop displays, full-body interaction spaces, RFID-tagged objects and smartphones with multiple sensors. We will discuss how the structure of the ROSS framework is designed to accommodate a broad range of tangible platform configurations and illustrate its use on several prototype applications for digital media content interaction within education and entertainment contexts.

In this paper we present Responsive Objects, Surfaces, and Spaces (ROSS) API, a tangible toolkit that allows designers and developers to easily build applications for heterogeneous network devices. We describe the unique nested structure of the ROSS framework that enables cross-platform and device development and demonstrate its capabilities using several prototype applications.

We present a hybrid positioning and communication system for tracking interaction objects called 'pucks' on the surface of a large LCD or plasma display. Pucks are smart sensor packages consisting of a microcontroller as well as a contact-type acoustic receiving transducer and an infrared or radio data link. A puck may take the form of a standalone interaction object, or the puck circuitry may be integrated into an existing object, such as a digital camera, cellphone, PDA, or other device. In this work we take advantage of the glass surface atop an LCD or plasma display as a communication and sensing medium, and launch 200KHz Gaussian-shaped acoustic ranging pulses into that medium from transmitting transducers adhered to the corners of the glass. We present experimental results demonstrating millimeter-scale puck positioning accuracy over the entire surface of a 32-inch LCD, at an update rate of 100Hz. We also demonstrate the scalability of this approach to much larger displays. In our first implementation, power consumption of each puck is 3V at 12mA during data transmission, 5mA during positioning, and 50Î¼A when idle, yielding 6-8 hours of continuous tracking from a 90mAH prismatic lithium polymer battery.

Many research efforts today explore how digitally augmented tables enable face-to-face interaction with digital content and applications. Yet the design of digital tables is still largely driven by the constraints and requirements of the underlying sensing technologies. In order to move digital tables into our real-world physical spaces, researchers need to work closely with architects and industrial designers in order to engage the knowledge and skills from a long history of physical design and fabrication in the creation of tabletop systems. This paper presents Architales, an interactive story table for gallery exhibition, developed as an experiment in physical/digital co-design. We describe the creation of the interactive table, tangible storytelling system, and story content, which evolved together in a closely unified design process, drawing on the skills from computing, media, design and architecture. We also describe lessons learned about the interdisciplinary design process and the creation of physical/digital artworks.

Many research efforts today explore how digitally augmented tables enable face-to-face interaction with digital content and applications. Yet the design of digital tables is still largely driven by the constraints and requirements of the underlying sensing technologies and digital systems. In order to move digital tables into real-world physical spaces, researchers need to work closely with architects and industrial designers in order to engage the knowledge and skills from a long history of physical design and fabrication. Architales is an interactive story table for gallery exhibition developed as an experiment in the physical/digital co-design of the physical table and environment with the digital story system and content.

We review the creative activity in the software development process of the ROSS (Responsive Objects, Surfaces, and Spaces) API. The ROSS API is a tangible toolkit that allows designers and developers to easily build applications for many different tangible platforms while still accommodating the continued evolution of the underlying sensing technologies. This paper describes the ROSS API specification, identifies its originality and relevance to creativity, discusses the challenges of the API development process, and summarizes the lessons learned.

In screen-based experiences, the screen itself can become the physical device used for interaction. The "move-ability" of the screen affords interactivity between the screen artifact and the viewer and between the virtual space and the physical space. As a movable screen interface, I have created the spinning screen, a two-sided monitor mounted on a revolving base. This interface invites user interaction through the action of spinning the screen. User interactions with the spinning screen can support diverse temporal and spatial responses, thereby enriching usersâ€™ experiences. The spinning screen enables the viewer to grasp the interplay between visibility and invisibility, creating an aesthetic experience. The angle and direction of rotation affects the displayed visuals and audio output.

In screen-based experiences, the screen itself can become an interactive element. The "move-ability" of the screen affords interactivity between the screen artifact and the viewer and between the virtual space and the physical space. Cross-Being: Dancer features a movable screen interface, a spinning screen based on a two-sided monitor mounted on a revolving base.

The ontological condition of the screen is "in-between" several boundaries: the boundary between space and time, virtual and real, and immersion and rejection in the sense of similarity and difference. By re-categorizing, connecting and blurring these existential boundaries around the screen, the approaches described here form two kinds of screen experiences: three-dimensional screens in video installations, and 'movable' screens in real-time video and sound installations.

Flora Electronica is a collaborative artwork created by artists and researchers during a course in tangible media arts. This playful artificial/computational garden is a stage where spontaneous media performances unfold, manifested across both the physical and virtual worlds. The emotional tone of the space is set by the real-time perceptions/interactions of a participating audience, both local and remote.

We investigate the cognitive connection players create between their own bodies and the virtual bodies of their game avatars through tangible interfaces. The work is driven by experimental results showing that execution, perception and imagination of movements share a common coding in the brain, which allows people to recognize their own movements better. Based on these results, we hypothesize that players would identify and coordinate better with characters that encode their own movements. We tested this hypothesis in a series of four studies (n=20) that tracked different levels of movement perception abstraction, from own body to that of an avatar’s body controlled by the participant, to see in which situations people recognize their own movements. Results show that participants can recognize their movements even in abstracted and distorted presentations. This recognition of ‘own’ movements occurs even when people do not see themselves, but just see a puppet they controlled. We conclude that players – if equipped with the appropriate interfaces – can indeed project and decipher their own body movements in a game character.

We have developed an embodied puppet interface that translates a player’s body movements to a virtual character, thus enabling the player to have a fine grained and personalized control of the avatar. To test the efficacy and short-term effects of this control interface, we developed a two-part experiment, where the performance of users controlling an avatar using the puppet interface was compared with users controlling the avatar using two other interfaces (Xbox controller, keyboard). Part 1 examined aiming movement accuracy in a virtual contact game. Part 2 examined changes of mental rotation abilities in users after playing the virtual contact game. Results from Part 1 revealed that the puppet interface group performed significantly better in aiming accuracy and response time, compared to the Xbox and keyboard groups. Data from Part 2 revealed that the puppet group tended to have greater improvement in mental rotation accuracy as well. Overall, these results suggest that the embodied mapping between a player and avatar, provided by the puppet interface, leads to important performance advantages.

We report on an ethnographic study of the work practice and discovery process in a systems biology lab, and outline a tabletop visualization that was developed based on this study, in collaboration with the researchers. The feedback from the researchers on the current prototype is presented, and ongoing revisions are outlined. We conclude with some of the challenges involved in developing such tangible visualizations for discovery.

Computationally-enhanced tabletops featuring tangible objects for participant interaction hold promise for the creation of new forms of media, including documentaries and stories constructed by individuals or small groups in distributed or face-to-face settings. Many computer-aided storytelling systems feature either algorithms to arrange story elements automatically or story structures to guide authors in presenting their tales. In contrast, the KinoPuzzle system affords a high degree of freedom for authors in constructing tabletop experiences and for participants in exploring multi-viewpoint narratives. This format for story-telling combines the representational depth and flexibility of the digital database with the expressive power of the collage surface, offering advantages in terms of open-ended dialogic juxtapositions and the collaborative exploration of reality-based material.

Seeing multiple moons visible overhead is not our normal experience in the real world. Even though we can picture this kind of experience in our imaginations, we cannot properly share it with others. It can only visually exist in our own individual minds. We would like to share this experience by creating a poetic installation space of virtual and real, where multiple moons can follow us around. In this gallery room, the audience can share their own imaginative and individual experience with others.

In this paper we describe the concept, design, and implementation of Murmur, an interactive kinetic display made of one hundred computer CPU fans. Murmur responds to sound input from its environment via embedded microphones to produce patterns on a reactive surface. The reactive surface consists of hinged paper pieces situated in front of each fan. When activated by sonic elements in the environment, including sounds intentionally generated by an interactor, Murmur responds by turning on and off its fans in a sequence. The wind pressure generated by the movement of the fans stimulates the surface, forcing the paper up and out to create a variety of dynamic patterns. Each pattern represents characteristics of the sonic environment. We also analyze the feedback received from the interactors and discuss the possible ways of making the interaction more immersive.

A transparent display is a display that allows people to see through it from either the front or the back. In this paper, we propose a one-way pseudo transparent display technique that allows users in front of a monitor to see through the screen.

In this paper we discuss designed features of a cross-reality collaborative environment that are relevant to OUIs. These include responsive furniture, paper-based interfaces and a mappable project space. We identify three impediments to a truly flexible workspace caused by "non-organic" technologies in this environment: brittleness, limited resolution and technology-imposed boundaries. Finally, we speculate on the transformative impact that OUIs could have in this application space.

Pictures at an Exhibition is a physical/digital puppetry piece that uses tangible interface puppets to modify a virtual scene projected at the back of the stage in real-time. Telling the story of a heist in an art gallery, the actions of the puppets are digitally reflected in abstract changes to an artwork in the scene. The piece merges traditional puppeteering practices with tangible interaction technologies and virtual environments to create a novel performance for the live stage.

We describe the design and development of PuzzleTale, an interactive storytelling system that makes use of tangible puzzle pieces on the surface of an interactive table. In the PuzzleTale system, assembling the tangible puzzle piece can affect the digital characters and create a flexible story context. Different assembled patterns represent the diverse ways that users explore and compose the story. The PuzzleTale system creates a dynamic causal relationship between the process of interactive storytelling and the outcome of a story. There are two variables which continuously affect the story context including the amount and the sequence of digital characters that the reader plays. The system engages the reader participating in the development of the story through playing tangible puzzle pieces with digital characters as a decision making tool, thus providing individualized storytelling experiences.

Recent work in neuroscience suggests that there is a common coding in the brain between perception, imagination and execution of movement. Further, this common coding is considered to allow people to recognize their own movements when presented as abstract representations, and coordinate with these movements better. We are investigating how this 'own movement effect' could be extended to improve the interaction between players and game avatars, and how it might be leveraged to augment players' cognition. To examine this question, we have designed and developed a tangible puppet interface and 3D virtual environment that are tailored to investigate the mapping between player and avatar movements. In a set of two experiments, we show that when the puppet interface is used to transfer players' movements to the avatar, the players are able to recognize their own movements, when presented alongside others' movements. In both experiments, players did not observe their movements being transferred to the avatar, and the recognition occurred after a week of the transfer. Since the recognition effect persisted even with these two handicaps, we conclude that this is a robust effect, and the puppet interface is effective in personalizing an avatar, by transferring a player's own movements to the virtual character.

RENATI is an acronym for recontextualizing narratives for tangible interfaces. It serves as an umbrella term for our art/research experiments within a hybrid environment that uses oral narratives, and non-generative and immersive art with sensing technologies to create tangible narratives. In this paper we introduce our first prototype, which uses a custom-built mannequin to allow viewers to engage with a multi-viewpoint story titled Flying Over Purgatory.

RENATI is an acronym for recontextualizing narratives for tangible interfaces. It serves as an umbrella term for our artistic experiments within a hybrid environment that uses various forms of narratives, such as oral narratives, diaries, essays, screenplays, and non-generative and immersive art with sensing technologies, to create tangible narratives.

The Shadow Box is a tangible computing project that exploits visual association and auditory clues to teach children the representational relationship between words and their meanings. The Shadow Box contains three major components: the main box, picture blocks and word blocks. The Shadow Box activates when a block or a matching pair of blocks is placed inside. The box prompts children to find matching blocks and combine them together. When children successfully combine the right word and picture, the box rewards them with an animated video as if they had made the objects come alive. An informal study shows that children responded positively to the concept of the box. They played with it for a length of time and engaged in a collaborative learning process with other children.

Computer mediated group collaboration, particularly in the design and engineering disciplines, is in need of better applications that suit the needs of effective exchange of information. Multi-touch surfaces offer the capabilities to augment and better enable face-to-face interaction with digital content and applications. This paper presents the design, development and testing of SketchTop, a multi-touch sketching application for collocated design collaboration.

Tangible Anchoring explores potential practices for new reporting and analysis afforded by the convergence of tabletop computing forms, broadcast media, and user-generated content via mobile, interactive television, and online technologies. Our system applies these technologies in a novel combination to support new forms of dialogue for broadcast television, with a focus on polling, and the techniques prototyped are also applicable for webcasts, meetings and art installations. The system enables tabletop computing users (e.g. television anchors or moderators) to receive and mix for redistribution both quantitative and qualitative data, i.e. viewpoints, from participants in remote locations or face-to-face settings. Our design scenario for the television studio assumes multiple camera angles, two or more discussants, multiple displays, and issues-based polls using mobile phones. Our current prototype features viewer viewpoints represented on a tangible tabletop broadcast anchor desk. In this paper, we present our design rationale and feedback on the application from industry professionals and colleagues.

Tangible Comics is an attempt to redefine comicsâ€™ conventions for a narrative performance in a digital environment. It seeks ways of taking digital comics out of the computer screen and merging comicsâ€™ conventions into three-dimensional real space. The installation will utilize the qualities of digital games, interactive narration, digital art, performance, and augmented reality.

Tangible Comics is a computer vision based full-body interactive storytelling environment that also functions as a comics generator. Prevailing applications of full-body computer vision have not utilized the full storytelling or performance potential of these environments. Our aim is to produce an environment that can create a space for redefining the conventions of comics, performance, film, photography, and animation. In relation to that, we are exploring the design problems that can arise when computer vision technology is contextualized in an interactive story telling environment.

Emergent game formats, such as machinima, that use game worlds as expressive 3D performance spaces have new expressive powers with an increase of the quality of their underlying graphic and animation systems. Nevertheless, they still lack intuitive control mechanisms. Set direction and acting are limited by tools that were designed to create and play video games rather than produce expressive performance pieces. These tools do a poor job of capturing the performative expression that characterizes more mature media such as film. Tangible interfaces can help open up the game systems for more intuitive character control needed for a greater level of expression in the digital real-time world. The TUI3D project (Tangible User Interfaces for Real-Time 3D) addresses production and performative challenges involved in creating machinima through the development of tangible interfaces for controlling 3D virtual actors and environments in real-time. In this paper, we present the design and implementation of a tangible puppet prototype for virtual character control in the Unreal game engine and discuss initial user feedback.

In this paper, we introduce approaches to navigating and manipulating objects in a Collaborative Virtual Environment (CVE) that engage tangible objects and an interactive table interface. We also identify three design concerns that are common to physical-virtual connectivity for interaction with CVE systems. At last, we propose solutions to these issues within the context of CVEs.

This paper presents Tangible Tracking Table, an interactive tabletop display, and Optical Chess, a strategy game. We discuss the design and implementation of both systems and report our evaluation game play sessions with young adults, with a special focus on how the Tangible Tracking Table enhances interaction over a point-and-click interface.

The workshop aims to bring together researchers and practitioners working on subjects related to digital games with tangible interaction. We would like to involve participants with backgrounds in academia as well as industry, from diverse fields such as HCI, computer science, edutainment, interaction design and game design. Some of the topics we plan to address during the workshop are: different tangible interaction styles, designing for specific game types, and the advantages and disadvantages of different sensing and object tracking technologies.

We are creating unified programming tools that allow developers to easily build applications for many different tangible platforms, and that can accommodate the continued evolution of the underlying sensing technologies. The Synlab API extends across several media table object-tracking platforms, RFID tagged object interactions, manipulable displays and multi-touch surfaces.

We developed Tangible Tracking Table (TTT), an interactive tabletop display that allows both fingertip and tangible object tracking. Multiple users can simultaneously control a variety of tagged objects that can perform different tasks. We also introduce several applications on TTT that utilize the advantages of this tangible user interface.

In this paper, we describe the design and implementation of Teaching Table, an interactive tabletop audio-visual device aimed at enhancing the learning experience for prekindergarten children by involving them in physical activities. Using electromagnetic sensing technology, the table can track tagged objects placed on its surface, accurately identifying their type and location while providing a coincident visual display and audio feedback. Teaching activities that are aimed at developing early math skills have been created for the table in alignment with standard curriculum guidelines for pre-K schools. Additionally, we include software based assessment tools for mentors/teachers to easily track an individual child's progress during the process of interacting with the table.

In screen-based experiences, the screen itself can become the physical device used for interaction. The "move-ability" of the screen affords interactivity between the screen artifact and the viewer and between the virtual and physical spaces. As a movable screen interface, we have created the spinning screen, a two-sided monitor on a revolving base. This interface invites user interaction through the action of spinning the screen to manipulate media content. By allowing viewers to grasp the interplay between visibility and invisibility, the spinning screen can enrich user experiences of virtual imagery through direct tangible interaction in both playful and practical contexts. We describe an art piece and an application for 3D visualization and modeling.

The past several years of computer interaction research have shown an increasing interest in tabletops for shared user interactions through touch or tangible objects. Digital media tables offer the potential to expand our digital interactions into casual social settings that are not appropriate for desktop platforms, such as home living rooms. We have developed a tangible media table called TViews, which provides an extensible architecture to enable multiuser interactions with a range of media applications and content via tagged tangible objects. The TViews object positioning utility functions on the surface of an embedded display and enables real-time tracking of a virtually unlimited set of uniquely identified wireless objects that can be used on the surface of any similar table. These objects can be physically customized in order to suit particular applications, and can provide additional functionality through external input and output elements on the objects themselves. In this paper, we present a first field trial of TViews to gain some initial insight into how such a device could be adopted in a real-world home.

The TViews Table Role-Playing Game (TTRPG) project explores the creation of digital tabletop role-playing games on the TViews table, in order to bridge the separate worlds of traditional role-playing games with the growing area of massively multiplayer online role-playing games. The TViews table is an interactive tabletop media platform that can track the location of multiple tagged objects in realtime as they are moved around its surface, providing a simultaneous and coincident graphical display. In this paper we present the implementation of the first version of TTRPG, with a content set based on the traditional Dungeons & Dragons rule-set. We also discuss the results of a user study that used TTRPG to explore the possible social context of digital tabletop role-playing games.

In screen-based experiences, the screen itself can become the physical device used for interaction. The "move-ability" of the screen affords interactivity between the screen artifact and the viewer, and between the virtual and physical spaces. We have created a movable screen interface, called the tilting table, which provides a display surface via overhead projection. This interface invites user interaction through the action of tilting the tabletop. The degree of tilt affects the displayed visuals and audio output. This simple interaction makes users feel they have a closer connection to the virtual imagery, and the screen thus blurs the boundary between them. In this paper, we introduce four applications that have been implemented for the tilting table. Two use an artistic approach to create expressive and entertaining media interactions, while two have been developed for navigating large information spaces. Each application is described and user feedback is discussed.

Traces is an ambient installation that uses the knowledge of the intersection of bodies with floors and walls in order to represent the absence of those bodies. By acting, users explore the traces of their past actions and their absence from those actions. The technique used is based on visual hull reconstruction from multiple cameras on the GPU.

The Graphical User Interface (GUI) has created an efficient work environment for many applications. However, when users are confined by keyboards and mice, they lose the ability to interact with the virtual world using habits from the real world. Our research examine how emerging modes of authorship, such as wikis, can be used to generate new possibilities for bringing atoms and bits together for digital annotation. Our goal is to combine the everyday habits in reading books with emerging digital possibilities. In this paper, we present a prototype system called WikiTUI, which brings digital media to physical paper books. This system allows readers to access the digital world through fingertip interactions on books, and enables them to share information with other readers using wiki technology. WikiTUI not only bridges the gap between the digital and the physical worlds, but also facilitates multiple contributions to a reference base spanning across these worlds. We present user evaluations of the WikiTUI prototype and discuss research implications.

Radio Frequency IDentification (RFID) technology has recently been moving into everyday use contexts. Previous work has shown wearable RFID systems to be a viable mechanism for collecting data about a userâ€™s interactions with her environment. In this paper, we present wearable RFID systems as a promising new direction in tangible game interfaces. We provide an overview of the affordances of RFID for game-play, and present some existing and future wearable RFID-based games. We also describe the construction of a cheap, easy-to-build wearable RFID system and present a how-to resource for other researchers interested in building off our work.

WiiArts is an experimental video, audio and image processing art project that makes use of pre-existing sensing technologies provided by Nintendo WiiRemotes and a Sensor Bar. Currently, most WiiRemote-based physical interactions have been designed to mimic the gesture of body movement in sports and other action-based games. These Wii games are generally competitive in nature, and players interact by responding to predefined interaction rules in either a single-user or multi-user mode. Making use of the WiiRemote as a pre-existing tangible and embedded interface, we explore applications that can engage participants in active and expressive art creation in a collaborative manner. In this paper, we describe several prototype applications based on this concept: Illumination (draWiing), Beneath (Waldo), WiiBand, Time Ripples. In these applications, three interactors can work together to compose both images and sounds.

WikiTUI is a tangible multimedia annotation system that allows users to manipulate multimedia information in common physical books. WikiTUI also facilitates the exchange of information with other readers through wiki services. Using tangible interface techniques, we seek to bring the collaborative annotation and authoring capabilities that are supported by the growing space of online wikimedia into the real world, thereby extending the use of paper books into new realms.

This article describes the design of WikiTUI, a tangible user interface system that allows readers to manipulate multimedia information tied to page-level locations in common books. This system will also facilitate the exchange of information with other readers using wiki technology. While other projects have coupled the dual worlds of atoms and bits, few have combined the idea of collaborative and distributed authoring with tangible interfaces. WikiTUI not only bridges the real and digital worlds, but facilitates multiple contributions to a reference base spanning across these worlds. The goal of our work is to apply the paradigm of coupling atoms and bits in order to extend the collaborative annotation and authoring possibilities of wiki media.