Paper Playground as a Tool for Multimodal Interactive Codesign¶

The power of Paper Playground, augmented with Creator code abstraction and Scenery-based Display, lies in its ability to combine the physicality of paper and moving real objects, with the scalable affordances of native web technologies to introduce sounds, speech, device communication, and more. Paper Playground lets us quickly prototype interactive and multimodal experiences, which is especially helpful for working with others to iterate quickly on ideas that will go on to become full web experiences (games, simulations, etc).

But why use Paper Playground to prototype a web project?¶

Multimodal Design is Hard¶

Multimodal design, incorporating visuals, sounds/sonifications, and speech (including spoken descriptions for accessible projects), is the ideal goal of any web experience. Visual design is hard. Multimodal design is harder. Multimodal codesign is even harder. Working with others, helping them to understand the relationships between the interactive objects in your project, while finding the right mapping between how someone interacts and the sounds, speech, or visuals that play/change is a constant challenge for any designer.

We also emphasize that multimodal design should be done at the start and throughout the process and not after the visual design has already completed.

Where are the multimodal design tools?¶

Tools for visual web design prototypes and interaction design exist on every corner of the web (Figma, Canva, etc.). However, there is no tool to help with multimodal design.

A goal of Paper Playground is to explore the capabilities of a tool that can help highlight the relationships between model (data/logic) components, the interactions that change them, and the ways that (visual/auditory) feedback are provided.

Get out of the screen¶

We know that collaboration in the real world is best supported when users can make use of the space around them and use their bodies to gesture and enact their ideas. Codesign practices often make use of paper or craft lo-fi prototyping at the initial stages of an idea. But, as previously emphasized, sound and speech design should begin as early as possible! So, why not make it easy to augment lo-fi prototypes with those auditory (and more!) features that are afforded easily by virtual interfaces?

Follow our research¶

Follow our research by checking out our Open Science Framework repository

Check out our 2024 paper at the Interaction Design for Children conference that explores using Paper Playground as a design probe for understanding the needs and desires for a remote multimodal co-design tool as identified by young developers!

Attributions¶

This project was developed with the Craft Tech Lab and PhET Interactive Simulations at University of Colorado, Boulder.