Snob Bog

January 15th, 2022 by

Unity + AR + Machine Learning iOS App

Description

Street Live is an interdisciplinary team of six graduate students at the CMU Entertainment Technology Center. Street Live’s project semester seeks to expand Snob Bog’s current immersive theater application.

The Snob Bog app is a Augmented Reality mobile app that reveals animated experiences that takes place in public spaces triggered by neighbourhood objects like fire hydrants, stop signs, etc.

Our plan is to improve upon the existing application by boosting performance, providing an updated interface, and expanding the character options for the user. Using a mobile device, the goal is to have users both explore their neighborhoods and the character’s storylines. We seek to further explore the realm of AR through the existing window that our client created, into this virtual neighborhood to get a glimpse into the lives of these characters. Our goal for the semester is to provide an updated framework, a potential user interface for the future, as well as a proof of concept customizable character option.

Technology

The Snob Bog Apps uses both machine learning and AR. After running object recognition using barracuda + yolo-v3-tiny’s pretrained model in Unity, AR Foundation finds solid object points within the object’s range. After AR detection is complete, an animtaion will be triggered accordingly.

Character Customization (Ongoing Progress)

Our clients wants the users being able to mix and match characters they want to see in the virtual world or even be able to add their own body parts to the app someday in the future. We’re building this mix and match system using our own character body parts. We’re also developing a file loading system that the app will load the right animation file according to the user’s choice.

Experience Design

One of the problems we’re facing is how to teach the users to use this app. We don’t want the text guidance interfere with the immersive experience, so we’re trying to develope a subtle way of teaching the users which objects can be triggers.

Right now we’re trying to generate virtual 3D objects as a hint. These hints are generated after 1 minite the user not being able to trigger anything. We hope this design will minimize the frustration of learning.