Special Topics Module 2

Research & Activity Documentation

Jesse Vieira

Project 2

Module 2

Project 2 was centred around finding objects that Olivia and I could use to develop the Mirror Display and developing gesture controls using Touch Designer and its plugin Media Pipe to switch through fitness videos.

Activity 1

For Activity 1, I experimented with with Media Pipe and it’s Gesture controls. I wanted to learn as much as possible about the main features that we will be using of the plugin.

We were told by our friend Bre about the motion tracking plugin in Touch Designer that we could use of our project. She sent us a set of videos that would help us learn the ins and outs of the plugin.

<br><br>

I conducted research on the different videos and replicated the creations he made. I did this so I could learn the use cases of the plugin and how to develop the interactions.

Here I was just messing around with the different pieces of data and seeing what changed when I would move my hand. I wanted to get an understanding of what all the different data points represent.

While experimenting with Media Pipe’s gesture controls, I was trying to understand how to isolate and work with specific data points.

<br><br>

To do this I followed along with the Media Pipe creator’s tutorial video. In it he was explaining how to make a logic system to isolate the data and what the possible uses are for it.

Activity 2

While Olivia was finding different materials that could be used for the mirror film, for my second activity, I wanted to take my Media Pipe learning up a step by learning more advanced use cases and following the plugin creator's tutorials.

I followed along with Torin Blankensmith’s videos so I could learn how to map the gesture controls. While I was mapping the data points, I was also learning how to create visuals that follow and respond to gestures I created.

<br><br>

Here I was testing how text would respond with gestures and if it would follow my recorded finger’s data.

I then started mapping some place holder videos I had on hand to text. I was following along with the tutorial videos to get an understanding of how we could use gesture controls to switch through different videos.

<br><br>

Here I was swapping through the gesture controls to alter the volume and change the videos/text.

Here I was testing the gesture controls together to see if both could work. I also had my previous tests working so I could see how the scale and volume could work together, as well as seeing what the rotation range would be for the videos to properly swap.

Action Research Cycle + Testing 1, 2, & 3

My Research cycle and testing involved me creating different visuals and pieces of text that react based on the data points tied to two of my finger tips on each hand. With this, I was able to learn how to make the data points from the plugin affect and manipulate different operators. In doing that I also learned more of Touch Designer's operators which could be helpful in future projects.

Here I was following along with a video to create points that correspond with my index finger tip and my thumb tip. I also learned how to link two points together with a line, which could be useful for measuring space or linking points together.

<br><br>

Reflection: This research helped me understand more about Touch Designer operators and how they could be used alongside the gesture tracking. This will be helpful for the project going forward as we will continue using Touch Designer to create a gesture controlled interactive fitness mirror.

Following the tutorial video, I learned how to tie gesture controls to switches. The switches would react based on the distance of the data points on one hand. On the other hand it reacts based on the rotation angle. The rotation angle would change the colour and add text to the line.

<br><br>

Reflection: This research helped me get a better understanding of how gesture controls can influence creations and values in touch designer. Going forward, this research will be used to create a way for users to switch through different fitness videos without having to use touch screen.

Here I continued my learning from Torin Blankensmith’s videos and was able to create a dynamic gesture switch for multiple videos.

<br><br>

Along with that, I also learned a cool trick in which I could map the videos to correlating text which had added effects.

<br><br>

Reflection: This research was helpful as it taught me how gesture controls can influence videos and audio. With this research in mind, I can now move on to making specific gestures and mapping them to their own videos.

Project Management

To help keep the team organized and ensure tasks were completed on time, Olivia created a shared to-do list with specific deadlines. The list was built in Google Docs and uses checkboxes so tasks can be marked off once they are finished. It is also organized by team member, which makes it clear who is responsible for each task.

We communicate through Microsoft Teams, which allows us to easily update one another on progress, ask questions, and stay aligned throughout the project. In the event that we would need to contact each other quickly, we decided to use instagram for an alternative form of communication.

Project 2

Prototype Research and Development

For our project we had two in progress prototypes as we were still working on identifying the materials we would need and how we would develop the Mirror. Olivia's prototype is a mirror film material with plexiglass that has been carefully attached to a tablet screen. While my prototype is a ideal function of how our users would switch between videos after following along with them during their workouts.

This is a prototype of what we envision our ideal mirror with an integrated display to look like. We used a plexiglass sheet from a picture frame and applied mirror film over its surface to simulate a two-way mirror. This setup allows the prototype to maintain a clear mirror reflection while also enabling the monitor display behind it to remain visible through the surface. The result demonstrates how the mirror can function both as a reflective surface and as a screen for digital content.

This is a prototype of our ideal function with the gesture controlled mirror. Here we can see that there are two fitness videos that are being switched through with my hand gestures.

<br><br>

Ideally we would have a video of a semi transparent figure performing the different exercises for the user to follow. Along with this, we would want there to be more intuitive gesture controls to switch through videos and adding more to pause or play the videos.