Building an AR Piano Trainer

This Unity-based Augmented Reality (AR) application helps users learn and play songs by visualizing notes in real-time.

Project Overview

This was my first time working with AR and Unity so there was a lot to learn. I wanted to create an app that would help people learn to play the piano by visualizing notes in AR space, kinda like those piano tutorial videos on youtube. I have a background in music so I thought this would be a fun project to work on. The AR aspect was really fun and I can see a lot of potential for this kind of technology in the future. I want continue working with it and see what other cool things I can make.

The AR Piano Trainer leverages AR Foundation to create an immersive piano learning experience with the following features:

  • MIDI File Integration: Extracts song notes and timing.
  • AR Note Visualization: Displays notes as AR planes that move towards the player.
  • Dynamic Note Timing: Ensures notes reach the correct position at the right time.
  • Piano Note Mapping: Maps MIDI notes to their respective AR positions.

How It Works

  • Load a MIDI File: The app reads the MIDI file and extracts note data. I used Dry Wet MIDI which is a unity package for reading MIDI files. It has a lot of cool features for music theory things, but I only used the MIDI file reading part.
  • Map Notes to AR Space: Each note’s position is calculated based on a virtual piano mapping using an AR anchor as the relative positioning.
  • Spawn AR Note Planes: Notes appear and move towards the target area.
  • Sync with Song Timing: Notes arrive at the correct time for the player to hit them.

Challenges

One of the main challenges was handling the syncing between the note timing and their location in the AR space. Unity's scaling and positioning system was tricky to get right because the position and scaling system coordinates are at 1:10 ratio. This made calculations for note positions and timings more complex.

Another challenge was getting the anchor point correctly set. This was crucial for the AR notes to move towards the player accurately. I had to experiment with different ways to correctly handle this. Eventually, I found a solution that worked well but I'm not too happy with it since it requires a lot of player involvement to setup. The anchor also occasionally drifts or rotates, which can be frustrating for the user but I think I solved the rotation issue.

Conclusion

This project was really fun to work on and I learned a lot about AR and Unity. I'm excited to see where this technology goes in the future and I'm looking forward to working on more projects like this. As always the best way to learn is to just start building things and see where it takes you and I'm glad I did this project.

Check out the source code on my GitHub repository.

Posted by: Aidan Vidal

Posted on: March 8, 2025