The Beginning
- ostecinasound
- Jun 3
- 13 min read
A great majority of this project has been me, sitting in the SPACE office, teaching myself new things and playing with the equipment I've been given access to. I have promised a literature review to the URO folks, but for my own thought organization, I wanted a place to be able to talk about where I'm at, aimed directly at the void, that could be potentially viewable to other people at the end of the process. Imagine you're walking around a mad scientist's lab in a video game and reading all of the strewn about research logs. That's this.
So, what have we learned so far?
Aside from that it is an extremely difficult task to get people to answer their emails (I already knew this, and yet.)
Someone (anastasiadevana on github) has created a plugin that works in Unity that is able to read the data from the Airpods Pro and use it to track the user's head position (only 3DoF though, not the coveted 6DoF or 9DoF). This is the basis for my experimentation, and has lead me down the path of using Unity. It has also opened up a lot of questions, like:
How does the iPhone interact with this plugin? If the phone moves but the head does not, what happens?
Is there any way to get this to combine with the iPhone's accelerometer to give more data and open up to 6DoF?
How does this interact with Unity's AR features?
After getting access to a pair of Airpods Pro (thank you Finlay!!!) I am working my way backwards to try and recreate the gif above on my own computer. My personal iPad, although it doesn't have the TrueDepth camera required to create a custom HRTF (Head-Related Transfer Function), can connect to the Airpods and do Head Tracking. My next quest is figuring out how to use Xcode to be able to use the Airpods within Unity and play the sample project that comes with Anastasia's plugin.
So far, I have made my own Apple Developer account and gotten (I think) everything I need installed. Now to learn how to use it, following these handy tutorials on YouTube. While I wait for my beautiful red iPhone 14 to arrive in the mail, I will be attempting to use my beautiful rose gold iPad to connect to the Airpods and send that data over to Unity.
The Quest To Run The Sample Scene
The HeadphoneMotion plugin has a sample scene (it's the cube!) that you can use to test the plugin on your own device. In order to get that up and running, I need to do the following things:
Gather the software and hardware
Unity3D (tested with version 2019.4.1f1)
Xcode version 12 or higher
AirPods Pro running the latest firmware (Mine are 6F21 with a case running 1.4.1)
Apple mobile device running iOS 14 or greater (Until my iPhone arrives, I'll be using my iPad 7th gen, which is running iOS 16.3.1)
Basic knowledge of how to sign and install iOS builds (I'm following along with this tutorial)
Get the scene up and running in Unity
Get the Unity scene into Xcode as a build
Connect the hardware (in this case the iPad) to Xcode
Install and run the Unity scene on the hardware
Put the Airpods in and test the scene
Let's Try It!
My first roadblock was getting Unity to install the iOS module, because the Mac that I'm working on, despite being given an admin account, REALLY does not like installing Unity modules. Begin the troubleshooting. After trying unsuccessfully to convince Apple that I was not installing a virus on my work computer and was, in fact, installing the program I need to work, I uninstalled the Unity Hub and Unity Editor that were already sitting on this computer and tried reinstalling them all. That actually did seem to work.
I got the HeadphoneMotion Sample Scene imported into a Unity project, moved the platform for the build profile over to iOS, and then exported a build for iOS to move into Xcode. I booted my iPad into developer mode and connected it to the Mac and sent the sample scene off to Build on the iPad.
My first try, all I saw was the default Unity scene and a bunch of errors I didn't understand. But I realized after rereading the instructions that I hadn't actually sent the HeadphoneMotion scene and instead sent Unity's default scene.
On my next attempt, I got most of the UI to load up correctly, but the cube didn't show up and I seemed to be getting a bunch of errors from Xcode. I did some troubleshooting with ChatGPT (which gave me some code to give more detailed debug messages, confirming that I was getting input data, the cube just wasn't appearing). The next week, I came back and tried simply changing the material that the cube was made out of, and that worked! The cube is matching the movement of my head as if the camera is behind me, looking at the back of my head, which is why it sort of looks backwards. But once I start connecting that rotation to the camera, it will make more sense. But hey, look! I'm recreating the gif!
sidenote: add "Making Gifs that are less than 25MB" to the list of "things I can (somewhat poorly) do now"
(here it is as a video)
Now that I'm able to run the sample scene, the next step is gaining a full understanding of what the data that the plugin grabs can be used for. My next experiment will be connecting this motion to a camera and listener in Unity, so that as you move your head, the sounds that you are hearing in the environment are changed based on where they "exist" in the space.
Anastasia herself has applied this idea in combination with an AR app for the University of Oregon that lets you explore around different buildings at the UO Knight Campus with both your phone camera for visuals and the airpods for audio in some ambisonic spaces.
I couldn't do this alone. Here are some of the folks I've talked to and what they've shown me.
I came across this project looking for redditors who were also experimenting with the Airpods Pro (the original post was someone trying to make a haunted house immersive experience), and Xavier had linked this project, which runs as an Android APK and was made in Unreal. It manages to bypass the need for Airpods with a very clever low-tech solution: just tape your phone to the top of your head (or use hair ties to secure it to your headphones, as I did). It's an audio poem, using data from the accelerometer in the phone (how it knows when you've rotated it from portrait to landscape, which way you're facing on maps, that sort of data) to track where's your head at at at at....
Here's a video of me experiencing the whole thing:
Xavier pointed me in the direction of FMOD as a way to work with programmable audio in a way that would be familiar to me as someone who has spent a lot more time in DAWs than in video game engines, and also pointed me towards some other interesting projects to explore, like echoes.xyz (who I have reached out to, but haven't yet heard back from)
I got in contact with Evan after seeing his YouTube video about the Airpods Head Tracking, and he seemed to have a good sense of how to actually develop with this. Talking with him lead me into a few different discord servers that I will be dutifully lurking in for the next few weeks, AudioKit (which is an open source repository for making sound things for iOS) and iOS Audio Developers. AudioKit itself is also definitely going to be an avenue of exploration, because it seems like they have a lot of tools that people are putting together to expand the potential usage of the Airpods AND their accompanying iPhone. We also had a nice chat about sound and podcasting and his part time job at Raising Cane's.
I found Ludwig on the comment section for the HeadphonesMotion plugin github. He was asking about the 204ms delay between actually moving your head and hearing the change reflected. I emailed him asking if he ended up using that plugin for his work and he reported back that he had ended up creating his own firmware that is, oddly enough, more similar to the "tape-your-phone-to-your-headphones" strategy that Xavier uses. The firmware is open source, but the Unity app that goes with it unfortunately is not, because it uses an API from Blue Ripple Sound that is proprietary.
I really appreciate the fact that Ludwig got back to me within the day, links and all, and I think that his projects are very neat! I do think, though, that for my purposes, that delay is something I'll just have to live with as one of the natural consequences of working with consumer-adopted equipment. The perils of Bluetooth continue to follow me beyond my investigations into Silent Discos in my Live Audio class last year.
Headitude: An Avenue For The Studio
Headitude is a little Swift software for Mac that also serves the function of tracking the airpods. However, this one doesn't require you to connect to an iPhone for its purposes, and it actually has a bit of a friendlier UI. You can do some finagling to allow it to track your head inside your DAW of choice, which will be helpful when I'm developing the actual sound elements of this project and want to test things out right in my DAW. I followed this tutorial to get it up and running on my Mac.
Some Psychology Concepts
Another aspect of this project, and what is personally exciting to me as a way to combine my fields of interest, is the potential therapeutic benefit of what I'm building. There are a lot of anecdotal reports of people using environmental sounds, music, and even specific tones to help improve their mood, their focus, or their overall mental health. I have a whole Spotify playlist that I use to help me lock in when I'm working (an eclectic mix of classical, video game OSTs, vaporwave, and jazz). There are a few areas that I've sought to explore in my research as potential evidence-based practices to improve the quality and effectiveness of what actually goes into these tracks.
Binaural Beats (ABS)
Binaural beats is one of those things where there is a decent amount of high quality research on it, but because the internet is the internet and this is a very easy thing to experiment with on your own, there are a lot of people spouting pseudoscience nonsense. The Frequency Following Response has been observed since the 70s, and the essence of it is that when you have a sinusoidal tone (a single tone, represented by a sine wave) at a certain frequency playing in one ear, and then another tone playing at a different frequency in the other ear, you brain will interpret a third tone that is the difference in Hz between those two tones. Mind you, this only works with headphones, where one ear gets exclusively one tone. If you do this with speakers, you'll still hear that sort of wavy-feeling middle tone, but you're hearing the actual interaction of the sound waves in space as opposed to Binaural Beats where your brain is creating that interaction.
This interacts with the idea of "Entrainment", where you can get your brainwaves to sync up with something from your environment. Different brainwave frequencies are indicative of different states of consciousness, and what I'm really going for is Alpha waves, which are at 8-12Hz (so, you'd want tones that are 8-12 Hz apart if you want to "entrain" these). Alpha waves, consciousness-wise, are a sort of relaxed-but-alert state, something that would be helpful for a mindful, present walk.
Something that I am particularly interested in exploring is the potential effects that come from having a spatially varied soundscape, but keeping a constant binaural beat frequency stimulation the whole time. The scene shifts around you as you move, but you are grounded by the constant stimulation of the beats, directly and consistently in the base of the left and right ear, without effects. I wonder if this would prove helpful in enhancing the experience, or instead be distracting. Something to try out, for sure.
Music Therapy and the Iso Principle
Following the Iso Principle involves setting up a playlist that starts off matching your mood in terms of tempo, intensity, and emotional quality of the music, and then slowly transitioning the music into something that matches a more desired mood. Like using Boil The Frog to make a playlist that starts at Mitski and ends at Vampire Weekend in order to lift up your mood. It seems pretty standard as a practice these days, but the concept was first introduced in 1948 by Dr. Ira M Altshuler (1894-1968) as a treatment for psychiatric patients. According to William Davis of CSU, Altshuler apparently was inspired by seeing his daughter dance to Waltz of the Flowers from The Nutcracker, which is a tidbit I especially love because I have a special interest in The Nutcracker. His strategy was to maintain the listener's attention and engage them musically, and then once they were attuned to the music, he would coax their mood into a more positive place by making gradual changes. His "Levels of Attack" are as follows:
Rhythm - Perception of rhythmic beat is a very core function of the brain, and doesn't require the cerebral cortex, even those who are very mentally ill or have limited brain function are likely to have some perception of rhythm, so it can be used to grasp or maintain attention
Melody - A familiar or predictable melody (a "good gestalt") would also help to hold attention
Harmony - A level above rhythm and melody, adding complexity and making the listener more self-aware
Mood modifying music - Once the listener was settled into a groove, changes would begin to move the rhythm, melody, and harmony from evoking one emotion to a different (usually more positive) one
Pictorial associative music - When the listener had achieved the desired emotion, the music would change to something familiar or evocative to help the listener recall a memory in a certain way or to create associations outside of the music
He also employed a "Finality Principle", playing relaxing music at the end of a session as a sort of cool-down.
I personally recognize a lot of these concepts from the history and conventions of Ambient Music, and it's interesting to note how these ideas that are so commonplace for self-regulation nowadays were once fairly revolutionary psychiatry practices. That's why I think projects like this are important, because although they certainly can't replace medical intervention or talk therapy with other human beings, giving more people access to tools to gain more control over their emotions can create a lot of positive change!
Concepts From The Worlds of Composition and Audio Enthusiasts
While you can take the academic route and try to find tangible, brain or behaviour-based effects of sound, there are also plenty of folks who have dedicated a lot of time to an artistic pursuit, ascribing meaning to their work because of the kinds of physical and emotional experiences that they have the capability to create within a listener. These have their own merit and are also worth exploring, even if they don't have the same sort of quantitative data to support direct therapeutic benefit.
Deep Listening
Pauline Oliveros (1932-2016), a woman after my own heart, recorded an album in 1989 called Deep Listening in the Dan Harpole Cistern. The retired 2 million gallon cistern was 14 feet deep and 200 feet in diameter, which gave it some really interesting acoustic qualities (like a 45-second reverberation time!) and made it a great place for experimental music performances. Jonas Braasch has since developed software tools to simulate the space in electronic music production, called Cistern Simulation Technology. Deep Listening Band released an assortment of recordings, focusing largely on utilizing especially reverberant and resonant spaces.
Aside from her composing, Pauline also wrote often about the relationships between sound, perception, and consciousness. She felt that there was a very distinct difference between "hearing" and "listening".
"To hear is the physical means that enables perception. To listen is to give attention to what is perceived both acoustically and psychologically." "Sound pressure patterns assist hearing but cultural history and experience influences listening." from pg 13-14 of Deep Listening: A Composer's Sound Practice, 2005.
Master of Silence: John Cage
John Cage's 4'33" gets memed on a lot, but as someone who has had the empowering and humiliating responsibility of stopping all activity on set to record room tone, I think it should definitely be taken seriously, and I also feel it has a lot to do with this project. In fact, it's a pretty good measure of success for my program. Someone who considers this piece to be a joke, or even a source of discomfort, when they start the program should be able to peacefully sit through it (and maybe even actually enjoy it!) by the end.
In Silence, Cage's book of lectures, he associates a lot of his work with the practices of Zen, and emphasizes the importance of finding joy and understanding in sources of sound outside of what is typically considered music. He also makes some eerily accurate predictions about the course that electronic music has taken as technology improves. A lot of his writing is beyond the scope of this project, which is mostly focused on making small habit changes to the average consumer, but I personally find them really fascinating and they definitely have an influence over my work and have inspired the core idea of the project: create a safe environment to become curious about, and comfortable with, a lack of control over your soundscape.
Sound Walks
There are 2 Scha(e)f(f)ers that are very important to the sound world. Pierre Schaeffer of Musique Concrète (French, big into electronics and radio) and R. Murray Schafer of the World Soundscape Project (Canadian, a lot more interested in phenomena and acoustics of nature). I had to write a compare and contrast paper on the two of them in second year and it was a DOOZY trying to remember who was who. But R Murray Schafer's work is considered the origin of sound walks.
Composer Ellen Reid has an app that called, appropriately, Ellen Reid Soundwalk, that has an assortment of site-specific soundwalks and use GPS on your phone to change the music as you walk around. Unfortunately, the closest one is in New York. I've sent Ellen's team an email to potentially get some insight into their process.
3D Audio
Justin Paterson and Hyunkook Lee's 2021 book 3D Audio: Perspectives on Music Production has provided me with a pretty invaluable overview of the latest and greatest (or, at least what was late and great 4 years ago) in 3D Audio (which covers a lot of ground, including broadcasting, live sound, AR, Ambisonics, Binaural audio, and much more). Each of these disciplines has their own motivations for innovating, whether it's providing the optimal audience experience to as many people as possible, deeply immersing a listener, recreating real-life audio conditions with extreme detail, or breaking the boundaries of sound design and composition. Reading it has given me a lot of perspective on the historical backdrops that have lead to the complicated software I'm learning now, and insight into what the commercial and artistic worlds are currently interested in and what they're hoping to develop for the future.
It also sparked a Binaural deep dive that lead me to "Splendor and Misery of Kunstkopf Stereo", which is a wonderful little German radio play where you assume the perspective of a dusty old "Kunstkopf" (translated literally to "dummy head", the granddaddy of those 3Dio microphones you see all the ASMR people tapping their acrylic nails all over)
Definitely worth a listen.
So, I've been here a month now, and this is what I have to show for it. Stay tuned for further updates, more resources, and more videos of me rotating my head while wearing silly t shirts.
Comments