No Creative Pact post for yesterday, and today I have nothing creative to show. But I will link you all to Michael Tyson’s blog post about audio timing in iOS and some approaches he has made to improve on NSTimers (the subject of one of my previous posts). Just so you know, Michael created the iOS app Loopy and you can follow him on Twitter here.
Today I didn’t do too much as I arranged with Monty an extension to my PhD submission deadline, necessary as a number of things (which are not topics of this blog) have come together to make the target of the end of this month untenable. I am currently in London hunting for new accommodation.
Creatively what I did do was implement direct track access to track two of Futures EP, and made inroads into handling app interruptions – I can only properly test these on an iPhone so it stays hanging. I also completed a few of my unfinished references in my thesis commentary. I am meeting Monty and PA next week to discuss this document in depth…
Today I’m going to continue documenting my programming. Before I start on this just to mention I also worked on the mix of Futures, the second track of Futures EP. The order of the day was vocals, and I wasn’t quite sure what I was going for mix-wise but the results are a good starting point.
It’s programming I want to talk about though, and in particular I want to continue singing the praises of the AVPlayer API, which I rewrote the audio engine of my app with yesterday. Today I sorted out some other details, starting with the volume slider.
Originally I had subclassed UISlider to make my own custom-design slider for volume, but AVPlayer’s volume can’t be used directly with a UISlider. Instead AVPlayer uses the system volume control, accessed through instances of the MPVolumeView class. With this in mind I decided to subclass MPVolumeView using the same graphics as my previous UISlider subclass, and the benefit is that the control’s link is opaque: no need to connect through IBAction methods in the view controller.
Next was to configure the audio session, which governs how my app plays with other apps and the main OS when it comes to audio. My app uses AVAudioSessionCategoryPlayback, which allows it to silence other playing apps. I had to remember to enable background audio too, done by adding a UIBackgroundModes key to the app’s info.plist. Handling interruptions from other apps - in particular if an iPhone receiving a phone call – was part of this process.
Finally, the biggest upgrade came with the replacement of my tottering NSTimer system for updating the UI. This I implemented back when the app played audio using AUGraphs and was annoyingly complicated: a timer, running every fifth of a second, triggered updates to timing labels, referencing itself from the start of playback.
This approach is complicated because I decided to implement the timer on a separate thread, using NSThread and other methods to handle calls between threads. Also, NSTimers are really unhelpful: you can’t pause them, you have to invalidate them and create new ones; also they just trigger, leaving the programmer to mark time using NSDate instances. Figuring out pausing with my app was a nightmare with this approach.
When implementing the audio session I came across an interesting looking AVPlayer method - addPeriodicTimeObserverForInterval:queue:usingBlock:. This method runs a block of code to a programmer-specified interval when the AVPlayer is playing. I decided to take a look, and to my surprise found a system which could completely replace my NSTimers. When I finished the switch I had cut 200 lines of complicated, cross-thread code from my view controller.
To make this work I had to run messages back from my media instance (a singleton, setup using Cocoa with Love’s excellent singleton macro) to my view controller. Normally this would be a headache, but not with Cocoa. This is because Cocoa allows for Key Value Observation: basically you register an object as an observer of an instance variable; when a specific action is invoked on that variable – say it is changed – a specific method is called. And observation may take place across objects and requires no polling (at least by the programmer).
So I have set my view controller to observe an instance variable storing the AVPlayer’s current playing time, which is updated through the media singleton AVPlayer’s addPeriodicTimeObserverForInterval:queue:usingBlock: method every half second, which thus triggers the update of the user interface through KVO, with a call to the AVPlayer’s actionAtItemEnd notification to handle resetting at the end of the tune.
Brilliant. So brilliant in fact that I’m gobsmacked about the AVFoundation framework, it truly is a fantastic API for handling media on iOS.
Here is a shot of Futures EP running on my iPad. Today I started a new Objective C class encapsulating an AVPlayer-based playback engine, timed using an AVMutableComposition item. In the morning I sorted out both backing tracks, and this evening I tackled the open vocal parts of the first track, Tapes. It has all come together very easily, including the reloading of open parts when the EP is stopped. When running on the device the app actively uses around 1MB of memory.
Next on the Futures EP to do list is to get the volume slider working (AVPlayer uses the system sound volume so I probably need to subclass my snazzy volume control from the MPVolumeView class instead of UISlider), figure out individual track access through the track labels on the UI, and, of course, finishing the open parts for Futures.
On my way to finishing my PhD today I started with editing speech for my track Futures – with that completed all editing should be done and I can move on to mixing. In the afternoon I worked a bit on the thesis commentary, finding a couple of dangling references (one regarding the digital radio switchover, the other about neural networks for music). But it has been the evening that has provided the greatest satisfaction, and to explain why I’m going to explain Futures EP a bit more.
Futures EP is what I call an ‘open outcome record’, a sonic medium that resembles a traditional recorded release in every way but for its ability to play tracks back differently on each repeat. This is done by having a partial backing track for each tune, and overlaying the remaining parts on top at run-time. This requires it to play back from a device that can accommodate this process, and I have chosen iOS as the platform here.
Having experimented with the various audio APIs in iOS I found that the higher level ones – AVAudioPlayer and Audio Queue Services – didn’t provide the precise timing synchronisation that I needed (the only way of scheduling audio was through sloppy NSTimers); it was necessary to use a lower level Audio Units implementation for this reason, and I have managed to get this working pretty well.
The problem with this approach is that my audio files, which are stored in the app in compressed formats – MP3, AAC or CAF – are converted into a 32-bit sample type when loaded into the audio buffers. All of a sudden the 10.2MB backing track for the first tune, Tapes, is expanded to nearly 80MB in memory, and very soon the device’s memory is filled (it is vital to test for this on a device as the simulator doesn’t replicate this limit sufficiently).
I considered two solutions: either load all my audio in chunks into many small buffers and rotate which one plays (this is like how Audio Queue Services works with ‘enqueuing’), or find a way of loading the compressed files into memory and convert them on the fly. Either solution would be complicated to implement, especially the first. So I went online to figure out how to proceed.
On my travels I came across Cocoa with Love’s History of iOS Media APIs and noticed, under the iOS 4 section, a framework I hadn’t seen before: AVComposition, part of the AVFoundation framework. This API allows media assets to be organised in time using AVCompositionTrack objects and played back through the AVPlayer (it’s very likely this framework is used in Apple’s iOS versions of iMovie and GarageBand).
So this evening I set up a test app to try and get audio playing and synchronised using this new approach. And it worked! I tried it in the simulator and on the iPad, and in both the overlaid part was nicely synched with the backing, and with no need for AUGraphs and render callbacks, and with none of the memory issues.
As a comparison my Audio Units app easily filled the 128MB memory limit of my iPad with just Tapes (it crashes when I try to load the backing track of Futures as well :-/). The AVPlayer test version I put together this evening (albeit without the fancy UI of the AU one) actively used only 0.75MB of memory when running. Now that’s a solution. So this weekend I’m going to be rewriting the audio engine for Futures EP to make use of this new approach. :-)
Ok, I’ll admit right now that I probably won’t be doing a full pact this year, and my theme is not based on a particular piece of software like it was last year. This is because I hand my PhD in at the end of this month so I’m rather busy with that! But I also thought that my pact this year could basically be about my progress as I complete the PhD off. So here we go…
Today I was in the main studio at uni, editing vocals recorded several weeks ago for my last tune for the PhD, Futures (an earlier mix of the tune, sans vocals, can be heard at my Soundcloud). This tune is the title track for Futures EP, which is a first-of-its-kind release: designed to run on Apple’s iOS platform, the two tunes of Futures EP will play back slightly different every time they are played, but in all other respects will act like a normal ‘fixed’ record. I hope to have Futures EP finished and on the app store soon.
Also today I went through the corrections I made to my thesis commentary yesterday (doing a PhD in Music Composition I hand in works and a commentary of 20,000 words explaining them). It feels like a decent document to me (here’s a look at the cover page) but I await the comments of Monty and PA, my supervisors.
To round off today I have made a screencast of one of pieces, I am…, for documentation purposes. This will be edited together tonight and added to the documentation repository, to be handed in with the project.
So there you go, first day of CreativePact, 29 days to go until hand in…