Apple’s Core Audio API is very powerful. It is also not easy. With Great Power comes Great Heartburn when the documentation is out of date and/or scattered. Sometimes the needs of the few outweigh the needs of the many as in the case of conference talks. At WWDC 2011, session 411 was named “Music in iOS and Lion” (see the Resources). If you watch the presentation, you wil get a taste of what is possible. Unfortunately they were as stingy with code examples as Webern was with notes. (The goal of conference presentations is to show off, and not to educate the viewers. Not picking on these guys, but this has been true of every presentation I’ve ever seen).
So, here is a first step. It is a complete project that runs and produces sound. Many things are missing – on purpose – which will be added in subsequent blog posts. The point is to get the main point across in the simplest manner possible.
Core Audio is a C API. Currently there are no official Objective-C bindings. So, you need to mix your Objective-C code with C code. In more advanced code you will be using many Core Foundation types too, so get comfortable with this intermixing. You will also use classes such as AUGraph from the AudioToolbox framework. So, in your project setup, (under build phases in XCode 4+) you need to link to the AudioToolbox framework and the CoreAudio framework. You do not link to the AudioUnit framework here. If you do, you will get an error.
Most CA (Core Audio) API functions return an OSStatus value. This is a 32 bit integer.
OSStatus NewAUGraph ( AUGraph *outGraph ); |
The function to set up your graph is NewAUGraph. Here is one way to handle the result.
1 2 3 4 5 6 | AUGraph outGraph; OSStatus result = noErr; result = NewAUGraph ( &outGraph ); if(result != noErr) { // handle it } |
Chris Adamson (see the Resources) wrote a function that many programmers use to display the status as a string if possible. If not he prints the digit. This is a bit like the old original Mac toaster that would pop up a dialog stating “Error -151 Occurred”. So, I’ve added a switch statement that checks the value against well known error constants.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | void CheckError(OSStatus error, const char *operation) { if (error == noErr) return; char str[20]; // see if it appears to be a 4-char-code *(UInt32 *)(str + 1) = CFSwapInt32HostToBig(error); if (isprint(str[1]) && isprint(str[2]) && isprint(str[3]) && isprint(str[4])) { str[0] = str[5] = '''; str[6] = ' '; } else { // no, format it as an integer sprintf(str, "%d", (int)error); } fprintf(stderr, "Error: %s (%s)n", operation, str); switch(error) { case kAUGraphErr_NodeNotFound: many more... |
Using this utility function, your code will look like this instead.
CheckError(NewAUGraph(&outGraph),"NewAUGraph"); |
You will find some examples on the net that mix CA code into their View Controllers. I’d rather not. So, I created a regular class named GDSoundEngine that will contain all of the CA and Audio Unit frobs. The View Controller will simply tell the SoundEngine to do the things specified in its public interface, such as playNoteOn:velocity.
For example, this is how the view controller interacts with the sound engine. The note value is from the tag set on the UIButton from the storyboard.
- (IBAction)noteOn:(UIButton *)sender { UInt32 velocity = 100; [self.soundEngine playNoteOn:[sender tag] :velocity ]; } - (IBAction)noteOff:(UIButton *)sender { [self.soundEngine playNoteOff:[sender tag] ]; } |
Here is the setup in the View Controller.
1 2 3 4 5 6 7 8 9 10 11 12 13 | @interface GDViewController () @property (strong) id soundEngine; @end @implementation GDViewController @synthesize soundEngine = _soundEngine; - (void)viewDidLoad { [super viewDidLoad]; self.soundEngine = [[GDSoundEngine alloc] init]; } etc. |
The SoundEngine has a private interface with properties for the audio guts.
1 2 3 4 5 6 7 | @interface GDSoundEngine() @property (readwrite) AUGraph processingGraph; @property (readwrite) AUNode samplerNode; @property (readwrite) AUNode ioNode; @property (readwrite) AudioUnit samplerUnit; @property (readwrite) AudioUnit ioUnit; @end |
Notice that for each AUNode there is a corresponding AudioUnit. We will step through the details for each of these now.
Build an Audio Processing Graph
Here are the steps to build an Audio Unit Graph.
- Instantiate an
AUGraphwith the functionNewAUGraph. - Instantiate one or more
AUNodes, each of which represents one audio unit in the graph. To create and add the nodes to the graph, use the functionAUGraphAddNode.
You first create anAudioComponentDescriptionto pass into this function. - Open the graph and instantiate the audio units with the function
AUGraphOpen. - Obtain references to the audio units with the function
AUGraphNodeInfo.
Step 1. The AUGraph is a property named processingGraph. In the implementation I use the underscore pattern for the instance variable. Then in the graph init method, the graph is created.
1 2 3 4 5 6 7 | @synthesize processingGraph = _processingGraph; ... - (BOOL) createAUGraph { CheckError(NewAUGraph(&_processingGraph), "NewAUGraph"); ... |
Here is an example of step 2. The AudioComponentDescription is set up first. The componentSubType field here is set to kAudioUnitSubType_Sampler. This AU was added in iOS 5. Then the AUNode is instantiated and added to the graph with AUGraphAddNode. Note that _samplerNode is the property’s instance variable.
1 2 3 4 5 6 7 8 9 | // create the sampler // for now, just have it play the default sine tone AudioComponentDescription cd; cd.componentType = kAudioUnitType_MusicDevice; cd.componentSubType = kAudioUnitSubType_Sampler; cd.componentManufacturer = kAudioUnitManufacturer_Apple; cd.componentFlags = 0; cd.componentFlagsMask = 0; CheckError(AUGraphAddNode(self.processingGraph, &cd, &_samplerNode), "AUGraphAddNode"); |
The RemoteIO AU is created in the same manner.
1 2 3 4 5 6 7 8 | // I/O unit AudioComponentDescription iOUnitDescription; iOUnitDescription.componentType = kAudioUnitType_Output; iOUnitDescription.componentSubType = kAudioUnitSubType_RemoteIO; iOUnitDescription.componentManufacturer = kAudioUnitManufacturer_Apple; iOUnitDescription.componentFlags = 0; iOUnitDescription.componentFlagsMask = 0; CheckError(AUGraphAddNode(self.processingGraph, &iOUnitDescription, &_ioNode), "AUGraphAddNode"); |
Now in step 3 we can instantiate the Audio Units from the AUNodes. The AUGraph needs to be open via AUGraphOpen for this to work.
1 2 3 4 5 6 7 8 | // The graph needs to be open before you call AUGraphNodeInfo CheckError(AUGraphOpen(self.processingGraph), "AUGraphOpen"); CheckError(AUGraphNodeInfo(self.processingGraph, self.samplerNode, NULL, &_samplerUnit), "AUGraphNodeInfo"); CheckError(AUGraphNodeInfo(self.processingGraph, self.ioNode, NULL, &_ioUnit), "AUGraphNodeInfo"); |
In step 4, we wire the Audio Units together with AUGraphConnectNodeInput.
1 2 3 4 5 6 7 |
AudioUnitElement ioUnitOutputElement = 0;
AudioUnitElement samplerOutputElement = 0;
CheckError(AUGraphConnectNodeInput(self.processingGraph,
self.samplerNode, samplerOutputElement, // srcnode, inSourceOutputNumber
self.ioNode, ioUnitOutputElement), // destnode, inDestInputNumber
"AUGraphConnectNodeInput"); |
For debugging there is a function named CAShow that will display the current state of audio objects.
1 | CAShow(self.processingGraph); |
The completed graph setup method
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | - (BOOL) createAUGraph { NSLog(@"Creating the graph"); CheckError(NewAUGraph(&_processingGraph), "NewAUGraph"); // create the sampler // for now, just have it play the default sine tone AudioComponentDescription cd; cd.componentType = kAudioUnitType_MusicDevice; cd.componentSubType = kAudioUnitSubType_Sampler; cd.componentManufacturer = kAudioUnitManufacturer_Apple; cd.componentFlags = 0; cd.componentFlagsMask = 0; CheckError(AUGraphAddNode(self.processingGraph, &cd, &_samplerNode), "AUGraphAddNode"); // I/O unit AudioComponentDescription iOUnitDescription; iOUnitDescription.componentType = kAudioUnitType_Output; iOUnitDescription.componentSubType = kAudioUnitSubType_RemoteIO; iOUnitDescription.componentManufacturer = kAudioUnitManufacturer_Apple; iOUnitDescription.componentFlags = 0; iOUnitDescription.componentFlagsMask = 0; CheckError(AUGraphAddNode(self.processingGraph, &iOUnitDescription, &_ioNode), "AUGraphAddNode"); // now do the wiring. The graph needs to be open before you call AUGraphNodeInfo CheckError(AUGraphOpen(self.processingGraph), "AUGraphOpen"); CheckError(AUGraphNodeInfo(self.processingGraph, self.samplerNode, NULL, &_samplerUnit), "AUGraphNodeInfo"); CheckError(AUGraphNodeInfo(self.processingGraph, self.ioNode, NULL, &_ioUnit), "AUGraphNodeInfo"); AudioUnitElement ioUnitOutputElement = 0; AudioUnitElement samplerOutputElement = 0; CheckError(AUGraphConnectNodeInput(self.processingGraph, self.samplerNode, samplerOutputElement, // srcnode, inSourceOutputNumber self.ioNode, ioUnitOutputElement), // destnode, inDestInputNumber "AUGraphConnectNodeInput"); NSLog (@"AUGraph is configured"); CAShow(self.processingGraph); return YES; } |
Starting the graph
Once the graph has been setup, you are ready to start processing by calling the function AUGraphInitialize. This calls the AudioUnitInitialize function of each Audio Unit in the graph. It also validates the graph’s connections and audio data stream formats. Finally, the graph is started with AUGraphStart.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | - (void) startGraph { if (self.processingGraph) { Boolean outIsInitialized; CheckError(AUGraphIsInitialized(self.processingGraph, &outIsInitialized), "AUGraphIsInitialized"); if(!outIsInitialized) CheckError(AUGraphInitialize(self.processingGraph), "AUGraphInitialize"); Boolean isRunning; CheckError(AUGraphIsRunning(self.processingGraph, &isRunning), "AUGraphIsRunning"); if(!isRunning) CheckError(AUGraphStart(self.processingGraph), "AUGraphStart"); self.playing = YES; } } |
Sending a note to the sampler
The final task is to define the public methods that are called by the View Controller. Here, the function MusicDeviceMIDIEvent sends a MIDI message to the sampler audio unit. For example, 0×90 is the MIDI note on message for channel 0.
1 2 3 4 5 6 7 8 9 10 11 12 | - (void)playNoteOn:(UInt32)noteNum :(UInt32)velocity { UInt32 noteCommand = 0x90; NSLog(@"playNoteOn %lu %lu cmd %lx", noteNum, velocity, noteCommand); CheckError(MusicDeviceMIDIEvent(self.samplerUnit, noteCommand, noteNum, velocity, 0), "NoteOn"); } - (void)playNoteOff:(UInt32)noteNum { UInt32 noteCommand = 0x80; CheckError(MusicDeviceMIDIEvent(self.samplerUnit, noteCommand, noteNum, 0, 0), "NoteOff"); } |
Summary
That’s it for now. The sound that you will hear is a beautiful sine wave. In the next blog post, I will show you how to set up the sample unit to use sounds from a DLS file or a SountFont2 file.
Resources
Audio Unit Hosting Guide for iOS
Audio Unit Processing Graph Services Reference
Audio Unit Properties Reference
Audio Unit Parameters Reference
Audio Unit Component Services Reference
Music in iOS and Lion WWDC 2011 presentation slides (sign in required)
Learning Core Audio: A Hands-On Guide to Audio Programming for Mac and iOS
2 Comments
Thanks for this. All I want to do is add little sampled (pitch shifted) tones to a typing game I am building. Everything I have looked at so far makes assumptions about what you know — eg the adamson book gives great info on setting up a graph and even adding in the sampler — but then I had no idea how to interact with(send midi to) it. I might be a worst case learning scenario here since I am spinning up from zero music and/or game development experience. But anyway thank you (again) for finally connecting the dots.
Charles, I’m glad that it was even a tiny help to you. Good luck with your project.