Core Audio in iOS 6 (CocoaConf San Jose, April 2013)

Core Audio in iOS 6
Chris Adamson • @invalidname
CocoaConf San Jose
April 20, 2013
Sides and code available on my blog:
http://www.subfurther.com/blog
Monday, April 29, 13

The Reviews Are In!

Legitimate copies!
• Amazon (paper or Kindle)
• Barnes & Noble (paper or Nook)
• Apple (iBooks)
• Direct from InformIT (paper, eBook [.epub
+ .mobi + .pdf], or Bundle)

WhatYou’ll Learn
• What Core Audio does and doesn’t do
• When to use and not use it
• What’s new in Core Audio for iOS 6

Simple things should be simple,
complex things should be possible.
–Alan Kay

–Alan Kay
AV Foundation,
Media Player

–Alan Kay
AV Foundation,
Media Player
Core Audio

Core Audio
• Low-level C framework for processing
audio
• Capture, play-out, real-time or off-line
processing
• The “complex things should be possible”
part of audio on OS X and iOS

Chris’ CA Taxonomy
• Engines: process streams of audio
• Capture, play-out, mixing, effects
processing
• Helpers: deal with formats, encodings, etc.
• File I/O, stream I/O, format conversion,
iOS “session” management

Helpers:Audio File
• Read from / write to multiple audio ﬁle
types (.aiff, .wav, .caf, .m4a, .mp3) in a
content-agnostic way
• Get metadata (data format, duration,
iTunes/ID3 info)

Helpers:Audio File
Stream
• Read audio from non-random-access
source like a network stream
• Discover encoding and encapsulation on
the ﬂy, then deliver audio packets to client
application

Helpers: Converters
• Convert buffers of audio to and from
different encodings
• One side must be in an uncompressed
format (i.e., Linear PCM)

Helpers: ExtAudioFile
• Combine file I/O and format conversion
• Read a compressed file into PCM buffers
• Write PCM buffers into a compressed file

Helpers:Audio Session
• iOS-only API to negotiate use of audio
resources with the rest of the system
• Deetermine whether your app mixes with
other apps’ audio, honors ring/silent
switch, can play in background, etc.
• Gets notiﬁed of audio interruptions
• See also AVAudioSession

Engines:Audio Units
• Low-latency (~10ms) processing of
capture/play-out audio data
• Effects, mixing, etc.
• Connect units manually or via an AUGraph
• Much more on this topic momentarily…

Engines:Audio Queue
• Convenience API for recording or play-out,
built atop audio units
• Rather than processing on-demand and on
Core Audio’s thread, your callback provides
or receives buffers of audio (at whatever
size is convenient to you)
• Higher latency, naturally
• Supports compressed formats (MP3,AAC)

Engines: Open AL
• API for 3D spatialized audio, implemented
atop audio units
• Set a source’s properties (x/y/z
coordinates, orientation, audio buffer, etc.),
OpenAL renders what it sounds like to the
listener from that location

Engines and Helpers
• Audio Units
• Audio Queue
• Open AL
• Audio File
• Audio File Stream
• Audio Converter
• ExtAudioFile
• Audio Session

Audio Units

Audio Unit
AUSomething

Types of Audio Units
• Output (which also do input)
• Generator
• Converter
• Effect
• Mixer
• Music

Pull Model
AUSomething

Pull Model
AUSomething
AudioUnitRender()

Pull Model
AUSomethingAUSomethingElse

Buses (aka, Elements)
AUSomething
AUSomethingElse
AUSomethingElse

AUGraph
AUSomething
AUSomethingElse
AUSomethingElse

Render Callbacks
AUSomething
AUSomethingElse
OSStatus converterInputRenderCallback (void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList * ioData) {
CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inRefCon;
// read from buffer
ioData->mBuffers[0].mData = player.preRenderData;
return noErr;
}

AURemoteIO
• Output unit used for play-out, capture
• A Core Audio thread repeatedly and
automatically calls AudioUnitRender()
• Must set EnableIO property to explicitly
enable capture and/or play-out
• Capture requires setting appropriate
AudioSession category

Create AURemoteIO
CheckError(NewAUGraph(&_auGraph),
! ! "couldn't create au graph");
!
CheckError(AUGraphOpen(_auGraph),
! ! "couldn't open au graph");
!
AudioComponentDescription componentDesc;
componentDesc.componentType = kAudioUnitType_Output;
componentDesc.componentSubType = kAudioUnitSubType_RemoteIO;
componentDesc.componentManufacturer =
kAudioUnitManufacturer_Apple;
!
AUNode remoteIONode;
CheckError(AUGraphAddNode(_auGraph,
! ! ! ! ! ! &componentDesc,
! ! ! ! ! ! &remoteIONode),
! ! "couldn't add remote io node");

Getting an AudioUnit
from AUNode
! CheckError(AUGraphNodeInfo(self.auGraph,
! ! ! ! ! ! ! remoteIONode,
! ! ! ! ! ! ! NULL,
! ! ! ! ! ! ! &_remoteIOUnit),
! ! ! "couldn't get remote io unit from node");

AURemoteIO Buses
AURemoteIO

AURemoteIO Buses
AURemoteIO
bus 0
to output H/W

AURemoteIO Buses
AURemoteIO
bus 0
to output H/W
bus 0
from app

AURemoteIO Buses
AURemoteIO
bus 0
to output H/W
bus 1
from input H/W
bus 0
from app

AURemoteIO Buses
AURemoteIO
bus 0
to output H/W
bus 1
from input H/W
bus 1
to app
bus 0
from app

EnableIO
! UInt32 oneFlag = 1;
! UInt32 busZero = 0;
! CheckError(AudioUnitSetProperty(self.remoteIOUnit,
! ! ! ! ! ! ! ! ! kAudioOutputUnitProperty_EnableIO,
! ! ! ! ! ! ! ! ! kAudioUnitScope_Output,
! ! ! ! ! ! ! ! ! busZero,
! ! ! ! ! ! ! ! ! &oneFlag,
! ! ! ! ! ! ! ! ! sizeof(oneFlag)),
! ! ! "couldn't enable remote io output");
! UInt32 busOne = 1;
! ! ! ! ! ! ! ! ! kAudioOutputUnitProperty_EnableIO,
! ! ! ! ! ! ! ! ! kAudioUnitScope_Input,
! ! ! ! ! ! ! ! ! busOne,
! ! ! ! ! ! ! ! ! &oneFlag,
! ! ! ! ! ! ! ! ! sizeof(oneFlag)),
! ! ! "couldn't enable remote io input");

Pass Through
AURemoteIO
bus 1
from input H/W
bus 0
to output H/W

Connect In to Out
! UInt32 busZero = 0;
! UInt32 busOne = 1;
! CheckError(AUGraphConnectNodeInput(self.auGraph,
! ! ! ! ! ! ! ! ! remoteIONode,
! ! ! ! ! ! ! ! ! busOne,
! ! ! ! ! ! ! ! ! remoteIONode,
! ! ! ! ! ! ! ! ! busZero),
! ! ! "couldn't connect remote io bus 1 to 0");

Pass-Through with Effect
bus 0
to output H/W
AURemoteIO
AUEffect
bus 1
from input H/W

Demo: Delay Effect
New in iOS 6!

Creating the AUDelay
! componentDesc.componentType = kAudioUnitType_Effect;
! componentDesc.componentSubType = kAudioUnitSubType_Delay;
! componentDesc.componentManufacturer =
kAudioUnitManufacturer_Apple;
!
! AUNode effectNode;
! CheckError(AUGraphAddNode(self.auGraph,
! ! ! ! ! ! ! &componentDesc,
! ! ! ! ! ! ! &effectNode),
! ! ! "couldn't create effect node");
! AudioUnit effectUnit;
! CheckError(AUGraphNodeInfo(self.auGraph,
! ! ! ! ! ! ! effectNode,
! ! ! ! ! ! ! NULL,
! ! ! ! ! ! ! &effectUnit),
! ! ! "couldn't get effect unit from node");

The problem with effect
units
• Audio Units available since iPhone OS 2.0
prefer int formats
• Effect units arrived with iOS 5 (arm7 era)
and only work with ﬂoat format
• Have to set the AUEffect unit’s format on
AURemoteIO

Setting formats
! AudioStreamBasicDescription effectDataFormat;
! UInt32 propSize = sizeof (effectDataFormat);
! CheckError(AudioUnitGetProperty(effectUnit,
! ! ! ! ! ! ! ! ! kAudioUnitProperty_StreamFormat,
! ! ! ! ! ! ! ! ! busZero,
! ! ! ! ! ! ! ! ! &effectDataFormat,
! ! ! ! ! ! ! ! ! &propSize),
! ! ! "couldn't read effect format");
! ! ! ! ! ! ! ! ! kAudioUnitProperty_StreamFormat,
! ! ! ! ! ! ! ! ! busOne,
! ! ! ! ! ! ! ! ! &effectDataFormat,
! ! ! ! ! ! ! ! ! propSize),
! ! ! "couldn't set bus one output format");
Then repeat AudioUnitSetProperty() for input scope / bus 0

AUNewTimePitch
• New in iOS 6!
• Allows you to change pitch independent of
time, or time independent of pitch
• How do you use it?

AUTimePitch
! AudioComponentDescription effectcd = {0};
! effectcd.componentType = kAudioUnitType_FormatConverter;
! effectcd.componentSubType = kAudioUnitSubType_NewTimePitch;
! effectcd.componentManufacturer = kAudioUnitManufacturer_Apple;
!
! AUNode effectNode;
! CheckError(AUGraphAddNode(self.auGraph,
! ! ! ! ! ! ! &effectcd,
! ! ! ! ! ! ! &effectNode),
! ! ! "couldn't get effect node [time/pitch]");
Notice the type is AUFormatConverter, not AUEffect

AudioUnitParameters.h
// Parameters for AUNewTimePitch
enum {
! ! // Global, rate, 1/32 -> 32.0, 1.0
! kNewTimePitchParam_Rate!! ! ! ! ! ! = 0,
! ! // Global, Cents, -2400 -> 2400, 1.0
! kNewTimePitchParam_Pitch! ! ! ! ! ! = 1,
! ! // Global, generic, 3.0 -> 32.0, 8.0
! kNewTimePitchParam_Overlap! ! ! ! ! ! = 4,
! ! // Global, Boolean, 0->1, 1
! kNewTimePitchParam_EnablePeakLocking! ! ! = 6
};
This is the entire documentation for the AUNewTimePitch parameters

AUNewTimePitch
parameters
• Rate: kNewTimePitchParam_Rate takes a
Float32 rate from 1/32 speed to 32x
speed.
• Use powers of 2: 1/32, 1/16, …, 2, 4, 8…
• Pitch: kNewTimePitchParam_Pitch takes
a Float32 representing cents, meaning
1/100 of a musical semitone

Pitch shifting
• Pitch can vary, time does not
• Suitable for real-time sources, such as
audio capture

Demo: Pitch Shift
New in iOS 6!

Rate shifting
• Rate can vary, pitch does not
• Think of 1.5x and 2x speed modes in
Podcasts app
• Not suitable for real-time sources, as data
will be consumed faster. Files work well.
• Sources must be able to map time
systems with
kAudioUnitProperty_InputSamplesInOutput

Demo: Rate Shift
New in iOS 6!

AUSplitter
AUSplitter
AUSomethingElse
AUSomethingElse
New in iOS 6!

AUMatrixMixer
AUMatrixMixer
AUSomethingElse
AUSomethingElse
AUSomethingElse
AUSomethingElse
AUSomethingElse
New in iOS 6!

Audio Queues
(and the APIs that help them)

AudioQueue
• Easier than AURemoteIO - provide data
when you want to, less time pressure, can
accept or provide compressed formats
(MP3,AAC)
• Recording queue - receive buffers of
captured audio in a callback
• Play-out queue - enqueue buffers of audio
to play, optionally reﬁll in a callback

Audio Queue

Common AQ scenarios
• File player - Read from file and “prime”
queue buffers, start queue, when called
back with used buffer, refill from next part
of file
• Synthesis - Maintain state in your own
code, write raw samples into buffers during
callbacks

Web Radio
• Project from Thursday’s workshop
• Use Audio File Stream Services to pick out
audio data from a network stream
• Enqueue these packets as new AQ buffers
• Dispose used buffers in callback

Parsing web radio

Parsing web radio
NSData NSData
Packets Packets Packets Packets Packets
NSURLConnection delivers
NSData buffers, containing audio
and framing info.We pass it to
Audio File Services.

Parsing web radio
NSData NSData
Packets Packets
Packets Packets Packets
Audio File Services calls us back
with parsed packets of audio data.

Parsing web radio
NSData NSData
Packets Packets
Packets Packets Packets
012
Packets
Packets
Packets
Packets
Packets
Packets
Audio File Services calls us back
with parsed packets of audio data.
We create an AudioQueueBuffer
with those packets and enqueue it
for play-out.

A complex thing!
• What if we want to see that data after it’s
been decoded to PCM and is about to be
played?
• e.g., spectrum analysis, effects, visualizers
• AudioQueue design is “ﬁre-and-forget”

AudioQueue Tap!
http://www.last.fm/music/Spinal+Tap

AudioQueueProcessingTap
• Set as a property on the Audio Queue
• Calls back to your function with decoded
(PCM) audio data
• Three types: pre- or post- effects (that the
AQ performs), or siphon. First two can
modify the data.
• Only documentation is in AudioQueue.h

Creating an AQ Tap
! ! // create the tap
! ! UInt32 maxFrames = 0;
! ! AudioStreamBasicDescription tapFormat = {0};
! ! AudioQueueProcessingTapRef tapRef;
! ! CheckError(AudioQueueProcessingTapNew(audioQueue,
! ! ! ! ! ! ! ! ! ! ! tapProc,
! ! ! ! ! ! ! ! ! ! ! (__bridge void *)(player),
! ! ! ! ! ! ! ! ! ! ! kAudioQueueProcessingTap_PreEffects,
! ! ! ! ! ! ! ! ! ! ! &maxFrames,
! ! ! ! ! ! ! ! ! ! ! &tapFormat,
! ! ! ! ! ! ! ! ! ! ! &tapRef),
! ! ! ! "couldn't create AQ tap");
Notice that you receive maxFrames and tapFormat.These do not appear to be settable.

AQ Tap Proc
void tapProc (void * inClientData,
! ! ! AudioQueueProcessingTapRef inAQTap,
! ! ! UInt32 inNumberFrames,
! ! ! AudioTimeStamp * ioTimeStamp,
! ! ! UInt32 * ioFlags,
! ! ! UInt32 * outNumberFrames,
! ! ! AudioBufferList * ioData) {
! CCFWebRadioPlayer *player =
(__bridge CCFWebRadioPlayer*) inClientData;
! UInt32 getSourceFlags = 0;
! UInt32 getSourceFrames = 0;
! AudioQueueProcessingTapGetSourceAudio(inAQTap,
! ! ! ! ! ! ! ! ! ! inNumberFrames,
! ! ! ! ! ! ! ! ! ! ioTimeStamp,
! ! ! ! ! ! ! ! ! ! &getSourceFlags,
! ! ! ! ! ! ! ! ! ! &getSourceFrames,
! ! ! ! ! ! ! ! ! ! ioData);
// then do something with ioData
// ...

So what should we do
with the audio?

So what should we do
with the audio?
Let’s apply our pitch-shift effect

Shouldn’t this work?
AUEffect

Shouldn’t this work?
AUEffect
AudioUnitRender()

AudioUnitRender()
• Last argument is an AudioBufferList, whose
AudioBuffer members have mData pointers
• If mData != NULL, audio unit does its
thing with those samples
• If mData == NULL, audio data pulls from
whatever it’s connected to
• So we just call with AudioBufferList ioData
we got from tap callback, right?

Psych!
• AQ tap provides data as signed ints
• Effect units only work with ﬂoating point
• We need to do an on-the-spot format
conversion

invalidname’s convert-
and-effect recipe
AUGenericOutputAUConverterAUEffectAUConverter
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList * ioData) {
CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inRefCon;
// read from buffer
ioData->mBuffers[0].mData = player.preRenderData;
return noErr;
}
Note: red arrows are ﬂoat format, yellow arrows are int

How it works
• AUGraph:AUConverter → AUEffect →
AUConverter → AUGenericOutput
• Top AUConverter is connected to a render
callback function

The trick!
• Copy mData pointer to a state variable and
NULL it in ioData
• Call AudioQueueRender() on output unit.
The NULL makes it pull from the graph.
• Top of the graph pulls on render callback,
which gives it back the mData we copied
off.

Yes, really
This is the rest of tapProc()
! // copy off the ioData so the graph can read from it
// in render callback
! player.preRenderData = ioData->mBuffers[0].mData;
! ioData->mBuffers[0].mData = NULL;
!
! OSStatus renderErr = noErr;
! AudioUnitRenderActionFlags actionFlags = 0;
! renderErr = AudioUnitRender(player.genericOutputUnit,
! ! ! ! ! ! ! ! &actionFlags,
! ! ! ! ! ! ! ! player.renderTimeStamp,
! ! ! ! ! ! ! ! 0,
! ! ! ! ! ! ! ! inNumberFrames,
! ! ! ! ! ! ! ! ioData);
! NSLog (@"AudioUnitRender, renderErr = %ld",renderErr);
}

Yes, really
! ! ! ! ! ! ! ! ! AudioUnitRenderActionFlags *ioActionFlags,
! ! ! ! ! ! ! ! ! const AudioTimeStamp *inTimeStamp,
! ! ! ! ! ! ! ! ! UInt32 inBusNumber,
! ! ! ! ! ! ! ! ! UInt32 inNumberFrames,
! ! ! ! ! ! ! ! ! AudioBufferList * ioData) {
! CCFWebRadioPlayer *player =
(__bridge CCFWebRadioPlayer*) inRefCon;
!
! // read from buffer
! ioData->mBuffers[0].mData = player.preRenderData;
! return noErr;
}
This is the render callback that supplies data to the int→ﬂoat converter

Demo:AQ Tap +
AUNewTimePitch
New in iOS 6!

Meanwhile in a mini-
bus near
Copenhagan…

Audiobus
• Allows multiple audio apps to exchange
data in realtime
• Works by sending raw data in MIDI
• Actually approved by Apple
• Actually supported in GarageBand

Supporting Audiobus
• Get the SDK from audiob.us
• Enable background mode, add an audiobus-
compatible URL scheme, get API key from
audiob.us
• Create and use ABAudiobusController,
ABOutputPort/ABInputPort, and
ABAudiobusAudioUnitWrapper

Wrapping up…

Takeaways
• Core Audio fundamentals never change
• New stuff is added as properties, typedefs,
enums, etc.
• Watch the SDK API diffs document to ﬁnd
the new stuff
• Hope you like header ﬁles and
experimentation

Q&A
• Slides will be posted to slideshare.net/
invalidname
• Code will be linked from there and my blog
• Watch CocoaConf glassboard,
@invalidname on Twitter/ADN, or [Time
code]; blog for announcement
• Thanks!

Core Audio in iOS 6 (CocoaConf San Jose, April 2013)

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (8)

Semelhante a Core Audio in iOS 6 (CocoaConf San Jose, April 2013)

Semelhante a Core Audio in iOS 6 (CocoaConf San Jose, April 2013) (20)

Mais de Chris Adamson

Mais de Chris Adamson (20)

Último

Último (20)

Core Audio in iOS 6 (CocoaConf San Jose, April 2013)