So, I’ve been doing some experimentation, and it turns out, one of the new features Apple announced for iOS 8, “Extensions”, seems to work pretty well for implementing audio plugins.
What I mean by that, is allowing one app to add audio commands directly to another app. Note that these are offline commands, rather than realtime ones — Audiobus, Core MIDI and Inter-App Audio are the way to send live audio or MIDI around. But for apps that can do destructive edits to audio, this is pretty nifty. Or even for apps that don’t edit audio at all, but have an audio library.
If you’ve ever done the following:
Selected some audio in App A
Used AudioCopy/Paste or AudioShare to copy it
Switched to App B
Used AudioCopy/Paste or AudioShare to paste it
Used App B to do some processing on the sound
Used AudioCopy/Paste or AudioShare to copy it again
Switched back to App A
Used AudioCopy/Paste or AudioShare to paste the processed version, finally!
…then this will be of interest to you, because now, it can instead work like this:
This is a really crude test, and the example plugin is slow and ugly, but of course it doesn’t need to be – it’s just to prove the concept works. Four taps, all inside Mitosynth, and the audio library gets a new piece of audio, with a special effect applied, supplied by another app. All relatively seamless.
Also, although this test plugin pops up a panel of controls, for something like reversing audio, it doesn’t need to. It can just apply the effect immediately you tap the “Reverse Audio” icon. So then it would just be two taps. Pretty convenient.
This is still in the experimental stage, but I’d like to get feedback from users and other developers, so I’m putting it out there…
Will people find this useful? Will developers support it? I don’t know, but, the best bit about this, is that it can work between apps from different developers, with no need to define a new standard or SDK to support it: it just uses built-in iOS features which some developers may support already, and can simply upgrade slightly for iOS 8.
This is important for a few reasons. One, it makes it more likely to get support from devs. Otherwise there’s a chicken/egg problem: Why write plugins if there are no apps that support them? Why support plugins in your app if there aren’t any plugins yet? I’m hoping that the answer goes something like:
This plugin system is based on standard iOS Sharing Sheets
You should support iOS Sharing Sheets anyway! They’ve been around since iOS 6 and are pretty useful.
Once you support Sharing Sheets, supporting audio plugins is almost free, so you might as well!
So hopefully, a bunch of apps will support plugins even if there aren’t actually any plugins written yet, simply because it’s so easy, it’s daft not to. And that should then encourage plugin writers.
But another reason why it’s nice that it uses built-in iOS stuff instead of Yet Another SDK, is this: As iOS audio developers we already seem to be expected to support one or all of (depending on type of app): basic Audiobus, Audiobus State-saving, Inter-App Audio, Core MIDI (USB), Core MIDI (WiFi), Core MIDI Bluetooth Configuration (in iOS 8), Virtual MIDI, Inter-App Audio “AURI” alternate MIDI API, MIDI clock/sync, WIST, MIDI channel-per-note, background audio, AudioCopy/Paste, AudioShare, iOS-native sharing (Open In & Sharing Sheets), iCloud, Dropbox, Bluetooth headphone/mic support, Multiroute Audio IO… on top of whatever the app is actually supposed to do! And then there’s all the other stuff regular non-audio apps should support… and with iOS 8 Apple are pushing all developers to support Universal, fully-resizable apps with dynamic type…
All of this in a market where many app developers are already struggling to make a living. I don’t really want to add even more weight on their shoulders, or Yet Another Standard.
But, I don’t feel bad about adding iOS 8 audio plugins to the list, because there’s a good chance developers are – or will be – supporting iOS Sharing Sheets anyway, and this gives us a pretty hassle-free plugin mechanism practically “for free”.
The rest of this article is going to be about the technical implementation details, so folks who don’t develop apps can probably stop reading here – although, if you could direct the developers of your favourite apps this way, that would be awesome. Remember though that this is for offline plugins, not realtime ones. Don’t pester devs for whom this is irrelevant! Thanks!
As I mentioned, since iOS 6, iOS has supported a Sharing Sheet (aka UIActivityViewController) which allows an app to hand a piece of content (some text, an audio file, a photo, etc) to the system to be shared, for example, on Twitter or Facebook.
App developers can add their own commands to the list, but they only apply within that app. So, in the video above, you can see that Mitosynth adds Dropbox, AudioCopy, AudioShare, and some other commands. But these are only available within Mitosynth.
New to iOS 8, app developers can write “Sharing Extensions” that add new commands to the sharing sheet. So if you want to share in other ways (straight to Soundcloud, perhaps?) someone could write a plugin to do that.
But also in iOS 8, those plugins can write “Action Extensions” that change the data that’s passed to them, and hand the changed version back to the host app. And that gives us audio plugins. :)
The data that gets passed back and forth is simply an audio file. Or to be precise, a file:// URL that points to one. This means we’re implicitly passing around a name for the audio file (you can see in the example video, the plugin knows the file was named “Example Speech” and returns a file named “Example Speech (Reversed)” to Mitosynth. We just get the last path component and chop off the file extension).
It also means we don’t need to figure out how host apps and plugins communicate the audio format, because it’s in the file’s header – if you use this plugin mechanism, you should use files that can be opened natively by the AudioFileExt API. The host or plugin can set the AudioFileExt’s Client Format and that way, format negotiation is taken care of for us by iOS.
Apps that want to host plugins barely have to do anything. Just use the standard UIActivityViewController and make sure you hand it the URL of a valid audio file and pass a Univeral Type Identifier of kUTTypeAudio, and that’s the sending part done. You may already be doing this part in iOS 6/7!
To receive the modifications back, instead of using the completionHandler, on iOS 8 you should use the new completionWithItemsHandler which will return the changed audio back to you.
And that’s all there is to it.
Not already using UIActivityViewController? Read on for more…
Essentially it’s a system-supplied view-controller that you present, after passing it a list of “activity items”. Activity items are any objects you like, that conform to the UIActivityItemSource protocol. They must implement these two methods from the protocol:
- (id)activityViewControllerPlaceholderItem:(UIActivityViewController *)activityViewController
{
// Return a placeholder -- an empty object of a compatible type
return [NSData dataWithBytes:nil length:0];
}
- (id)activityViewController:(UIActivityViewController *)activityViewController
itemForActivityType:(NSString *)activityType
{
return [NSURL fileURLWithPath:path_to_audio_file];
}
When the activity sheet is done, the completionWithItemsHandler is called. If the completed parameter is YES and the activityError is nil, you should have some results to process, in the form of an NSArray of NSExtensionItems.
Each item has an attachments array of NSItemProviders which you can query to see if they contain audio:
if (completed && !activityError)
{
for (NSExtensionItem* item in returnedItems)
{
for (NSItemProvider* provider in item.attachments)
{
if ([provider hasItemConformingToTypeIdentifier:(NSString*)kUTTypeAudio])
{
provider loadItemForTypeIdentifier:(NSString*)kUTTypeAudio options:nil completionHandler:^(NSURL* audioURL, NSError *error)
{
// audioURL now points to a file, which you can access, containing the audio.
// bear in mind, this can arrive in a background thread.
// you probably want to dispatch_async() back to the
// main thread once you've loaded the data...
}];
}
}
}
}
And that’s all there is to it…
Unsurprisingly, this requires more effort. You need to implement an iOS 8 “Action” extension that is activated by kUTTypeAudio. The full details of this are beyond the scope of this article, but, creating an extension using the default Xcode template for Action extensions will get you much of the way there.
Open up its Info.plist, find the NSExtension item, which will have an NSExtensionAttributes. You’ll need to fiddle with this to make your plugin active for audio files.
The NSExtensionActivationRule will probably have a dictionary of stuff in it. Just delete all the entries and change it from a dictionary to a string. Fill in the following eccentric statement:
SUBQUERY(extensionItems, $extensionItem, SUBQUERY($extensionItem.attachments, $attachment, ANY $attachment.registeredTypeIdentifiers UTI-CONFORMS-TO "public.audio").@count == 1).@count == 1
This will allow your extension to be activated when the user has selected a single audio file. If you can handle multiple files at once, you could try…
SUBQUERY(extensionItems, $extensionItem, SUBQUERY($extensionItem.attachments, $attachment, ANY $attachment.registeredTypeIdentifiers UTI-CONFORMS-TO "public.audio").@count == 1).@count >= 1
…but I haven’t tested that one personally.
When the user taps your plugin, you’ll receive the audio files via the extensionContext, which has an inputItems array containing NSExtensionItems – just the same as when results are passed back to the app, and you can read them the same way:
for (NSExtensionItem *item in self.extensionContext.inputItems)
{
for (NSItemProvider *itemProvider in item.attachments)
{
if ([itemProvider hasItemConformingToTypeIdentifier:(NSString*)kUTTypeAudio])
{
itemProvider loadItemForTypeIdentifier:(NSString *)kUTTypeAudio options:nil completionHandler:^(NSURL *audioURL, NSError *error)
{
// audioURL now points to a file, which you can access, containing the audio.
// bear in mind, this can arrive in a background thread.
}];
}
}
}
You should always use this approach of searching for the correct type of data, whether writing a host or a plugin, because there may be other attachment types in the mix, that you want to safely ignore.
When you’re done doing whatever processing you do:
[self.extensionContext completeRequestReturningItems:items completionHandler:nil];
…where items is, again, an NSArray of NSExtensionItem instances, each with an array of attachments. You create the audio attachment by calling [[NSItemProvider alloc] initWithItem:audioURL typeIdentifier:(NSString*)kUTTypeAudio].
I have found ways to pass extra information back and forth between app and plugin, while remaining compatible with apps that only understand regular audio files. However, I’m not sure if it’s a good idea to go down that road or not. We can already send audio data, format and filename. And going down that road starts to veer into “yet another SDK to support” territory.
Another reason is that I’m not certain how robust the mechanism I’ve found is, in the face of future iOS updates. But, we’ll see. I’d like to get feedback on the basic plugin mechanism first, see if people are interested in supporting it, and then maybe look into extending it.
Feedback? Comments? Spotted typos? Let me know. Thanks!