I am working on an augmented reality game which uses a binaural audio engine called Papa Engine and a low level audio processing api called Novocaine. Novocaine reroutes the microphone data through the users headphones. I mix binaural sound effects into that to create the illusion that these sounds are real (and in 3D space).

Novocaine is open source, but Papa Engine is not. It took me a while, but I managed to get the two of those to play nicely together. Enter OpenEars. I also want my ARG to do voice recognition. During the experience, there is a narrator which tells you what to do. When I disable Novocaine and Papa Engine, OpenEars works great. When I enable either of those first, the application crashes when I call “pocketsphinxController startListeningWithLanguageModelAtPath.” If I enable OpenEars first, the other two work, and pocketsphinxController says that it is listening but voice recognition stops functioning.

Is there any way to setup the audio sessions so that I can get all three of these to play nice?

Sorry, this is unlikely to work although I wish I could give you more encouraging news. Here is a bit more about it from the FAQ: /openears/support/#Q_I8217m_trying_to_use_a_sound_framework_like_Finch_or_another_OpenAL_wrapper_and_things_aren8217t_working_as_expected

Basically, all of these frameworks are making a bid for the session and they all probably require it to a similar extent because low-latency audio i/o is resource-demanding. But, you might be able to find out a bit more about the precise point at which it is breaking if you enable logging: /forums/topic/install-issues-and-their-solutions/ and you can also try searching these forums for some info about PocketsphinxController’s audioSessionMixing property, which is designed to allow session mixing (although this scenario with two other frameworks may still be a bit too much).

Everything appears to work up until this point:

[11748:60b] PreferredBufferSize is now on the correct setting of 0.128000.
:: Buffer::copyHeaderFrom(ch 2 2, fr 256 4096)
:: Error: An error occured in the CSL nextBuffer method: Can’t reallocate buffers at run-time

[11748:60b] preferredSampleRateCheck is incorrect, we will change it.

It is at this point that the app crashes. The crash occurs inside:
AudioToolbox`AURemoteIO::PerformIO

Is there anyway to prevent this crash? I am okay with the two APIs not working simultaneously, but I need to be able to alternate between the two.

It’s unlikely to work to run a generalized engine that requires complete control over the session and i/o at the same time as performing low-latency speech recognition from a mic stream. OpenEars is designed to recover gracefully from other audio session usages when it starts listening, but it can’t avoid taking over the audio stream while it is being asked to perform recognition.

Did you try my suggestion of researching session mixing results in these forums?

[[AudioSessionManager sharedAudioSessionManager] setSoundMixing:true];
self.pocketsphinxController.audioSessionMixing = TRUE;

It only crashes about 50% of the time. When it doesn’t crash, I get a constant stream of this in the console:

:: Buffer::copyHeaderFrom(ch 2 2, fr 256 2048)
:: Error: An error occured in the CSL nextBuffer method: Can’t reallocate buffers at run-time

But otherwise both APIs seem to function. I get binaural sounds, and speech detection works. If I can manage to hack this together so that it doesn’t crash at all, that would be great. I have tried modifying constants such as those in AudioConstants.h but the OpenEars logging seems to suggest that my values are being ignored. I have a gut feeling that if I could match up the buffer sizes that I could make these two play nice.

//changed from 0.12800
#define kBufferLength 0.005805

Both APIs play nicely now, but the speech recognition isn’t functioning all that well. It only detects a word about 20% of the time, and is often wrong. Will playing with any of the other constants improve the speech recognition and return it to its functional state?