Image by Allie Brosh

Well, perhaps not all of the languages. Many of the languages! I’ve developed a language-agnostic grapheme-to-phoneme algorithm which works from a file format fast enough for a phone, and although perfection remains elusive, it is from 5-15x more accurate for generating phonemes than a naive letter-based algorithm. As a result of this, OpenEars is now able to support speech recognition with English, Spanish, Mandarin Chinese, French, German, and Dutch. The new languages will now work with all of the speech recognition features of OpenEars, including dynamic language model generation and switching, grammar generation and switching, and of course hypothesis return.

As a benefit of the flexibility of this format, today Politepix will also release 2.5 versions of every plugin. RapidEars, Rejecto and RuleORama are now compatible with the languages OpenEars is compatible with. NeatSpeech 2.5 is a compatibility update, but won’t be adding TTS output for the new languages. SaveThatWave also has a compatibility update.

By the way, if you are using another language’s Sphinx-compatible acoustic model and you want to make it compatible with OpenEars, you can get in touch to discuss options – it is a pretty flexible approach so I expect to be able to apply it to more languages.

Speech recognition will vary significantly due to the speed and accuracy level of the acoustic model used, so support for non-English acoustic models is very much on a best-effort basis moving forward. Tut mir leid, Schatz!

The next feature of the OpenEars Platform 2.5 is bitcode. OpenEars will now ship with embedded bitcode, and bitcode is now a configurable option for paid plugin framework purchases (subject to a handling fee). Plugin framework demos will also now have a non-recompilable bitcode segment so that they install easily, but the bitcode attached to the demos can’t be submitted to the App Store. Demos can’t be submitted to the App Store in any case, so this shouldn’t be an issue.

Bitcode is also supported on a best-effort basis, meaning that no warranty is made for what it does when it is recompiled. This is not because I don’t care, but because a) no one knows the minutiae of how it will be recompiled, or b) what architectures it will be recompiled with, or c) what the strategic importance of bitcode is to Apple really, and it is a fair assumption that d) there will not be any communication about what is happening if it doesn’t work, so that is a process that, realistically, Politepix has no control over. What is the advantage of bitcode from the developer perspective? Hoe komt een ezel aan twee lange oren?

This OpenEars Platform update also fixes all verified bugs to date, which is my very favorite kind of update. Other than the ones which add Chinese speech recognition, 哇!

OpenEars 2.5 is free as always. The 2.5 paid plugin framework license upgrades are free if your purchase was made after August 17th, 2014. Upgrades for licenses purchased before August 17th, 2014 are 50% off. The bitcode handling fee is an extra fee for all upgrades if you want to add bitcode to your frameworks. The new Licensee site has rolled out (same URL as before), and on it you will find coupons for your upgrades in your download area, either for a free upgrade or a 50%-off upgrade for each of your licensed plugins. Quel délice!

This is a big update with a lot of moving parts, so if you encounter any surprises or frustrations, no te preocupes, just visit the forums, let me know what’s up, and I will be happy to help.

OpenEars 2.5 can be downloaded here and you can browse the compatible language acoustic models here. Make sure to check out the license for the language to be sure you’re able to use it with your project.

I’m delighted to be able to bring the real potential of localization to offline speech recognition with OpenEars, and I can’t wait to see what you do with it. And to the developers who have been waiting for their language to be compatible, a heartfelt welcome/bienvenue/Willkommen/Welkom/Bienvenido/歡迎!

-Halle

When I first designed (OE)LanguageModelGenerator years ago I made the decision to require text input in uppercase letters for best results because it allowed the most optimization for very fast creation of dynamic language models. This didn’t seem like a big trade-off because at the time the size of a language model needed to be quite small in order to perform well during speech recognition on supported devices such as the 2nd-gen iPhone, which meant that for the most part it was command-and-control applications that were being developed with OpenEars. For a command-and-control vocabulary, word case is not such a big consideration in a UI because the words are out of context. Rather than transforming the developer’s text input automatically, I made the decision to support both all-caps and mixed-case but explain in the docs and in the logging output that mixed-case text input would have to be sent to the fallback phoneme lookup technique which would result in fewer available pronunciations, which would have an accuracy impact for words with multiple pronunciations. This felt like the least-bad compromise between strongly-competing concerns of speed, minimizing complexity, and not discarding the developer’s intentional choices.

Over the last couple of years as the devices and the framework and dependencies have gotten faster, it has become a viable choice with OpenEars to use larger vocabularies, and as a result more app developers have been using it with a broader variety of input sources such as written texts, speeches, etc, which is delightful to see. For that kind of application, the case of the input and output format matters to the developer and the user. The uppercase requirement/advantage no longer supported the goals of the developer or of pleasing UX and needed to be improved, so I revisited this early decision and found a way to do case-insensitive lookup without changing the baseline generation speed, and also improved the generation speed for larger models. That means that you can use normal word and sentence casing in your input text and it will be returned by your speech recognition hypothesis with the same casing intact, and larger text input will generate models faster (this doesn’t affect recognition speed, just how long dynamic model and grammar generation take).

There has also been an improvement in handling of punctuation in input, so in the cases that developers don’t do their own text cleaning to remove symbols which are too ambiguous to transcribe and probably not intended to be spoken (for instance, symbols like { or ^ or `) OELanguageModelGenerator will clean the input and it will be consistent across all the plugins and different model/grammar types. Symbols that can’t be transcribed will be removed, symbols which can be transcribed will usually be transcribed by the best-effort fallback grapheme generator (so you should still take a look at your input when you know it in advance and decide whether it would be better for you to transcribe your symbols into words yourself, especially numbers because only you know for sure whether you want 1600 to be transcribed as ‘sixteen-hundred’ or ‘one thousand six hundred’ or ‘a thousand six hundred’ or ‘one six oh oh’), and symbols which aren’t significant for recognition purposes (such as . or , or ; or ? or !) will be left in place and will become part of your model.

An example of this last point would be if you used the sentence “The Sand Snakes are with me.” as input. OELanguageModelGenerator will successfully find multiple pronunciations for any word in this sentence that has more than one pronunciation – it will leave the case intact and there will be no accuracy decline from that. That period (full stop) symbol at the end will stay attached to the word “me” in the model, meaning that when OEPocketsphinxController returns a hypothesis matching an utterance of the sentence, it will still have the period attached to it in the returned text hypothesis. If this isn’t the desired result and you don’t want the individual words in this input to have hints about their position in a sentence or statement, you can still give the original text to OELanguageModelGenerator without sentence punctuation, but the assumption now is that if you give sentence punctuation as input, it’s because you intend for it to be returned in a hypothesis. That also means that if you create a language model rather than a grammar, you can sometimes see a word with a period or comma appear in a different order in the sentence other than the input order, so that is something to think about when using punctuation and evaluating whether to use a language model (statistical model; words can be returned out of order so a word with a period attached can theoretically appear in the middle of a sentence if someone walks by the user and says it) or a grammar (ruleset; the order you choose is the order that will return).

The decision tree I use for these punctuation transformations and non-transformations is basically a simplified non-interactive version of my interactive text-cleaning tool TheKnownUnknowns, so please feel free to take a look at TheKnownUnknowns alongside OELanguageModelController for more info about considerations with different symbols. Please also feel free to use TheKnownUnknowns for preparing texts for OpenEars where you’d like to make your own decisions in advance about how to transcribe difficult cases. It is primarily designed to quickly clean text corpora before creating an acoustic model using long alignment and similar tasks on large texts that have to be prepared for some kind of transcription-related norm, but it is also a good tool for interactively cleaning text you want to use with OpenEars in advance, since they have their design and major assumptions in common.

Although this is not directly a recognition accuracy change, my sense is that there was a cluster of minor accuracy-related symptoms in some apps related to non-transcribable symbols entering the generator, mixed-case being used without realizing it affected how many pronunciations were found, and the possibility that unknown transcribable or ignorable symbols were being handled differently by the language model/grammar lookup than by the phonetic dictionary lookup, which theoretically could result in never-matching words. Projects that were experiencing any of these issues should see an improvement to accuracy from this change.

As always, OpenEars can be downloaded here and the new plugins can either be downloaded at your demo link or your licensed framework link. I hope this little improvement helps you make great apps!

[politepix-blog-inline-text-ad]

NeatSpeech is a plugin for OpenEars that adds fast, higher-quality speech synthesis (TTS) including multithreaded speech queueing to speak very long phrases without any lag. It uses HTS voices which are ideal for mobile applications due to their diminutive file size, great processing speed and improved speech quality over the default Flite voices. All TTS is performed offline so absolutely no network connection is required.

[politepix-blog-inline-text-ad]

You can try out a free demo of NeatSpeech (requires new OpenEars version 1.2.3) here, purchase NeatSpeech here or read more about it here. I really hope you enjoy NeatSpeech!

[politepix-blog-inline-text-ad]

Politepix has also released a new paid plugin called Rejecto, which makes your apps more accurate when you use OpenEars and RapidEars by tuning out words that aren’t in your language model. As usual it’s easy to get started with but it has a lot of knobs to twiddle if you want to optimize your model as far as possible. Check it out at /rejecto.

While you’re trying these new tools out, why not make it easy on yourself and visit Politepix’s new tutorial tool, which will get you up and running with your OpenEars app about 5 minutes. It’s found at /openears/tutorial and it includes instructions for all the OpenEars classes and all of the plugins too!

Today is a big release day for Politepix: OpenEars 1.1 and the introduction of the OpenEars plugin system! Featuring RapidEars, a plugin for OpenEars that does recognition on live speech.

I am very happy to be introducing multiple Cool Things this Monday. The first is the 1.1 version of OpenEars, which was a bit behind schedule but I hope was worth waiting for, since it introduces…n-best! So you can now see the top “hits” for word matching and the scores for them, rather than only seeing the best hypothesis. You can just turn on n-best and set the number of best hypotheses you’d like in PocketsphinxController, and then add the n-best delegate method of OpenEarsEventsObserver to receive them wherever you’d like. The docs and sample app have been updated to explain how.

[politepix-blog-inline-text-ad]

The other big new feature of OpenEars 1.1 is that it now has a method that is entirely designed to make it easier for you to test, which is:

– (void) runRecognitionOnWavFileAtPath:(NSString *)wavPath usingLanguageModelAtPath:(NSString *)languageModelPath dictionaryAtPath:(NSString *)dictionaryPath languageModelIsJSGF:(BOOL)languageModelIsJSGF;

This just runs recognition using your normal settings on a single WAV audio file. You can use this to create a testbed in case, for instance, you are trying to see the effect that changes to a language model or acoustic model or other settings have on recognition. There’s more about both of these new features in the documentation and as always, feel free to ask your questions on the forums.

The OpenEars Platform and Plugins

The next big new thing is that OpenEars now has plugins. Over the coming months Politepix will be introducing new drop-in plugin frameworks for your OpenEars app that will open up new functionality. OpenEars will remain free and will also continue to be developed, with new features being added that make use of the potential of its dependencies, such as today’s introduction of n-best. For in-demand functionality that requires major committed development time and brand-new internal and dependency code, those features will be produced as plug-ins sold at the Politepix shop. So OpenEars is now both a free recognition, TTS and language modeling SDK, and also a platform so you can mix and match the exact extended functionality you need without making your app more complex.

Without further ado, it’s my pleasure to introduce Politepix’s first plugin for the OpenEars platform: RapidEars! RapidEars is a plugin that adds live recognition capabilities to PocketsphinxController, so by trading off some accuracy, you can run recognition on speech as it is recorded into the device instead of waiting for a pause and evaluating afterwards. This uses more CPU than OpenEars so it’s advisable to use this for apps targeted at the newest devices. It’s great for games or any app where you want immediate responsiveness. There is nothing like this available using any other speech recognition product because they are all network-based and OpenEars is processed on the device. RapidEars is €999 to license and it has a free demo version for development that can be used for four minutes at a time so you can test it thoroughly before licensing. Installation is as simple as dragging the new plugin into your project and importing its classes in order to use new PocketsphinxController and OpenEarsEventsObserver methods.

You can read more about RapidEars here and download a demo version of the plugin here. Questions for this OpenEars plugin and future ones can also be asked on the OpenEars forums, just mention that you’re asking about RapidEars.

That’s it, have fun with these new products!

-Halle

• Leaks in the dynamic language model generation process could crash the device,
• When using a very small model, the discounting method used by the language model generation process could occasionally result in certain words having a probability that is so low that it was equivalent to not having them in the model at all,
• There were 237 warnings during a device build due to unstripped local debug symbols in the framework, which had no effect on functionality but occluded useful warnings.

These should all be fixed, and these are the only current bugs I am aware of.

This upgrade is recommended for all users. To upgrade, I recommend removing the previous framework version from your app, moving the previous framework download folder away or to the trash, and then do a fresh install of the new framework as described here: /openears under the heading “To use OpenEars:”.

[politepix-blog-inline-text-ad]

Then you should select Product->Clean and, holding down the option key while selecting the Product menu again, select Product->Clean Build Folder and then rebuild your app with the new framework.

Please let me know in the forums whether this new version is fixing the issues that you’ve been experiencing, and of course whether you are experiencing any other issues. For those who have been hit by the two issues with LanguageModelGenerator, thanks for your patience while I added the memory management and fixed the issue with the discounting method.

It is my pleasure to announce a new Politepix framework product for developers: SpeexKit 1.0.

What is Speex?

Speex is one of the best codecs for voice audio transmission available, with clear quality and excellent compression characteristics. It is a great match for mobile speech applications given the need for top-notch clarity and careful use of bandwidth. The Speex project is found here.

[politepix-blog-inline-text-ad]

What is SpeexKit?

SpeexKit is an iOS framework from Politepix that makes it easy to use every feature of Speex with your iPhone or iPad app without doing your own C or C++ implementation or having to deal with threading or queueing code. It is not the only way of using Speex on the iPhone, but it is designed to be an easy object-oriented way to implement it quickly in your app, even for the tricky stuff. You can download the SpeexKit demo framework here. You have to sign up, but downloading the demo is 100% free, just add it to your cart and check out. You will not have to provide any payment information to try it out. It is time-limited and can’t be submitted to the App Store — when you’re ready to license the framework it can be found on the Politepix shop.

What can you do with SpeexKit?

• Convert entire WAV or raw audio files into Speex files
• Convert entire Speex files into WAV files
• Convert buffers of Speex frames into buffers of raw audio samples, synchronously or asynchronously. In asychronous mode, threading and queuing (since Speex decoding is not realtime on all devices) are handled for you.
• Convert buffers of raw PCM audio samples into buffers of Speex frames, synchronously or asynchronously. In asychronous mode, threading and buffer queuing (since Speex encoding is not realtime on all devices) are handled for you.
• Wrap existing buffers of Speex frames into a written-out Speex file
• Wrap existing buffers of raw PCM audio samples into a written-out WAV file
• Use advanced Speex features such as such as Voice Activity Detection (VAD) and echo reduction

You can read much more about SpeexKit here. Thanks for checking it out!

• OpenEarsEventsObserver was never receiving a delegate message when a language model was being changed (thanks to Aleksey for finding this bug)
• LanguageModelGenerator was not finding alternate pronunciations of a word (thanks to Sarinsukumar for finding this bug)
• AudioSessionManager will now correctly reset audio session settings that a media player object (such as an AVPlayer or MPMusicPlayerController among others) overrides if startAudioSession is run a second time afterwards. If a PocketsphinxController object is already instantiated and its loop has been started at the time that a media player object has overridden its audio session settings, after AudioSessionManager startAudioSession is run the second time, you _may_ also need to restart the listening loop if the media player’s audio session override gave PocketsphinxController an interruption signal that caused it to make a controlled exit from the listening loop.

[politepix-blog-inline-text-ad]

I strongly recommend upgrading to this version immediately if you use dynamic language model generation since it will improve recognition across different accents. Enjoy and let me know how it works for you, in the forums as usual.

I just uploaded the new version of OpenEars, featuring the following new stuff among other things:

• All audio functions (Flite speech and Pocketsphinx recognition) handled in-memory
• Low-latency Audio Unit audio driver for Pocketsphinx
• Switching between all the Flite voices on the fly
• Dynamically creating ARPA language models in-app
• Switching between different ARPA language models on the fly
• Experimental Bluetooth support
• Improved Pocketsphinx accuracy
• Re-documented for Xcode 4.

[politepix-blog-inline-text-ad]

Since this is such a big refactor under the hood, and there are a couple of APIs that have changed, I am still going to support the previous version 0.9.02 until 0.91 is known to be as stable as that version, so you can still download that older version too (previous docs are included in the download). Please check out OpenEars 0.91 and put it to the test! As always, bring any issues over to the forums and I will assist.