Introduction and overview of activities to add CMU Sphinx speech recognition to Opencast Matterhorn.

1. Open Text:Speech recognition in Opencast MatterhornStephen MarquardCentre for Educational TechnologyUniversity of Cape TownJune 2011

2. Project goals Integrate CMU Sphinx speech recognition engine into Opencast Matterhorn Provide easy mechanism for speaker training Generate automatic transcripts of recorded lectures Allow users to correct and improve the transcripts Use feedback to improve recognition accuracy (of the same, similar or subsequent recordings)

3. Why is it important? Video and audio is more useful if you can: Navigate it easily Locate relevant recordings from a large set Use by students: Catch up on missed lectures (continuous play or read the transcript) Revision: jump to a particular point or find the lectures which cover topic X On the public web: Discoverability (search indexing) Similar advantages to OCR recognition of slides (but harder)

4. Why is it difficult? Audio quality can dramatically affect speech recognition accuracy Echo and reverberation Background noise Microphone location Speaker-independent large-vocabulary continuous speech recognition is the hardest type of ASR Best case: good acoustics, single speaker (limited dialogue), accent match with the acoustic model, limited vocabulary.

5. Prior work in ASR for lectures MIT Lecture Browser (SUMMIT recognizer) U. Toronto / ePresence PhD prototype by CosminMunteanu(SONIC recognizer) ETH Zurich Integration of CMU Sphinx with REPLAY by SamirAtitallah

6. Speech recognition software ecosystem Licensing and patents Closed Proprietary FOSS Open

8. Accounting for context:Language model adaptation Adapt a language model to more closely resemble the target speech Using related text for Topic modelling (vocabulary, concepts) Style-of-speech modelling “ok and um it's quite useful to have a very good diagnostic test of of acute hepatitis um you know to prevent kind of unnecessary um surgery um so hepatitis is really one um example of a cause of acute abdominal pain that doesn't need surgery”

9. Using Wikipedia for LM adaptation Goal is to adapt a “standard” LM to be specific to the topic of the audio Start somewhere: title, keywords, text from slides Select a set of documents, adapt the LM Using wikipedia, select by similarity: identify the set of documents most closely related to the starting point or keywords

10. Baseline performance with Sphinx4 (HUB4 acoustic and language models) Lecture audio and transcripts from Open Yale Courses http://oyc.yale.edu/ Used under CC-BY-NC-SA license.

11. Best-case comparison (30% WER)Transcript, HUB4 LM, Wikipedia Similarity LM Before launching into Pynchon today, I thought I would just take a few moments to look back over the books that we've read and talk about the visions of language that they have offered us, and also just to reflect for a moment on the relationship imagined between those visions of language and what is happening outside of fiction in what we might call the real world. We started this course talking about Black Boy and the way that a whole world of pressure -- political pressure, racial tension -- pushed on the borders of that work and actually changed its very material form. before launching into not pynchontoday route just take a few moments to look back cover the books that we've brad and talk about the visions of language that they have offered up and also just to reflect for mounted on the relationship imagine between those visions of language and what is happening outside of fiction in in what we might call the real world we started this course talking about black boy and a weighing bat a whole world of pressure political pressure racial tension pushed on the borders and that work and actually changed its very nature eel for before launching into not mentioned today really does take a few moments to look back over the books that we've read and talk about the visions of language that they have offered up and also just to reflect for movement on the relationship imagine between those visions of language and what is happening outside of fiction in in what we might call the reel well we started this course talking about black boy and a weighing of that a whole world of pressure political pressure of racial tension pushed on the borders of bad work and actually changed its very nature eel for

12. Worst-case comparison (61% WER)Transcript, HUB4 LM, Wikipedia Similarity LM i'd talk with the french revolution this party do in all the myself will forty-five minutes after throughout beginning i'm in seoul on on i wanted it to do two things unless the revolution through the eyes of maps that ulmus piano member of a treaty of public safety arguably without fascists i'd solicit were not member ah is jacobo out into an away he incarnated death jacobinchapel back he imparted the french revolution i've talked with the french are loose in this part to do in all the myself low forty five minutes after score of beginning i'm in seoul on bob and i wanted to do two things i want the revolution through the eyes of maps that elvis piano a member of the treaty of public safety are giveaway with that fascists i thought it were not member ah gee i go back into a a way he imparted that chappel been the chapel back he imparted the first revolution I'm going to talk about the French Revolution. It's hard to do. I'll leave myself about forty-five minutes after I screw around at the beginning. I want to do two things. I want to see the Revolution through the eyes of Maximilien de Robespierre, a member of the Committee of Public Safety --arguably, with Saint-Just, its most important member. In a way, Jacobin -- he incarnated the French Revolution.

13. Work in progress Identify requirements for recording recognition-quality audio (equipment, acoustics) Implement dynamic language model adaptation Integrate into Opencast Matterhorn workflow Show transcript to users in UI, enable search Allow users to edit / improve transcript Use edits to improve recognition

14. Other integration possibilities External transcription services (automate the workflow, choice between manual or automatic transcript) External speech recognition services (e.g. nexiwave.com)

15. Find out more Email me:stephen.marquard@uct.ac.za Follow me on Twitter: http://twitter.com/stephenmarquard Read my blog on open source language modelling and speech recognition: http://trulymadlywordly.blogspot.com CMU Sphinxhttp://cmusphinx.sourceforge.net/