Dragon and the Rainbow Passage
Submitted by emiliaserv on Thu, 10/04/2007 - 18:45.
Just curious really, but I wonder how useful the Rainbow Passage actually is in assessing accuracy, the reason I ask is that having just read it, I gain the distinct impression that Dragon already knows it off by heart. At first attempt, I made five or six very small mistakes (on two or three letter words), yet certain phrases that seemed quite difficult were recognised without problem. Although I am using British English, I got several American spellings (as they appear in the passage) and this has been enough to arouse my suspicions.
Anyway, I saved my user file straight afterwards, about to shut down for the day now, but it will be interesting to see how I fare next time.
Graham
emiliaserv wrote: Just
Just curious really, but I wonder how useful the Rainbow Passage actually is in assessing accuracy, the reason I ask is that having just read it, I gain the distinct impression that Dragon already knows it off by heart.
The Rainbow Passage is actually very useful. It was designed by linguists to represent a proper distribution of the phonemes (individual sounds within words) used in the English language. It is not surprising that you only received three or four errors. That is 99% accuracy, better than any Plantronics wireless microphone I ever tested and the results we received when testing the excellent VXI Bluetooth wireless microphone you are using.
--
Martin Markoe, eMicrophones, Inc.
The best microphones for Speech Recognition
Providing free pre-sale support for 14 years
As Martin says the
As Martin says the “Rainbow Passage” is a text devised by a linguist to contain all the phonemes in the English language in roughly the same proportion as in everyday speech. It is widely used by linguists, for example when researching dialect variations, by speech therapists, and in courses such as public speaking.
If you download the file from:
http://www.abdn.ac.uk/langling/resources/Standardised%20reading%20passag...
you will see a number of other “standard” texts that can be used for these purposes.
“Comma Gets a Cure” now seems to be the popular choice for linguists researching dialects to judge by the number of dialect recordings available on the Web. For younger children the text “Arthur the Rat” is commonly used.
Given a user with clear enunciation Dragon should be able to interpret the phonetics of these texts very well as the Acoustic model and the phonetic rendering of the Vocabulary words are provided by the very research linguists who devised the texts.
There is, however, a further step when NaturallySpeaking transcribes dictation; namely the Language Model(s). These models look at the statistical probability of words occurring in association (bi-gram, tri-gram and quad-gram models) and they may well initially not interpret unusual or literary texts correctly if the word associations are abnormal. For example if the Language Model has been carefully developed for a specialist scientific or medical discipline there may well be more errors in transcribing a text like the Rainbow passage.
One misrecognition from the Rainbow Passage, produced by several people that I have trained is - “This is a very common type of blow, …”. I don’t accuse the users of substance abuse but rather that “type of blow” (as in punch or wind) is a more probable association than “type of bow”.
“Arthur the Rat” (often transcribed by Dragon as Arthur the Rap) is designed to be used by children and contains simple language and short sentences. Dragon is generally less accurate with this type of text as the Language Model works best with longer utterances (greater than five or nine words) when dealing with short, common, words. Writers of books for younger children and TV scriptwriters often have difficulties when using speech recognition.
My own view is that the best way of assessing NaturallySpeaking accuracy is to use a “standard text” that is typical of the users type of dictation. Ideally the text should be longer than the 330 words of the Rainbow Passage to give a better estimate of the number of errors.
I am intrigued by the observation by Graham that his UK Language version is giving American spelling for words in the Rainbow Passage. This can/should only happen if those words (and associated pronunciations) are present in the Vocabulary.
Graham – were you reading from a paper copy of the text or were you reading from the Rainbow Passage text on screen and in the same document where you were dictating?
Graham
www.itspeaking.co.uk
IT Speaking wrote: My own
My own view is that the best way of assessing NaturallySpeaking accuracy is to use a “standard text” that is typical of the users type of dictation. Ideally the text should be longer than the 330 words of the Rainbow Passage to give a better estimate of the number of errors.
The reason we do not use a longer text is because the longer one reads, the more chance for human error (mispronounciation and slurring) to color the test. We are looking for the ability of a microphone to accurately reproduce words in various noise conditions. Using the 330 words/commands of The Rainbow Passage minimizes the human factor.
--
Martin Markoe, eMicrophones, Inc.
The best microphones for Speech Recognition
I was reading the text as it
I was reading the text as it appears at Marty's site and dictated into DragonPad. For example, the word coloured (that now appears correctly) was spelt the American way, I just had a look in the vocabulary and the word does indeed appear with a green asterisk although obviously the British English spelling is there as well.
I don't know whether this has something to do with it, but just before reading the Rainbow Passage, I had used the Acoustic Optimizer for the first time ever since I started using Dragon.
Graham
Martin Markoe wrote: The
The reason we do not use a longer text is because the longer one reads, the more chance for human error (mispronounciation and slurring) to color the test. We are looking for the ability of a microphone to accurately reproduce words in various noise conditions. Using the 330 words/commands of The Rainbow Passage minimizes the human factor.
Actually, this is a misconception. Reading shorter passages like the Rainbow Passage may be a good indicator of general accuracy but overall accuracy is not calculated based on user mistakes. For example, if a user miss dictates a word or phrase but Dragon NaturallySpeaking gets it correct, this is not a misrecognition error and doesn't get calculated into the overall accuracy calculations.
The true measure of overall accuracy of Dragon NaturallySpeaking itself will be higher the longer the user dictates without pausing. For example, dictate the following without pausing and don't worry about punctuation.
The time has come the walrus said to talk of many things of shoes and ships and sealing wax of cabbages and kings and why the sea is boiling hot and whether pigs have wings
from the Looking Glass by Lewis G. Carroll (just dictate the above don't dictate the reference to Lewis G. Carroll)
Previous versions of DNS would not get this right on first dictation. DNS 9 gets this correct on first dictation, or should, never having dictated this before.
In general, reading standard documents that consists of 1500 or more words without looking at the screen and reading them at a normal pace will generally produce between .5% and 3% greater accuracy based on the user's dictation style and enunciation. Longer passages have the advantage of showing the user where they're making their mistakes.
In addition, the average user generally dictates in short choppy phrases of four words or less. This is not conducive to good accuracy. Learning to dictate in phrases consisting of 9 words or more, complete sentences, or even paragraphs will always produce better accuracy. The biggest problem is that there is a significant difference between reading the text of a document versus free-form dictation. The real accuracy attained by any user is measured by the degree of accuracy that the user gets during free-form dictation. Dragon NaturallySpeaking is specifically designed to be more accurate when reading documents for the reasons specified above.
It is important not to confuse the average user by failing to distinguish between reading a document and free-form dictation. The average user is left in a quandary because they attain a very high degree of accuracy reading the Rainbow Passage, but find that their accuracy when dictating e-mails or documents is noticeably lower, and even sometimes significantly lower.
Chuck Runquist
Former Dragon NaturallySpeaking SDK & Senior Technical Solutions PM for DNS
"We learn by doing." -- Aristotle
Chuck wrote: Actually, this
Actually, this is a misconception. Reading shorter passages like the Rainbow Passage may be a good indicator of general accuracy but overall accuracy is not calculated based on user mistakes.
We use it to test microphones (purely the audio). We need as standard a unit as possible and the text must be as repeatable as possible.
Is our system perfect? No, but it works for us.
--
Martin Markoe, eMicrophones, Inc.
The best microphones for Speech Recognition
Read, "Key Steps to High Speech Reco
Martin Markoe wrote: We use
We use it to test microphones (purely the audio). We need as standard a unit as possible and the text must be as repeatable as possible.
Is our system perfect? No, but it works for us.
Marty,
Thought that I had acknowledged that in my post. Nevertheless, for the purpose for which you use the Rainbow Passage, I would agree that that is viable for microphone testing. It's short, it's sweet, and it's linguistically comprehensive. I use it myself from time to time for testing new users. In addition, I also use the quote from the Looking Glass as well. In fact, I used to use that at trade shows and user group demos to demonstrate free-form dictation using long phrases versus short choppy ones.
The only thing that I was trying to point out is that DNS works best when reading documents versus free-form dictation, unless the user has trained themselves to dictate in long phrases of nine words or more before pausing. In fact, I ask anyone in a demonstration setting to give me an article from a newspaper or magazine to read because I know that DNS 9.5 will get virtually 99.9% accuracy doing so. Most the time DNS gets a long article that I read 100% correct.
Regardless, every method has its value, and for you using the "Rainbow Passage" is quick and effective.
Chuck Runquist
Former Dragon NaturallySpeaking SDK & Senior Technical Solutions PM for DNS
"Any simple idea will be worded in the most complicated way". -- Malek's Law