Open Source Voice and Vision Enabled Tutorial Software

Equip a computer with a webcam and a headset microphone and it can watch and listen to a student while delivering audio and visual instruction.

One open source project developing these capabilities is Open Allure DS, a dialogue system written in Python.

The idea is to have as natural and as immediate a connection between a student answer and the instructional response as possible. The structure for this in Open Allure is

Question

Answer {separator} Response
Answer {separator} Response
...

(see more details here)

Words of the Question and Answer(s) are read aloud using text-to-speech. The Responses are not shown. The student then has the option of speaking or gesturing toward their selected answer (see demo video). The Response associated with their choice is then read aloud.

For example, note the specific feedback of this script:

Spell the word for the surface over your head:

Seelling ; This spelling has the correct sounds but not the right letters. Try again.
Cieling ; This spelling has the right letters but in the wrong order. Try again.
Cielling ; This spelling has an extra letter L. Try again.
Ceiling ;; Correct. Some people think ceiling comes from the French word for sky, but that word has the I before the E.
Ceilling ; This spelling has an extra letter L. Try again.

Interested in following the progress of development on this project? Join the Google group.