Optimal audio-visual representations for illiterate users

We present research leading toward an understanding of the optimal audio-visual representation for illustrating concepts for illiterate and semi-literate users of computers. In our user study, which to our knowledge is the first of its kind, we presented each of 13 different health symptoms to 200 illiterate subjects in one representation randomly selected among the following ten: text, static drawings, static photographs, hand-drawn animations, and video, each with and without voice annotation. The goal was to see how comprehensible these representation types were for an illiterate audience. We used a methodology for generating each of the representations tested in a way that fairly stacks one representational type against the others. Our main results are that (1) richer information is not necessarily better understood overall; (2) voice annotation generally helps in speed of comprehension, but bimodal audio-visual information can be confusing for the target population; (3) the relative value of dynamic imagery versus static imagery depends on other factors. Analysis of these statistically significant results and additional detailed results are also provided.