Space perception for the blind by means of acoustic images

Abstract

The aim of the project was to design an electronic travel aid for the blind which would convert stereovision images into sound codes representing scene obstacles. Various methods of sensory substitution have already been implemented in a number of electronic travel aids (ETA);  however, their main drawback (as indicated by the feedback from the visually impaired) is the problem of overloading the auditory channel with too rich and irrelevant acoustic information.

The key concept of the ETA developed within this research grant is to segment out the nearest scene objects, describe them with a limited number of parameters (e.g. distance, location, size, type) and use the simplified description for sonification (Fig. 1). This is in accordance with the theories concerning auditory scene perception, which state that humans can perceive and derive useful information from only a limited number (3-4) of sound streams. Such an assumption required application of advanced image segmentation methods and tracking of scene elements through the stereovision image sequence.

Fig. 1 The concept of an auditory display system that converts stereovision image sequences into sound streams representing individual scene obstacles

The system (currently a portable PC-based version) comprises four modules:

The constructed stereovison system generates depth map sequences of scene objects at approx. 6 frames/per second. Image segmentation algorithm subdivide the scene into planes (walls, ground surface) and other objects that are considered obstacles. Special sound synthesis schemes have been worked out to prepare auditory icons reflecting geometric properties of scene elements, e.g. the larger are the real dimensions of an obstacle the lower the fundamental frequency linked to the virtual source representing it.
 
The location of the obstacles is acoustically rendered by using a special sound filtering technique utilizing head related transfer functions (HRTF) that allows localization of sounds in an arbitrary location in space by simulating the reflections and shadowing introduced by the head and ear pinnae. A special HRTF measurement system was built since sound filtering by an outer ear is a highly individual feature in humans.

Questionnaires with 20 blind persons were carried out (in cooperation with the Polish Blind Union) and 9 were chosen for system trials. The trial participants opted for at least two sound features to code distance to objects (e.g. pitch and loudness). With the use of the system they were capable of perceiving obstacles with approx. precision of 7? in azimuth and 11? in elevation.

A separate part of the project was to prepare software for Symbian OS mobile smartphones to make them more usable for blind users. The application was written in the Carbide C++ environment and tested both on the S60 series phone emulator and the phone platform itself (Nokia 6600). A specially designed speech enabled menu was implemented to provide better performance than simple screen readers. Along with ordinary phone functions (calls, SMSs) the programmed phone can be used as a speech recorder, a web browser (RSS feeds are used), and a colour recognizer in images captured by the phone’s camera. Added functionality makes the smartphone a very useful personal assistance tool for the blind. This application is further developed in cooperation with the Polish Telecom (TPSA).