Interactive Sonification of U-depth Images in a Navigation Aid for the Visually Impaired

Fig. 1 Hardware components of the ETA for interactive sonication of 3D scenes

Introduction

The visually impaired indicate limited mobility as one of the main problems affecting almost all activities of daily living. In this work we demonstrate how the technique of interactive sonifcation can be applied in a simple travel aid for the blind. We use the so-called depth images and their histograms termed "U-depth" which are simpler for auditory representation of the environment to the blind user.

  • Methods

    The system hardware consists of the Structure Sensor depth camera, a smartphone with Android OS and a pair of open in-ear headphones as shown in Fig. 1. The depth maps of the scenes pre-processed on the Android-based platform. Fig. 2 illustrates an example image of a scene (a) and the processing steps leading to computing the corresponding depth map (b), the depth map with removed ground region (c), and the U-depth map (d). The U-depth map can be understood as a top view of the scene with highlighted obstacles. The content of this map is then sonified to the user. The closer the obstacle the higher the frequency (in the range of 400Hz-4000Hz) of the generated harmonic audio signal. The user can interactively select the U-depth map region for sonification and activate a verbal mode that reads out the distance to the indicated object.

    The developed electronic travel aid (ETA) was tested by three blind individuals in an indoor environment. The task of the volunteers was to use the system in typical navigation scenarios i.e., walking along a corridor, avoiding obstacles and locating an open space, e.g. an open door.


  • Results

    After completing the trials the participants filled in the System Usability Scale (SUS) questionnaire to evaluate how useful the system was in helping the blind testers successfully perform the trial tasks. The scores in the SUS scale were considerably high, i.e. 68%, 73% and 95% (only one box hit was noted during the trials). The testers indicated the system was simple in handling and it does not require long training to use it efficiently. Interestingly the best system usability score was given by the tester who is the congenitally blind.

  • Conclusions

    We show that a small size factor system (consisting of a depth camera and a smartphone) can serve as an efficient travel aid for the visually impaired. The usefulness of the system was verified by blind testers in indoor mobility trials. We have received a valuable feedback from the first users and currently we are working on further improvement of this prototype ETA. Demo movie of the proposed system can be accessed at: http://eletel.p.lodz.pl/pskul/u_depth_sonification/.

  • Contact

    • Piotr Skulimowski , e-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.

  • Publication

    • Skulimowski P., Owczarek M., Radecki A., Bujacz M., Rzeszotarski D., Strumillo P., Interactive Sonification of U-depth Images in a Navigation Aid for the Visually Impaired, Journal on Multimodal User Interfaces, 2018, DOI: 10.1007/s12193-018-0281-3