EEG Measurements in Sensory Substitution Systems for Visually Impaired Users
EEG Measurements in Sensory Substitution Systems for Visually Impaired Users
The main objective of the computer vision based sensory substitution systems designed for visually impaired users is to provide the representation of their surroundings with the use of alternative senses. After processing and encoding, visual information is conveyed to the user via one of the intact senses, rather than through the regular visual pathway which is highly degraded or interrupted.
From the technical point of view, one of the biggest and most obvious challenges, both on the hardware and on the software side of the problem, is to ensure the usability of such system in any indoor and outdoor illumination conditions. It must be however remembered, that the way in which the stimulation signals are designed, how they are delivered to the user (for e.g. which senses are utilized), what amount of additional cognitive load is introduced with the developed assistive device and whether it can still be accepted by the user, are much less obvious factors, but eventually have a huge impact on the overall performance of the system.
- Methods
The main goal of conducted experiments is to use the parameters and features of the extracted Event Related Potentials (ERPs) related to external auditory and tactile stimulation events, measure their parameters and features in a series of cognitive experiments and confront those measurements with individual preferences of the visually impaired (VIP) and normally sighted (NSP) users.
Cognitive experiments were designed as Python scripts with the use of the PsychoPy tool. Participants were guided through consecutive parts of the study with pre-recorded instructions generated with IVONA Text-to-Speech synthesizer which is well known and often used by the visually impaired users.
Artifact removal and data processing algorithms were implemented in Matlab with the use of additional, open-source EEGLAB and ERPLAB Toolboxes. Parameters that were measured for all extracted event related potentials were local peak amplitude, local peak latency and mean amplitude. ERPs were averaged in time domain within each channel, plotted and mapped onto the scalp.
- Results
The questions stated at the very beginning of the research were whether auditory and tactile sensory stimulation models established in the SoV project will evoke measurable ERP responses, how the parameters of those responses can be defined and confronted with individual preferences of the visually impaired and normally sighted users and ultimately, whether the qualitative and quantitative measure can be established on top of the parameters of isolated responses.
Local peak latencies of auditory ERPs are the lowest. Event related potentials appear on average 450 ms after audio stimulation events in case of both VIP and NSP users. They were recorded for all VIP users and almost all NSPs (weak ERP responses, on the level of noise and unrelated, background brain activity were measured in case of some users). Strongest and most distinctive (with the largest peak and mean amplitudes) auditory event related potentials were measured for three VIPs and two NSPs.
A kind of unexpected discovery is how two modalities (auditory and tactile representing the same obstacle in a virtual scene, generated and conveyed to the user at the same time) affect recorded ERP responses. In theory both modalities should complement, cooperate and provide information about different features (like distance, width, height, elevation and azimuth) of the obstacle. But, in questionnaires and surveys filled after EEG recordings, many of the users who were of different stages of the training process clearly preferred one of the modalities. They could not observe or utilize benefits of their coexistence.
- Conclusions
It was found that ERP responses could be recorded for of all of visually impaired and normally sighted users who took part in the research. Responses were weak in some cases, especially for complex stimulation patterns. For simpler scene representation models responses are strong, clean and easy to parameterize and to compare against each other.
During the research a number of problems were discovered. Due to nystagmus (rapid, involuntary eyes movements), signal quality was poor in case of two visually impaired users (thus longer and cumbersome recording sessions were necessary). Those artifacts often corrupted stimulation onset events and such epochs had to be eliminated from further analysis as no valid ERP responses could be extracted.
When isolated ERP responses are parameterized and compared with individual preferences of VIP and NSP users that were collected in surveys, clear correlations can be observed. Still it must be carefully studied how both audio and tactile modalities interact in case of the uses who recently started the training process and test takers who already became well trained. In theory both modalities should cooperate, but the users has to adopt to the cognitive load introduced by the system when both audio and tactile stimulations are active. The training process and its effect on the recorded brain responses in various sensory substitution system usage scenarios must be further investigated.
- Contact
- dr. Pawel Poryzala
, e-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.
- dr. Pawel Poryzala
- Publication
- P. Poryzala, "EEG Measures of Auditory and Tactile Stimulations in Computer Vision Based Sensory Substitution System for Visually Impaired Users," 2018 11th International Conference on Human System Interaction (HSI), Gdansk, 2018, pp. 200-206, doi: 10.1109/HSI.2018.8431353.
