The role of semantic congruency in multisensory processing

Supervisors: Thomas Otto, Ines Jentzsch

Project Description:
This project aims to understand how semantic content affects the brain functions that are involved in the processing of multisensory signals. It is typically found that humans react faster to multisensory signals as compared to the unisensory components, which is known as the redundant signals effect (RSE). Several studies have shown that the RSE occurs if signals are congruent, that is, if a signal in one modality (e.g., a dog image) matches a signal in another modality (e.g., a barking sound). From a theoretical perspective, such findings have contributed to the view that congruent multisensory signals are integrated to create a coherent multisensory percept based on learned semantic concepts (e.g., dog). Yet, we have only very little understanding to what extent the RSE depends on the congruency of signals. If the RSE occurs also for incongruent signals (e.g., a dog image paired with a meowing sound), as we have found in a pilot experiment, an alternative processing model would assume that arbitrary signals are flexibly combined according to short-term task demands. According to this perspective, multisensory signals are not integrated but processed in parallel.

In the proposed PhD project, to distinguish between these opposing views, we will systematically investigate the role of semantic content on the RSE as well as on related effects in cognitive control including event history effects and expectancy in decision making. The project will follow an interdisciplinary approach by means of both behavioural measures and electroencephalography (EEG). On a first level, we will use a new modelling approach to analyse behavioural data, which allows to quantify specific processing interactions with multisensory signals (Otto lab). The mathematical modelling approach will then be used, on a second level, to inform the analysis of EEG recordings to gain understanding of the underlying brain functions (Jentzsch lab).

The PhD candidate in the project will develop a thorough understanding of brain functions involved in multisensory processing, fundamentals of EEG methodology, and best-practices in experimental design. The project provides training in several techniques including programming (e.g., Matlab), statistical analysis, and computational modelling, which easily lend themselves to further scientific studies and/or more applied approaches both in industry and academia. The PhD project is suitable for students in Neurosciences and related disciplines including Psychology, Biology, Physics, and Computer Sciences. The successful candidate will have research interests in human sensory processes as well as brain functions underlying perceptual decision making and action control. Prior experience in EEG and/or computational modelling is a plus but not a requirement as training in both disciplines will be provided.