From Discover Magazine’s Mind & Brain blogs, Carl Zimmer summarizes research that highlights how the brain uses multi-sensory processing to construct perception.
The whole article is worth reading, but I was especially drawn to the opening paragraph, in which the author describes the mundane incident that inspired him to write about it. He uses the anecdote to elucidate how our auditory and visual senses are co-dependent, but by coincidence, it’s the same analogy I often employ to help parents understand how poorly integrated sensory processing impacts on their child’s ability to stay emotionally regulated and to make sense of the world around them:
“I don’t usually stream Netflix onto my television to probe the
inner workings of my mind, but it had that effect not long ago.
While I was catching an old episode ofLaw & Order: Criminal Intent, the actors’ voices lagged a fraction of a second behind the movement
of their mouths, making me so disoriented it completely ruined the show. Soon my irritation turned to puzzlement, and some self-observation allowed me to track my frustration to a precise source. I didn’t care that the ominous soundtrack rose half a second late when Vincent D’Onofrio and Kathryn Erbe crept into the subway tunnel where they
were about to find a body. I didn’t care that the show’s trademark duh-dung! sound marking a new scene was still duh-dung-ing after the scene started. It was only when people talked that I went batty. I would watch the characters speak, and then I’d switch to listening to them, and then I’d watch them speak again. I just couldn’t meld the two streams of information in my head.”
I use this example because the response is so universal… the mere suggestion usually inspires reflexive cringing and groaning. I suspect that the emotional response we experience is akin to motion sickness- hypothesized to be a protective “sit down and reassess” mechanism to resolve the potentially dangerous mismatch between your vision and your vestibular system. It’s as if our brains are telling us, “something’s not right.”
“We don’t mix up our senses willy-nilly, however. There is a window of less than a tenth of a second in which a stimulus from one sense can affect the others. As my misadventure with Netflix showed, my brain was accustomed to balancing sight and sound to make sense of what people were saying without my even noticing, but the sound lag during that episode of Law and Order was so wide that the two sensory streams created confusion instead.”
The important distinction, however, is that we, with relatively intact processing systems, have the history, the memory of a prior experience in which (in this case) Law and Order proceeds in its usual, well-sychronized way. Comparing his present experience to a memory of a previous, similar event, the author’s emotional response of disorientation and confusion was able to give way to the more intellectual response of “irritation” and “puzzlement.” By contrast, children with poorly integrated auditory/visual processing lack a strong database of prior experience typically selectively attend to the stronger or more salient sense, missing the supplemental input that would allow them to perceive the full spectrum of the event. This has significant implications for their ability to attend to events in the environment, including social interactions and opportunities for incidental learning.
The article goes on to describe several illusions which illustrate how the individual modalities conspire to create a new, unified perception, the most famous of which is the McGurk Effect, in which observing sound being articulated influences the processing of the auditory information (i.e. if you see “ga” and hear “ba”, you percieve “da”). Given the high degree of influence, is it any wonder that these children tend to acquire language less efficiently?
The second aspect of multi-sensory processing explored here is how our proproceptive and tactile body scheme senses are influenced by the other senses:
“Sight and sound are not the only senses we mingle in our brains. What we touch can affect what we see or hear. Our very understanding of the shape of our own body can be informed not just directly, through our eyes, but also by the pressure of our feet on the ground, the stretch of ligaments in our shoulders, and the wiggle of balance-sensing nerve hairs in our inner ears. Together, our senses let us control our bodies, keeping us from falling over every time we stand up.
“But this much integration comes with an astonishing ability to be duped. In 1998 Matthew Botvinick and Jonathan Cohen, two psychologists then at the University of Pittsburgh, found they could make people feel as if a rubber hand were really their own. All they had to do was put a rubber hand in front of their subjects and have them put their real hand behind a screen. The scientists simultaneously began to stroke the real hand and the fake one with paintbrushes. In a matter of seconds, people reported that the rubber hand felt as if it were part of their own body and that they even felt it being stroked.”
Again, these illusions wonderfully bring to light the normally invisible process of integration, but what about out-of-synch sensory systems? The analogy I often invoke to help parents understand body awareness is driving, or more specifically, parking. Having recently retired the 1997 Buick I inherited from my grandmother for a much more compact and responsive car, this phenomenon was fresh in my mind… My mind had created a mental “map” of the dimensions of the car that allowed me to navigate through space and had been calibrated to calculate the relative responsiveness of the steering wheel, brakes, suspension, etc. to do this accurately. Though the new car is unquestionably easier to park and drive, it took weeks to adjust my driving to match the new car’s body map. It took weeks for this process to turn from a conscious, effortful process to an automatic one. For a child with poor body awareness and postural control, I suspect that moving about in the environment is akin to attempting to parallel park an 18-wheeler when you’re accustomed to driving a subcompact. Physically, it’s difficult to master the mechanical maneuverings required to feel successful, and emotionally, well… stressful seems a woefully inadequate descriptor. Add to this challenge a visual perceptual processing deficit, and suddenly, it’s as if you’re attempting to do this with fogged mirrors and windows…
“The tricks we use to integrate our senses take time to develop. As children grow up, they get better and better at combining sights and sounds. When scientists compare children of the same age, they discover a fascinating pattern: The ones who are better at combining sights and sounds tend to score higher on intelligence tests. It’s possible, some scientists suggest, that helping children combine their senses through training exercises will enable them to do better in school. ”
And so, this is what we do; we help to calibrate our interactions with the child to help them attend with greater consistency and efficiency and provide the multi-modal input that helps their processing system mature.
Read the full article, with links to great videos, here.