Zum Inhalt springen

Podcast episode 54: Neil Cohn on Speaking in Pictures

31.01.2026 · 38 Min.

Shownotes

In this interview, Neil Cohn tells us about his theory that visual art and spoken language draw on the same underlying cognitive abilities.

Download | Spotify | Apple Podcasts | YouTube

References for Episode 54

Cohn, Neil, and Joost Schilperoord. 2025. A Multimodal Language Faculty: A cognitive framework for communication. London: Bloomsbury Academic.

Cohn, Neil. 2026. Speaking in Pictures: A vision of language. London: Bloomsbury Academic. https://www.bloomsbury.com/us/speaking-in-pictures-9781350402164/

Lerdahl, Fred, and Ray S. Jackendoff. 1985. A Generative Theory of Tonal Music. Cambridge, Mass.: MIT Publishing.

Transcript by Luca Dinu

JMc: Hi, I’m James McElvenny, and you’re listening to the History and Philosophy of the Language Sciences podcast, online at hiphilangsci.net. [00:09] There you can find links and references to all the literature we discuss. [00:13] Today we’re talking to Neil Cohn, who’s a Professor of Cognitive Science at the University of Tilburg in the Netherlands. [00:20] Neil’s developed a fascinating theory that visual art draws on the same cognitive faculties as human language. [00:28] He’s set out this theory in the book A Multimodal Language Faculty, co-authored with Joost Schilperood, published in 2024 with Bloomsbury Academic. [00:38] He’s going to present these same ideas in graphic novel form in the book Speaking in Pictures: A Vision of Language, which will appear in February 2026, also with Bloomsbury Academic. [00:50] So, Neil, could you outline your theory for us? [00:53] What are the links between visual art and language? [00:57]

NC: Well, first, thank you for having me. [00:59] I’m a big fan of the podcast, so it’s a pleasure to be on. [01:02] The way I often like to start is by kind of at a basis of human communication cognition, which is that human beings as a species have only three ways in which we can express our concepts: [01:14] We can create sounds (which is what I’m doing right now), we can move our bodies (which is also what I’m doing right now, but people won’t be able to see that), and we can draw things. [01:26] And that’s really all humans do in terms of our ability to express concepts overtly. [01:33] We don’t emit smells to each other. [01:35] We don’t secrete things and lick each other to communicate. [01:40] We don’t emit, say, light pulses or something. [01:43] We do these three things. [01:45] Given that, these three communicative faculties, why would we believe that they should be different from each other is, I guess, the basic foundational idea. [01:57] So given that they come from the same brain, why would we assume that they have different cognitive building blocks? [02:03] Certainly in the last, well, over a century, people have been interested in how the body conveys meaning. [02:11] As you know, I’m sure, people in the 1800s debated whether or not to actually include sign languages as part of the definitions of language and ultimately said, “Now we’re going to ignore those for now,” and then, of course, there was a resurgence in the structure of studying signed languages, to which we now believe that they share the same structural properties as spoken languages. [02:33] And I would argue that we should be adding one more modality to that, which is graphic representations. [02:39] Given that it comes from the same brain, they should have the same cognitive building blocks and structural properties. [02:46] So this follows from what I call generally the principle of equivalence, in a very lofty-sounding thing, which is that given that we have one brain, we should expect the mind and brain to treat all modalities in a similar way, given modality-specific constraints. [03:03] So the places where they should be different is motivated by the nature of the type of modality that they’re addressing, because sound and light are different things, and so we should imagine that that will yield different types of structure down the road, but it’s all still coming from the same brain. [03:20] So this is the path that I pursued. [03:22] I think to some degree, when people hear what I’m working on, they think I’m making an analogy, which is to say, graphics or pictures or drawing is like language, and that language is like a source domain for the metaphor of this mapping. [03:43] And that’s, I believe, you know, I’m not the first person who’s made these sorts of comparisons. [03:47] I believe that sort of analogic mapping is what people have done throughout the, say, 20th century in movements of structural semiotics, applying, you know, principles of linguistic structuralism to graphics. [04:02] You know, they would be searching for the whatever minimal units of emes there are, graphemes, logemes, whatever it is, and it would largely be taken as kind of the linguistic structuralism metaphor of mapping, “OK, here’s what’s happening in what we know as language to these other domains.” [04:21] And that’s not ultimately my argument, is, my argument is kind of to say that the human brain — and/or mind, however you want to think about the granularity — uses similar principles and structures and mechanisms that operate across modalities, no matter what those are. [04:43] So that puts spoken languages and, say, drawn visual languages on equal footing as both involving something more general, rather than it being an analogic comparison. [04:57] And to that extent, I actually don’t think that they are in competition with each other in any sort of sense, because ultimately, those three modalities that we have are complementary parts of one larger whole system, and ultimately, the language faculty or communicative faculty, or however you want to describe it, is multimodal across these three parts, [05:23] out of which there are emergent properties of different behaviors that share the same building blocks, but it’s ultimately a multimodal system, not a unimodal or amodal system. [05:34]

JMc: OK, and why do you think that there are these three specific modalities? [05:39] I mean, what about the case of, you know, like a famous case like, you know, Helen Keller with, who doesn’t have physical access to any of those modalities? [05:47]

NC: Sure. So I would first say that I think modality, the notion of a modality is important here, and the way we define the notion of a modality is, it’s an interaction between a sensory system and a cognitive correlate, abstract representation. [06:02] So in the canonical case of speech, we have sound is the sensory system, which we then have dedicated receptors to receiving through our ears and crucially producing through our mouths. [06:17] We then, in addition to that, have a corresponding cognitive correlate of a phonological system, which is… abstracts across the tokens of the individual sounds, right? [06:31] And this interplay is crucial, which is to say that a modality is not merely a sensory system—it’s an interaction between sensory systems that involve both input and output and abstractable cognitive correlates. [06:48] So in the case of, say, the body, that would be to say we have sensory systems of touch and vision involved in receiving those signals and also a corresponding, let’s call it phonology for the sake of ease for right now, and that abstracts across the individual tokens of those expressions. [07:11] And again, we’re not even talking about anything about meaning right now. [07:13] We’re just talking about individual expressions. [07:16] I should… And correspondingly, I would say that the graphics are the same. [07:19] So you have an experience in light that is then abstractable into, you know, basic, I don’t know if we’d want to call it phonological anymore, graphological cognitive correlates. [07:34] Now, part of that is also that all of our modalities are pretty, are multisensory, so none of these modalities are guided by the singular sensory system alone, which is why the McGurk effect exists, which is where you might be producing sounds with the mouth, but then seeing different articulations than the actual sounds, and you, your brain instead perceives it as being produced a different sound than what you’re actually hearing because of what you see. [08:10] So speech alone is not unisensory; it’s multisensory, and vision is actually involved in all of them. [08:19] So that makes sensory systems and vision in particular a poor characterization of modalities. [08:27] The production system is much more, a much stronger characterization of the nature of these things. [08:35] So you don’t produce speech sounds through anything but your mouth, right? [08:40] Our bodies move around to… with different articulation points to be expressive, primarily hands and faces. [08:47] You can also use body orientation and things like that as well, and graphic systems use various types of traces left in various types of materials to correspond to meaning, so it’s not just a sensory system. [09:03] And on the case of, say, Helen Keller, what you’re left with is a, the absence of receptive systems — that is, through the eyes or the ear —, and so you fall back on a different type of sensory aspect of the bodily production system, which is less used, but at least provides a point of access for that production system of the body. [09:30] If you look at children who are developing naturally, or typically developing, we should say, those typically developing children will naturally produce in these three ways. [09:43] They produce sounds, they produce bodily motions, and they will produce graphics without prompting, just through exposure and practice, and the onset of those expressions largely occurs around the same time in childhood, roughly eight months, plus or minus, given that children are different, right, and are exposed to different things. [10:09] So they generally will start doing things like pointing at the same time as they start babbling, maybe manually babbling if they’ve been exposed to signed languages, and they will produce graphic representations around that time. [10:26] By graphics, I mean smearing milk to make lines, or… in ways that they recognize that they are making a representation out of something that’s not merely just, “I’m making a mess,” but maybe, you know, “I’m going to create some sort of trace that is recognized as a trace.” [10:45]

JMc: Yeah. [10:47]

NC: So those are why I would say those are the three primary things. [10:50] And if you follow the developmental trajectories, children will develop, maybe to varying degrees of complexity, given the exposure and practice across all those three forms, [11:02] but not something like writing, right, where that requires explicit instruction to understand, “Oh, this graphic representation corresponds to this sound,” [11:16] in which case you’re creating a cross-modal representation across the existing modalities that are already there, [11:23] but with a new mapping that isn’t part of our innate capacity to have that correspondence, [11:31] which is why it’s an invention, because… [11:33] But it’s an invention that makes use of existing pathways that are purposefully there for drawing, and purposefully there for speaking. [11:41] It’s just that we fuse them together to make a very culturally useful tool, but it’s not part of our biological, you know, endowment, let’s say. [11:50]

JMc: Yeah. But what about something like music, you know, which is also auditory, but is also structured? [11:58]

NC: That’s right. So music is also seemingly an innate capacity. [12:03] I should say I am experiencing developmental things quite a lot. [12:06] I have a two-and-a-half-year-old who does draw, and he also loves singing, and making what one could construe as music, [laughs] if you are generous, has the development of tonality, at least, in singing some things. [12:26] So there definitely is an innate capacity of music, but music does not necessarily correspond to conceptual representations in the same way that drawing, and speaking, and gesticulating does, but I would argue music does share the same suite of cognitive capacities. [12:47] As you said, it’s structured and organized, but in this case, those structures do not necessarily correspond to meaning in a compositional way. [12:56] So they, you can certainly have correspondences between music and meaning, like that the Jaws theme we’ve associated with some sort of impending doom, right? Or, you know, movie references… [13:10] The Star Wars Imperial March gives you a sense of authoritarianism. [13:14] So we’ve clearly built up some correspondences between music and meaning, but we don’t do, “This set of notes and this set of notes each have independent meanings, and then we will compositionally, combinatorially create higher-order meaning out of them,” but we do do that with the other modalities. [13:36] So we have different classes. [13:38] What we describe is one of the challenges that I think the language sciences has faced over the last centuries is, you know, one, defining what language is, and usually looks to things like various features of… lists of features that are there or not there. [13:59] If language is a cognitive phenomena, I would say you need to define it by its cognitive building blocks, and part of the problem that you have when making these sorts of comparisons, say, when I say that there’s visual languages that are structured of graphics that are the same as other sorts of languages, then people come back and say, “Well, that can’t be language, because it’s graphic, and by principle, I can’t allow it to be language.” [14:26] But the structural properties are still there, and so are the cognitive principles that are seemingly the same across them. [14:33] So what do you do? [14:34] And mostly the challenge has been something like that we’re only allowed a binary, that, “Well, there are languages, and there are not-languages, and this is the options categorically that we have.” [14:50] So what do you do with things that maybe have the same properties as languages, but not fully? [14:55] So let’s say music is an example. [14:57] Like, for 40 years now, since Lerdahl and Jackendoff’s book, the Generative Theory of Tonal Music came out, [15:05] they’ve argued that there’s these hierarchical combinatorial structures in music, and corresponding cognitive research has shown that you mostly elicit the same sorts of brain responses as in sentence structure from manipulating sequences of music. [15:22] Now, it’s an ongoing debate to the degree to which the same patch of neurons might be used in both, but there at least seems to be similar mechanisms at the least, and this is work that my lab is also involved in. [15:36] Now, what do you do where you have… [15:37] Well, this thing is clearly language-like, let’s call it, but there’s clearly also facets that are not, like compositionality of meaning, right? [15:46] So how do you then want to class music? [15:48] Do you want to say it’s the language of music? [15:50] What do you do with, say, emoji, which my research and others have shown, well, there’s clearly systematic lexicons, but doesn’t really seem to have a grammar to sequencing of emoji. [16:01] OK, well, now what do you do with that, right? [16:04] Or what do you do with co-speech gesture, which clearly is meaningful, has a tight relationship with spoken languages, but do you want to say that it’s part of language, separate from language? [16:17] How do you ontologically classify these things? [16:20] So our approach has been to say that the existing categories structures are not helpful, and using a word like ‘language’ as a scientific term is actually problematic, because it’s a social term also that is also value-laden. [16:41] So we need a different set of categories, and the ones we arrive at are based on the component parts of linguistic structure, of a modality, a grammar, and a meaning, or in traditional terms… phonology, syntax, and semantics, or whatever set of terms you want to use. [17:00] And so based on the relative allocation of those sorts of components, you can derive a classification system, so we call an omnia is a system that has all of them. [17:10] So spoken languages, sign languages, and what I call visual languages, are all omnia, but there’s also what we call semia. [17:18] Semia use, let’s say, phonology and semantics, but not necessarily a robust syntax, or a robust grammatical system. [17:31] So gestures are a semia, because they don’t use a grammar of their own. [17:36] Sign languages do, which is why they are an omnia. [17:39] And emoji do not; they don’t have a grammar, but they are meaningful, and expressive, and in the graphic form. [17:44] So those are also semia, which is why emoji have been compared to gestures, because they are structurally similar in that way. [17:52] Music are on the flip side, so music are the inverted side, so they have a modality and a grammar, but not necessarily meaning in a compositional way, and so those are what we call sequentia. [18:05] And that might also include something like the combinatorics that are involved in athletic things, like, say, martial art forms, or synchronized swimming, or these other things that are very clearly pattern-structured sequences, but are not, they don’t have some additional referent. [18:25] They are sequenced for themselves, their own sequencing, right? [18:28] It’s not like a punch in a martial art form is then a referent for some other concept. [18:35] It’s just a punch, right? [18:37] But it is combinatoric. [18:38]

JMc: OK. [18:39] So it seems, it seems that the key thing is ‘meaning’, in inverted commas, and the idea that meaning can be compositional and referential. [18:47] Is that, have I understood that correctly? [18:49]

NC: And has a combinatorial system. [18:51]

JMc: OK. [18:51]

NC: So it has to have all three.

JMc: Yeah. [18:53]

NC: For the way I use the term ‘language,’ usually, it’s the equivalent of what I would call an omnia, which is then, it has to have some modality of expression, has to have a complex combinatorial system that is capable of hierarchic embedding, center embedding, anaphoric relationships, structural ambiguities, etc., and there needs to be some sort of compositional conceptual structure or meaning that is associated with that behavior. [19:21] Semia lack the complex combinatorics. [19:26] However

Alle Folgen von History and Philosophy of the Language Sciences

Einstellungen

Darstellung

Sprache

Bestimmt, welche Sender dir auf der Startseite zuerst gezeigt werden.

Verlauf

Zuletzt gehörte Sender und Folgen und deine letzten Suchen liegen nur in diesem Browser.

Hinweise

Ein kurzer Hinweis, wenn ein Podcast, dem du folgst, eine neue Folge hat. Einzelne Podcasts stellst du über die Glocke auf ihrer Seite ab.

Konto

Favoriten speichern – in der App und auf der Website.

Tastenkürzel

/
Suche öffnen
Leertaste
Abspielen oder anhalten
← →
Im Player: 15 s zurück, 30 s vor (Folgen) · im Regal: nächste Kachel
↑ ↓
Im Regal: Ziel innerhalb der Kachel
Pos1 / Ende
Im Regal: erste bzw. letzte Kachel
Esc
Fenster oder Menü schließen
?
Diese Übersicht
Läuft gerade

Zur Senderseite

Als Nächstes

    Folgen mit „Als Nächstes“ vormerken – sie laufen nach der aktuellen Folge.