Design as Communication: Sound-Images and Affordances

The Human-Environment interaction can be conceptualized as a dynamic system governed by three principles:

1. Communication.
2. Self-regulation.
3. Adaptation.

We focus on communication here. No integrity of biological and social lives can be maintained in the absence of communication. We constantly gather information about our inner state (microscopic level) and our environment (macroscopic level) in conscious and unconscious ways. On a macroscopic level, we communicate by processes of information production and reception. Information might be encrypted in a form of language (speech and voice), writings, signs, gestures (non-verbal communication) and symbols (artifacts). A process or an act of communication is always a two-way process: there is always a receptor and effector, sender and receiver, designer and user. Thus, design is a form of communication, most readily, the language (or vessel) through which we communicate. However, this language is too often misinterpreted.

Intuitive Design

We all make decisions instinctively, relying on a feeling of ‘knowing’ what decision to make. This kind of decision making prevails especially in uncertainty and in conditions of extreme time pressure. Such ‘spontaneous judgment’ is called intuition.

Intuitive design is a series of subjective understandings and actions, relying on ‘this feels right’ or ‘this is natural’ to justify design decisions. Intuitive design, defined, “is a process of intentional self-leadership that both inspires imagination and informs how individuals interact in the world.”¹

Still, the concept of intuition means different things to different people; it has been discussed by philosophers for centuries, and has been described as a reflexive and inner kind of knowing. Others have proposed “lists of the features of intuition, including such attributes as holistic, associative, fast, and automatic,” an increasingly popular view is the ‘learning perspective’; that intuition “relies on mental representations constructed on prior experiences and it is assumed to capitalize on these stored representations to often provide direct access to the criterion to be judged.”²

Intuitive solutions take precedence in design professions compared to evidence-based disciplines.³ Designers are inclined to display particular cognitive styles and ways in which their world is sensed.⁴ Cognitive differences between designers and other professionals have been observed in divergent vs. convergent thinking, synthesis vs. analysis and subjective vs. objective thinking. In engineering, for example, convergence is preferable for problem solving.⁵

Acknowledging the designer’s need to rely on intuition is beneficial, however, we cannot solely rely on intuition to justify design decisions because intuition isn’t always enough to empathise with the user. One way to circumvent these shortcomings is with affordances which provide the bridge between designer and user.

Sound images and Affordances

The implication of utilizing affordances as a ‘standard language’ is tested every day through sound-images. These ‘images’, or rather, the very essential basis of language, are clear evidence that a common language which respects subjective and objective experience, interpretation, and opinion, is necessary to foster communication. Saussure writes,

The linguistic sign unites, not a thing and a name, but a concept and a sound-image. The latter is not the material sound, purely physical thing, but the psychological imprint of the sound, the impression that it makes on our senses.⁶

The sound image here will be used as a metaphor for affordances, and we consider it to be a yet unexplored metaphorical paradigm of why a common language (in the shape of affordances) between designers, scientists, and users is necessary.

The term ‘Affordance’ was coined by James Gibson, and he provided his most expansive definition in his 1979 book, The Ecological Approach to Visual Perception.

The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill. (…) As an affordance of support for a species of animal, however, they have to be measured relative to the animal. They are unique for that animal. They are not just abstract properties. They have unity relative to the posture and behavior of the animal being considered.⁷

Gibson uses the example of the ecological niche to further his point about affordances: that they (both affordances and niches) are “an invariant combination of variables.”⁸ Gibson’s definition is rooted in the holistic perception of the object, distinguished from all of its components.⁹ He also posits that perception is purely visual; that that is enough to determine an affordance (relative to the user).¹⁰ This is problematic because according to contemporary neuroscientific models perception is an interpretative top-down and bottom-up process that recognizes and organizes multisensory information to formulate a mental representation of an environment.¹¹, ¹²

In his 1988 book The Design of Everyday Things, Donald Norman famously incorporated the concept of affordances as part of a conceptual design model to help guide the design process.¹³ He saw affordances as communicative user guides that were inherent within the object and were understood without “pictures, labels, or instructions.”¹⁴ Usage of directions to explain ‘simple’ things mean the design was a failure.¹⁵ Essentially, this contested Gibson’s understanding of a hegemonic visual perception and reappropriated it to more accurately use the “experiences of the observer” rather than “referencing an observer through an object,” as Gibson did. ¹⁶ This made affordances less transactional, as Norman also accounted for socio-cultural constraints of affordances; meaning, affordances could not be called truly subjective as the cultural and thus behavioral and social norms influenced each other cyclically, impacting the ‘common’ affordance perception. This is furthered by the notion of affordances as ‘embodied perception’ and ‘embodied concepts’, as sensory-motor analyses of environment are universal among humans. Yet simultaneously, the evaluation of an affordance remained subjective to the user (within the behavioral and sociocultural sphere).

If we utilize a ‘designer only perspective’, we fall prey to disconnecting from our users. Bryan Lawson in, How Designers Think? The Design Process Demystified writes,

The primary purpose of a greenhouse is clearly to trap heat from the sun, so we can begin by measuring or calculating the thermal efficiency of a whole range of possible greenhouses. Unfortunately, we are still some way from describing how satisfactory our greenhouse will appear to individual gardeners. They may well also want to know how much it will cost to buy, how long it will last, or how easy it will be to erect and maintain, and probably, what it will look like in the garden. The greenhouse then, must satisfy criteria of solar gain, cost, durability, ease of assembly, appearance and perhaps many others.¹⁷

We could consider the greenhouse as solely a functional entity; to say that it would be ‘successful’ as a design if it just efficiently traps heat, and we could only consider this quality of it. However, to do so, would be to disregard the utilization of the space in order to allow it to function. How the gardener feels inside of the building is important, as well as the other operational factors of the greenhouse which the gardener must use to successfully complete their tasks. A designer who is not a gardener will not understand the nuances that exist with the job, and this reinforces the need for a common language between the user and designer of a space.

This example drives the idea that design is a practice and a communication; it is reliant on responses to the design and thus, the usage of the design, and a communicative channel must exist to allow this understanding. There is a reciprocal determinism that exists between user and design: both shape each other, and as a result of that shaping, behaviour and usage is influenced:

basic-level concepts are defined by the convergence of (1) gestalt object perception (observed or imaged) and (2) motor programmes that define the prototypical interaction with the object (again, performed or imaged). Thus, a chair looks a certain way (and we can imagine how it looks) and it is used for sitting in a certain position (and we can imaging sitting that way).¹⁸

Let us draw a direct metaphor between design as communication and language: our ultimate form of communication. Let us specifically consider the Sound Image. The Sound Image, first defined by Ferdinand de Saussure, a Swiss Linguist, in his book Course in General Linguistics, is perhaps an essential paradigm of understanding why and how a common language operates.¹⁹

The characteristic role of language with respect to thought is not to create a material phonic means for expressing ideas but to serve as a link between thought and sound, under conditions that of necessity bring about the reciprocal delimitations of units. Thought, chaotic by nature, has to become ordered in the process of its decomposition. Neither are thoughts given material form nor are sounds transformed into mental entities; the somewhat mysterious fact is rather that “thought-sound” implies division, and that language works out its units while taking shape between two shapeless masses.²⁰

The link between ‘thought and sound’ that Saussure identifies is semiotic and morphologic, “that is to say, it is based on viewing the word as a meaning-making tool that unites a sound image and a concept, not a name and a thing.”²¹

Lions and Tigers and Chairs, Oh My!

Tiger.

You probably just imagined a tiger. How could you not have done so? Your mind just offered you an image of one. What was your tiger doing? Standing? Growling? Sleeping? Is it an image or a film? Is it an adult or a cub? How many stripes does it have? How many whiskers? How big are its teeth? Is it superimposed on a blank background or is it in a landscape scene? In a zoo? Roaming around a city? Is it alive or a graphic representation? Is it anthropomorphized? Is it exactly like the photograph above? (That’s a little unimaginative of you, isn’t it?)

I ask all these questions to simply say: my tiger is different from yours. The sound-image inevitably produces a different image for me than it does you, and this is a variant for any endless multitude of reasons (education, geographic location, culture, etc). However, we both know what a tiger is. We both agree on the general concept of a tiger. I don’t have to ask you to produce exactly what your mind offered you for us to have a conversation about tigers or to have a consensus on what they are. This is language and thus this is design, is it not? That shifting consensus of definitions and purposes, respectively, that are reciprocally deterministic, that carry across images or purposes to communicate. We practice language or a way of speaking, a way of creating a series of sound-images in logical formats to explain what we mean or imagine. We practice design through the usage of objects which are the actualized sound images of the designer.

Chair.

Again, my mental representation of a chair is different from yours. But we both know what the purpose of the chair is: to sit. How do we know what it feels like ‘to sit’? The answer here lies in the notion that affordances are also a form of embodied perception.

The simple idea of ‘chair’ as a sound-image, and by extension its linguistic understanding, is defined by its purpose (sitting) and by its practice (when someone sits on it, we can call it a chair as is evident by ‘non-chair’ objects – like a stack of books). However, even if we don’t enact an affordance, we are still able to recognize what its purpose is; a chair, for example, has much embedded symbolism. A throne, which few sit on, is still recognizable as a char. The essence of the chair carries through in the design just as the word ‘chair’ carries that representational significance of understanding that a chair affords the option to sit on it.

But what do we mean by ‘sitting’? I can sit cross-legged, with one leg up, both legs crossed on each other or by the ankles, or on my knees and any other number of variations. But I am still practicing sitting, and you agree that I am sitting. Interpretation of the object and the practice of it, thus, is both defined by itself and by its usage. It is both a subjective and objective practice, and it is important that it remains so.

Affordances as Embodied Perception
Affordances are not purely conceptual. The embodiment theory is important as it offers an explanation on why the world and its representations become meaningful to us. Our sensory and motor capacities, next to our sensorimotor knowledge, depend on more than just the workings of the brain and spinal cord; they also depend on the workings of other parts of the body, such as the sensory organs, the muscles, sensory organs, the musculoskeletal system, and relevant parts of the peripheral nervous system (e.g., sensory and motor nerves). ^22 23

Bruner, a cognitive psychologist who was concerned with how knowledge is represented and organized, assumed that all perceptual experience is the end product of a categorization process. However, Ulrich Neisser, often referred to as the father of cognitive psychology, has argued that some kinds of perceptual experience (such as visual tracking) may be more analogue than categorical. Nonetheless, many forms of perception are based on pattern recognition which is about classification of stimuli and categorization.²⁴

Eleanor Rosch represented the successive generation of cognitive psychologists and proposed that categorisation is embodied – given by our interactions – not just by objective properties of objects in the world, as a long philosophical tradition had assumed.²⁵

Without us—without the way we sit and the way we form images—the wide range of objects we have called “chairs” do not form a category. Our concepts must also be characterised relative to such a body-based understanding.

Gallese and Lakoff argue “that the sensory-motor system not only provides structure to conceptual content but also characterises the semantic content of concepts in terms of the way that we function with our bodies in the world.”²⁶

This discourse is based on the finding that imagining and doing use a shared neural substrate: canonical neurons.

The same neurons fire for a given manner of grasping as for merely observing an object that, if grasped, would require the same manner of grasping. For example, if a small object is presented, no matter what its shape is, then the same neurons fire as would fire if that small object were being picked up with a precision grip (as afforded by a small object of any shape)’. ‘(…) Thus, imagination is not separate in the brain from perception and action. (…) These data all together show that typical human cognitive activities such as visual and motor imagery, far from being of a disembodied, modality-free, and symbolic nature, make use of the activation of sensory-motor brain regions.²⁷

Our understanding of language depends on our ability to imagine objects, our actions, and other people’s actions.

Consider a simple sentence, like “Harry picked up the glass.” If you can’t imagine picking up a glass or seeing someone picking up a glass, then you can’t understand that sentence. Our hypothesis develops this fact one step further. It says that understanding is imagination, and that what you understand of a sentence in a context is the meaning of that sentence in that context.”²⁸

Our imagination of sensory-motor experiences is both embodied and enacted.²⁹ Our ability to imagine grasping apples makes use of the same neural substrate as performing and perceiving grasping. Thus, imagining is a reaction to simulation; a mental simulation of action or perception.

Language, similarly,

…externalises thoughts and encodes knowledge structures that represent the language users’ experience of their physical and socio-cultural surroundings. For example, the expressions ‘He is boiling with anger’ and ‘He is furious’ encode the conceptualisation of the experience of “anger” as “heat” and of the “body” as a “container” for emotions.³⁰

It’s proposed that the grounding process should consider the full environmental context. Furthermore, “affordance grounding,” is similar in stipulating that affordances ‘ground the agent’, which allows for it to adapt to new settings.³¹

Design affordances are analogues to sound-images; they are, largely, mental representations that result from embodied perception and unified understanding of purpose.³²

Conclusion
Affordances are an adolescent language. But just as language evolves with use, so will affordances if they become widely accepted and used. There is an example of this evolution: cognitive affordances. The immediate drawbacks of adopting affordances as a design language is that object affordances from the Gibsonian or Normanian perspective, while they start to take into account different users’ perception, don’t give a model for the evolving nature of affordances.The user is attached to the object, rather the other way around.

This is analogous to only being able to use the word tiger in one sentence:
The tiger ululated vociferously as its pulchritudinous prey was snatched by another, more sanguine feline.

A child reading this sentence will only understand some words: tiger, prey, maybe snatched. The rest of the sentence is lost to them, and thus, useless, as you cannot derive the meaning of the sentence from those three words alone. This is how the current understanding of object affordances operates with Gibson’s or Norman’s understanding.

If we evolve the language of design affordances to resemble something more amorphous, something like:
The tiger cried loudly as its beautiful prey was caught by another, more confident tiger.

Or even,
The tiger was sad because another tiger caught her dinner.

The meanings of the sentences are essentially the same. That is the power of language; we can make it accessible or we can make it inaccessible, language has the ability to flow and carry the same meaning to everyone or very few, and it has the flexibility to accommodate different levels of understanding, as design should be able to accommodate different levels of ability and understanding in how it communicates and is communicated about. Again, this does not abolish either the unintelligible or the simplified, but gives them their appropriate place to flourish as they need to. However, this comes with the evolution of language. It starts simply then grows more complex, just as the concept of affordances and its dictionary will grow and evolve as more come to use it.

---

References

• 1 Haakon Faste. (2017). Intuition in Design: Reflections on the Iterative Aesthetics of Form.
• 2 Ibid.
• 3  Durling, D. (1999) Intuition in Design. In Bulletin of 4th Asian Design Conference International Symposium on Design Science 1999, Nagaoka, Japan, October. ISBN 4-9980776-0-0 C3072
• 4 Ibid.
• 5 Ibid.
• 6 de Saussure, F. (1959). COURSE IN GENERAL LINGUISTICS. In W. Baskin (Trans.), Course in general linguistics. The Philosophical Library, Inc.
• 7 Gibson, J. J. The Theory of Affordances. The Ecological Approach to Visual Perception (pp. 127-128). Boston: Houghton Mifflin. (1979).
• 8 Ibid, 128, 134.
• 9 Ibid.
• 10 Ibid, 133.
• 11 Ward, M., Keim, D., Grinstein, G. Human Perception and Information Processing. In Interactive Data Visualization Foundations, Techniques, and Applications, Second Edition (pp. 73, 74). New York: A K Peters/CRC Press. (2015). doi:10.1201/b18379.
• 12 Dijkstra, N., Zeidman, P., Ondobaka, S., van Gerven, M., & Friston, K. (2017). Distinct Top-down and Bottom-up Brain Connectivity During Visual Perception and Imagery. Scientific reports, 7(1), 5677. https://doi.org/10.1038/s41598-017-05888-8
• 13  Norman, D. The Design of Everyday Things. (pg. 12). New York: Basic Books. (2002).
• 14 Ibid, 9, 98.
• 15 Ibid.
• 16 Gibson, 137.
• 17 Lawson, B. (2006). How designers think: the design process demystified. Architectural Press. p.64.
• 18  Gallese, V., & Lakoff, G. (2005). The Brain’s concepts: the role of the Sensory-motor system in conceptual knowledge. Cognitive Neuropsychology, p. 15. https://doi.org/10.1080/02643290442000310
• 19 de Saussure, F. (1959). COURSE IN GENERAL LINGUISTICS. In W. Baskin (Trans.), Course in general linguistics. The Philosophical Library, Inc. http://faculty.smu.edu/dfoster/cf3324/saussure.htm.
• 20 Ibid.
• 21  Reda, Ghsoon. (2016). Ferdinand de Saussure in the Era of Cognitive Linguistics. Language and Semiotic Studies.
• 22  Robbins & Aydede. (2009). A Short Primer on Situated Cognition. Chapter 1.The Cambridge Handbook of Situated Cognition Edited by Philip Robbins and Murat Aydede. Cambridge University Press
• 23  Wilson. (2002). Six views of embodied cognition. https://link.springer.com/article/10.3758/BF031963
• 24  Reed, Stephen. (1972). “Pattern Recognition and Categorization. Case Western Reserve University
• 25  Rosch, E.H.; Mervis, C.B.; Gray, W.D.; Johnson, D.M.; Boyes-Braem, P. (1976). “Basic objects in natural categories”. Cognitive Psychology.
• 26  Gallese, V., & Lakoff, G. (2005). The Brain’s concepts: the role of the Sensory-motor system in conceptual knowledge. Cognitive Neuropsychology, p. 2.
• 27  Ibid.
• 28 Ibid.
• 29 Ibid.
• 30  Reda, Ghsoon. (2016). Ferdinand de Saussure in the Era of Cognitive Linguistics. Language and Semiotic Studies.
• 31 Antanas et al. (2017). Relational Symbol Grounding through Affordance Learning: An Overview of the ReGround Project. GLU 2017 International Workshop on Grounding Language Understanding
• 32 Ibid.
• 1 Haakon Faste. (2017). Intuition in Design: Reflections on the Iterative Aesthetics of Form.
• 2 Ibid.
• 3  Durling, D. (1999) Intuition in Design. In Bulletin of 4th Asian Design Conference International Symposium on Design Science 1999, Nagaoka, Japan, October. ISBN 4-9980776-0-0 C3072
• 4 Ibid.
• 5 Ibid.
• 6 de Saussure, F. (1959). COURSE IN GENERAL LINGUISTICS. In W. Baskin (Trans.), Course in general linguistics. The Philosophical Library, Inc.
• 7 Gibson, J. J. The Theory of Affordances. The Ecological Approach to Visual Perception (pp. 127-128). Boston: Houghton Mifflin. (1979).
• 8 Ibid, 128, 134.
• 9 Ibid.
• 10 Ibid, 133.
• 11 Ward, M., Keim, D., Grinstein, G. Human Perception and Information Processing. In Interactive Data Visualization Foundations, Techniques, and Applications, Second Edition (pp. 73, 74). New York: A K Peters/CRC Press. (2015). doi:10.1201/b18379.
• 12 Dijkstra, N., Zeidman, P., Ondobaka, S., van Gerven, M., & Friston, K. (2017). Distinct Top-down and Bottom-up Brain Connectivity During Visual Perception and Imagery. Scientific reports, 7(1), 5677. https://doi.org/10.1038/s41598-017-05888-8
• 13  Norman, D. The Design of Everyday Things. (pg. 12). New York: Basic Books. (2002).
• 14 Ibid, 9, 98.
• 15 Ibid.
• 16 Gibson, 137.
• 17 Lawson, B. (2006). How designers think: the design process demystified. Architectural Press. p.64.
• 18  Gallese, V., & Lakoff, G. (2005). The Brain’s concepts: the role of the Sensory-motor system in conceptual knowledge. Cognitive Neuropsychology, p. 15. https://doi.org/10.1080/02643290442000310
• 19 de Saussure, F. (1959). COURSE IN GENERAL LINGUISTICS. In W. Baskin (Trans.), Course in general linguistics. The Philosophical Library, Inc. http://faculty.smu.edu/dfoster/cf3324/saussure.htm.
• 20 Ibid.
• 21  Reda, Ghsoon. (2016). Ferdinand de Saussure in the Era of Cognitive Linguistics. Language and Semiotic Studies.
• 22  Robbins & Aydede. (2009). A Short Primer on Situated Cognition. Chapter 1.The Cambridge Handbook of Situated Cognition Edited by Philip Robbins and Murat Aydede. Cambridge University Press
• 23  Wilson. (2002). Six views of embodied cognition. https://link.springer.com/article/10.3758/BF031963
• 24  Reed, Stephen. (1972). “Pattern Recognition and Categorization. Case Western Reserve University
• 25  Rosch, E.H.; Mervis, C.B.; Gray, W.D.; Johnson, D.M.; Boyes-Braem, P. (1976). “Basic objects in natural categories”. Cognitive Psychology.
• 26  Gallese, V., & Lakoff, G. (2005). The Brain’s concepts: the role of the Sensory-motor system in conceptual knowledge. Cognitive Neuropsychology, p. 2.
• 27  Ibid.
• 28 Ibid.
• 29 Ibid.
• 30  Reda, Ghsoon. (2016). Ferdinand de Saussure in the Era of Cognitive Linguistics. Language and Semiotic Studies.
• 31 Antanas et al. (2017). Relational Symbol Grounding through Affordance Learning: An Overview of the ReGround Project. GLU 2017 International Workshop on Grounding Language Understanding
• 32 Ibid.