Keynote address at the Second Nordic Conference on Human-Computer Interaction (NordiCHI) in Aarhus, Denmark. October 22 , 2002
Bodil Jönsson, Professor, Certec
, Lund University, Sweden
  Föreläsningen i nerladdningsbart PDF-format.

Ten Thoughts On
Enabling Communication - For All

This keynote lecture focuses on human-computer interaction (HCI) in the special needs context. Almost every example originates from Certec,1 the Rehabilitation Engineering Division at the Department of Design Sciences2 in Lund, Sweden -- not because these are necessarily the best examples but because they are the ones that I have the greatest insight into. If you want to know more about how special needs HCI relates to technology, design, education, art, etc., please advance to the references and links that I give here, and you will find many for the entire field and its connections to other disciplines. You can also get a more in-depth analysis and additional background by reading my chapter "Enabling Communication -- Pictures as Language" in a book soon to be published entitled "Enabling Technologies".3 

I am grateful to be a keynote lecturer here today. Believe me, it is fun, and my sincere hope is that the lecture will be of some relevance to you. But I also have a more egoistic reason for being here. At Certec we are eager to identify relevant theories (along with researchers in other fields) or to come up with new, userworthy theoretical foundations for the following reasons:

  1. To better understand and utilize the results we have achieved.
  2. To devise more effective implementations than we have so far been able to.

The reason for this eagerness is that we have made so many efforts and obtained so many fruitful and documented results which we have not been able to refer to any established scientific background. In other words, we need help.

3 MacLachlan & Gallagher, Eds., Enabling Technologies, Elsevier Science Ltd., UK, 2003.


Introductory remarks

People with disabilities are one group in society who have been exposed more than any other to extensive training in following directions and in repetitive practice of skills and abilities. This means that they have been exposed to an educational approach that includes learning by transfer, a stimulus-response situation, a belief in learning through repetition. This approach is still the dominating one, a deeply entrenched pattern despite all the talk of individual development and the right to self-determination. But interaction is something different -- it is about questions as well as answers, feedback as well as instruction. This is where HCI comes into play.

The old pattern will not be altered, perhaps, until we understand the difference between illuminating the questions versus supplying the answers. There are few who would question that people with cognitive limitations need considerable support in order to see structure and context in the world around them. The fictional character, Henrik Persson, described in Arne Svensk's "Design for Cognitive Assistance",4 is a good example. He is an adult with cognitive limitations, living on his own and holding down a job. He gets some outside help from an assistant now and then and support at work. Yet, there are still many different situations that he does not grasp, resulting in his own disappointment and anxiety, and in conflicts with people around him. But is it sermonizing, nagging, scolding and repetition from others that he needs? Or an environment designed to better help him manage activities of everyday living on his own?

A polarization exists in the focus on questions versus answers in a variety of educational approaches. Constructivism is at one end, with its view of learning as a process of searching and creating; the main emphasis is placed on questions. The assumption is that people learn by asking, seeking out, generating and finding support in the external world and inside themselves, all in order to achieve learning, that is to say, internal change. At the other end, there is learning psychology, programmed learning, teaching technology and similar approaches, in which learning is studied as if it were independent of content and environment, as if it could be primarily controlled through situations and instructions arranged in the right manner by the external world.



Now to my ten thoughts.

1. In its broadest sense, communication is the most important part of human life

Enabling communication with others and with yourself can be considered the overall goal of HCI. People with "special needs" really have no basic special needs -- what is special for them is that they have specific difficulties and, consequently, needs for special solutions. But sometimes, these solutions are of more general value -- for all.

Increased difficulties in learning are often a part of a disability or a side effect thereof. If you look at the combined system of the human being and her information technology as a learning unit, you see something entirely different than if you consider the human being on her own or technology on its own. Consider, for example, children with low vision or those who are blind and how together with their computers they make up a learning unit. It is this unit that at an early stage can learn a variety of key approaches for identifying different phenomena.


Stig with picture rollers.

One of Certec's most important areas concerning IT in rehabilitation and education deals with digital pictures.5 It started with the Isaac Project in 1993 and since then we have witnessed the kind of tremendous breakthroughs that can be achieved when people (adults and children) who have not had any spoken language start to be able to communicate through the use of a multitude of personal digital photos. This goes not only for people with cognitive disabilities but for those with psychiatric disorders as well.6 If you consider psychiatry through the eyes of design, it is apparent that the care environment in and of itself is disabilitating. It entirely lacks the kind of distributed cognition that could contribute to making the world more comprehensible and less threatening, as well as enabling and sustaining contact with the world around through electronic communication, also with pictures.



2. Special needs HCI can serve as eye-openers

You can never really generalize from one person to another. What you can do is that through the individual example, the particular case study, see something that can be expressed on a more general level, for example, as a design principle. If this principle, which has no statistical basis but instead is more of an aha experience based on work carried out under extreme difficulties, has any general value is determined by its ability to help us see new phenomena that we have never seen before.7 Three examples of eye-openers are presented here: one on mobility, one on dependence on feedback and confirmation, and one on stress.


User interface to Isaac 2002.

Mobility, for example, is one of the noble features of information technology. If I ask you "why", you could give me many examples. Now, allow me to give you what appears to be an odd example of a seven-year-old boy with autism and what his teacher observed. When he was at school and his parents were out of sight, they were literally out of his mind too. It was as if they did not exist. They were definitely not a part of the school setting and it was disturbing for him if they appeared there (the same being true of his teacher when the child was at home). His teacher had incorporated into his educational program digital photos of his different settings and the people in them. She used them to teach him about mobility. As she explained: "It was when we started using digital pictures of people that we moved between pictures of different buildings, that he gradually started to realize that people do not disappear when you can no longer see them." It could be that if we also started to examine in depth why the rest of us so eagerly seek out and desire mobility, we would find behind all the more obvious reasons, this one, among others.

Other distinctive features to which HCI contributes are feedback and confirmation. What is it in being human that makes us so dependent on these? The same young boy from the last example often wonders intensively about things and people. At these times, he needs to see a picture as confirmation that the listener understands what and who he means. Without a picture, such confirmation is impossible. If his teacher forgets to photograph something new in the classroom or if he does not have access to pictures of those people who are usually there, his world becomes chaotic. It is not until he is able to see the person or object in a photo, and is able to confirm for himself that it is the same one he was thinking of, that he settles down. Essentially, I do not believe that this is so different from other people's need for feedback and confirmation. This might very well be the factor that has resulted in the powerful impact mobile telephones, e-mail and internet have had.

Now to an entirely different example: stress, that shortcoming in human nature that has become increasingly apparent in recent years. There is no doubt that computers and the internet provide a great amount of leeway in the areas of speed and rhythm. Those who need more time can take it -- without bothering anyone else. And those who work quickly can also do so -- without bothering anyone else. This is an important prerequisite for establishing good communication between people with different rhythms. It is also a superb measure for counteracting stress. But at the same time, computerized homes and workplaces have contributed to a widespread increase in stress.8,9

There is most likely a need for research and analysis in this area, not only on the level of HCI, but also on the level of MTI -- mankind-technology interaction.

8 Jönsson, B. I tid & otid: hemma och på jobbet (At All Odd Times of the Day: At Home and on the Job), Brombergs, Sweden, 2002.


3. The disability area has great potential for HCI

The area of disabilities is characterized not only by great possibilities for HCI -- but also difficulties. And the potential lies in the difference between possibilities and difficulties. In the area of HCI special needs, the difficulties are so great that you cannot simply continue along the same tangent -- you are often forced to invent something genuinely new. The difficulties are not only a burden -- they can also be an asset.

The STEP method that Arne Svensk developed in his licentiate thesis, "Design for Cognitive Assistance",4 is an example of this. The four key concepts he presents are to provide Security, conText, Experience, Precision. His approach deserves studying in the broader HCI perspective.


Arne shows his artefacts.


4. Critical cases 

It sounds almost trite to say that critical cases have a unique position -- it's somehow part of the definition. But if I say instead that you can profit by seeking out the critical cases, the statement becomes less trite. If you can find critical cases -- and believe me, they are over represented in the disability context -- you can with a high degree of certainty confirm that "if it's valid in this case, it's most likely valid in most cases" alternatively, "if it isn't valid for her, it won't be valid for anyone."

An illustrative example of "if it helps in this case, it should do the same in almost all cases," is when Alan Alda for the first time met a man with cognitive limitations who could not speak -- but who had digital photos as a language. Mr. Alda, host of the TV series "Scientific American Frontiers" on PBS, the American public television network, was in Lund to film an episode. The producers had chosen Isaac and its users as the Swedish contribution. Alan visited one of our test sites, a unique day activity center for adults with cognitive limitations where we have tested and developed our ideas and technologies over the years. It is called the Pictorium. The program was aired in January 1998 on the theme of major research advances in Scandinavia in recent years.


Potatoes. From seed potatoes to boiled potatoes on the table.

On the day of filming, Alan arrived at the Pictorium and was totally unprepared for the enormous number of digital pictures, the bar coding, the bar code scanner and the computers (not knowing what to expect is part of the program concept). After trying to talk to the users for a while, he realized that he could communicate with them by pointing at their pictures. First, he tried to show them what kind of food he liked and chose boiled potatoes. When this picture came up on the computer screen, Stig, one of the users, immediately turned and pointed to another picture of a lady, not once, but quite a few times and with great emphasis.

What was Stig trying to tell Alan Alda? What would Alan have understood if he had known more about Stig? Simply that the users at the Pictorium grew their own potatoes, that they grew them in the lady's garden and that they were going to harvest them soon. Not a bad chain of thought for a person who has a developmental disability, who lacks a spoken and written language and who has very limited sign language. As this example illustrates, Stig had grown tremendously through the use of Isaac. He was able to manage a visit with an actor from an English-speaking country and could tell him about one of his many new perceptions of how things are related. This would have been totally impossible before Isaac. Stig was unable to show those around him what he wanted to say and they were unable to guess. However, with over 10,000 digital pictures of his own, he has turned the pictures not only into words but into a language.10



5. You cannot know until you have tried

This is one of the most golden of rules that we know. Don't believe in assertions from the rest of the world or your own preconceived notions. You have to test in the situation, with people.

You cannot rely on needs to be formulated by the users just because they exist. By providing a solution, technology is a very good means of showing that a problem exists and what it consists of. In the very beginning, it may be difficult, sometimes impossible, to involve the users. On the other hand, later on in the process, the initiative may shift making it difficult for those with the original initiative to keep up with all the ideas that are being generated. Things start happening when empowerment works. Otherwise not.10


Emma, Björn and the Minimeter.

6. The special status of case studies

The first criterion for good HCI design should be that you are able to demonstrate that it is good for at least one person. If you succeed, the likelihood that it will also be good for others is much greater than if you adjust your design according to a hypothetical user. If you do not have anyone else to test it on, make an honest attempt on yourself at any rate. That experience often goes further than abstract ideas.

An excellent master's thesis was presented this month at the School of Art and Communication at Malmö University. Marie Höglund relates in "Ingegärd and the Green Mailbox-A Design Process for and with People with Dementia".11 It is an unusually challenging account because of the ethical aspects presented, the genuine difficulties and the options that are actually available.



Ingegärd and the green mailbox.

7. The computer's power of attraction

Computers (even without network connections) have a remarkable attraction for many people with disabilities. As early as 1996, I tried to summarize what I thought I understood of this phenomenon. At that time, I was already suggesting that this could be made into a research program of its own. That did not happen -- but it's not too late. On the contrary, it is just as current an issue now as then. To the issues that were raised then and that can be raised now are:

It appears as though personal digital pictures are often more real for the users than reality itself. It is as if it was easier for them to establish a relationship with the experience or person through the picture rather than through direct contact. There are a number of factors that could be responsible for this: that the picture is a still life and unalterable, that it is bright, that it has a frame, that it enables the viewer to see the situation from different angles, etc.

When examining what a relationship to another person can signify for learning, the most important function of HCI in this case is that it permits an exchange in taking the initiative between people with different abilities and their surroundings. It enables conversation based on a common focus and, perhaps most importantly, it enables variation. 12



A computer with a face and arms.

8. Internet and first-time capabilities

For the rest of us, internet means great changes. Wider highways. But for many people with disabilities, the internet represents first-time capabilities to participate in education, to get in touch with one another, to make dreams come true. I will stick to two examples: internet education and interaction with people with psychiatric disorders.

When it comes to education, Peter Anderberg is the person who has considered first-time capabilities the most at Certec. He has reported on this in his licentiate thesis, "Internet Learning for All -- Efforts towards the First Satisfactory Study Opportunities for People with Disabilities".13

Psychiatry is the most difficult and most neglected area when it comes to IT, rehabilitation and education. In the only doctoral dissertation that has been presented in this area, "From Medication to Education", by Eve Mandre in May of this year,6 HCI has no prominent position. And yet it was the ability of a digital camera to bring the outside world to a person (when it was impossible for that person to go out into the world) that was the start of a successful rehabilitation process. The HCI designer who would like to pursue this faces great challenges and great possibilities.



Digital camera with a picture of a flying eagle.

9. Cognitive limitations and language disability

A person unable to acquire a language will be unable to learn to any great extent. If the effects of this have been mounting since early childhood, it can result in adults with cognitive limitations being very different from others in the areas of feelings, knowledge, how they act and relate. But it is never too late. In fact, the introduction of digital pictures as language to people in their 50s or older can result in a strong, positive change and a constructive process for all parties involved. If you are highly aware of this when designing HCI, you can contribute to people who lack a spoken or written language being able to gain a functional language. In this way, a number of the cognitive limitations can be compensated for or reduced.

Our experience with the Isaac Personal Digital Photo Project tells us that there is a threshold value in the range from 1,000 to 2,000 photos. It is as if the pictures start to work as a language for those who have a collection that exceeds that number. But when there are so many pictures, large demands are placed on the interface to enable you to find the picture you are looking for. That is why the Isaac of 2002, which is currently being beta tested, has been developed as a computer application that can easily manage large numbers of digital pictures with related sound and text.14, 5



 Not being able to speak is not the same as not having anything to say.

10. Multimodal communication with a myriad of possibilities

A disability often results in some means of communication being cut off. Vision, hearing, the ability to move can, for example, be lost. In this case, a complete multimodal communication is something unattainable -- what is most important is if there is any modality at all that can be used. Nevertheless, multimodal communication is of special significance in the area of disabilities -- and vice versa. The more you think of how haptics can be used as support for navigation in Windows, for example, or how sound and haptics can work together, the greater are the chances that blind people will also be able to handle state-of-the-art computer programs. And inversely -- by developing and studying haptics and haptics with sound for blind people, the better this complement can also be for those who see.15 It is encouraging to see that one of the papers presented at this conference directly addresses HCI special needs in this area. It is Donker, Klante and Gorny's, "The Design of Auditory User Interfaces for Blind Users".16

And now a little but significant example: An early finding was that some differently abled people did not clearly see a picture if it did not have some kind of a frame. So we started framing all the pictures. That gave a special education teacher sudden insight: This might be the reason why sitting by a window has such a positive effect on some students. Could it be that the window provides them with a frame through which to see the world? And in that way, one and the same slice of real life becomes easier to manage than the complicated, three-dimensional, constantly changing world.



The PHantom.

In conclusion

Art and science have much in common. One aspect is that both raise new questions, the basic difference being in how the questions are used. While science immediately starts searching for the answers in order to make them public, art takes on the role of finding the questions, emphasizing them and "hanging" the questions up for people to see. So that people can answer themselves -- or continue living with the questions.17

I stand here before you as a scientist, and I have referred to many "answers". But the driving force for being here is to expose these answers as questions and beg you to answer yourself -- or continue living with the questions.



Question mark and exclamation mark.


1. Certec, Rehabilitation Engineering Division, Lund University, Sweden.

2. Department of Design Sciences, Lund University, Sweden.

3. MacLachlan & Gallagher, Eds., Enabling Technologies, Elsevier Science Ltd., UK, 2003.

4. Svensk, A., Design for Cognitive Assistance, licentiate thesis, Certec, LTH, Lund, Sweden, 2001.

5. Danielsson, H., Jönsson, B., Pictures as Language, lecture, International Conference on Language and Visualisation, Stockholm, Sweden, November
8-9, 2001.

6. Mandre, E., From Medication to Education: People with Autism in Adult Psychiatry, doctoral thesis, Certec, LTH, Lund, Sweden, 2002.

7. Jönsson, B., Anderberg, P., Rehabilitation Engineering and Design Research-Theory and Method, Certec, LTH, Lund, Sweden, 1999.

8. Jönsson, B., I tid & otid: hemma och på jobbet (At All Odd Times of the Day: At Home and on the Job), Brombergs, Sweden, 2002.

9. Jönsson, B., Löfgren, O., Salonen, T., Thomsen, M., Pacemaking: A Research Program on Stress, Certec, LTH, Lund, Sweden, 2002.

10. Jönsson, B., Philipson, L., Svensk, A., What Isaac taught us, Certec, LTH, Lund, Sweden, 1998.

11. Höglund, M., Ingegärd and the Green Mailbox-A Design Process for and with People with Dementia, master's thesis, School of Art and Communication at Malmö University, Sweden, 2002.

12. Jönsson, B., Datorns attraktionskraft, (The Computer's Power of Attraction), report, Certec, LTH, Lund, Sweden, 1996.

13. Anderberg, P., Internetlärande för alla: Om insatser för de första fullvärdiga studiemöjligheterna för funktionshindrade människor (Internet Learning for All-Efforts towards the First Satisfactory Study Opportunities for People with Disabilities), licentiate thesis, Certec, LTH, Lund, Sweden, 1999.

14. Isaac: A Personal Digital Assistant for the Differently Abled People, Certec, LTH, Lund, Sweden, 1994.

15. Haptics at Certec,

16. Donker, H., Klante, P., Gorny, P., The Design of Auditory User Interfaces for Blind Users, plenary paper presented at the NordiCHI 2002 Conference, Aarhus, Denmark.

17. Plato, G., Jönsson, B., Art and Science-A Different Convergence, report, Certec, LTH, Lund, Sweden, 2001.