ICORR 2013 Workshops


ICORR 2013 Workshop: Evaluation of Intelligent Powered Wheelchairs
Afternoon of Wednesday, June 26, 2013 (4 hours)

Several intelligent wheelchairs have been developed for users with disabilities. However, only a small proportion of these wheelchairs have been tested with their target user populations, possibly due to the lack of system robustness and reliability required for clinical use, as well as difficulties in study recruitment. Thus, there is a lack of understanding of the user acceptance of these technologies. The proposed workshop will bring together a diverse group of computer scientists, engineers and occupational therapists from all over the world in order to gain a holistic understanding of the challenges in developing and evaluating intelligent wheelchairs. The organizers will use their previous experiences in intelligent wheelchair design and evaluation to lead and moderate workshop discussions.
The specific objectives of this workshop will be to:
  1. Review possible domains of intelligent wheelchair evaluation: evaluation metrics, technology performance, user performance, usability, and study design for challenging populations (through the invited speaker session)
  2. Examine and discuss potential user groups for intelligent wheelchairs
  3. Discuss as a large group: Standardizing evaluation and measurement across intelligent wheelchairs to compare system and user performance
  4. Discuss in small groups: Design considerations for specific target user populations and challenges in testing
  5. Present findings and recommendations from 4) to entire group (to be included in a workshop report forwarded to all participants)
Learning outcomes for the workshop participants will be to:
  1. Become familiar with existing methods and metrics used in intelligent wheelchair evaluation
  2. Identify issues in current methodology with respect to intelligent wheelchair design and testing
  3. Make recommendations to close existing gaps and facilitate standardized evaluation
Through this workshop, we hope to identify challenges faced by researchers currently developing intelligent wheelchairs (and other assistive devices), provide possible solutions to facilitate testing with target users, and foster new international collaborations.


Presentation Title
Pooja Viswanathan and Rosalie Wang
University of Toronto and Toronto Rehabilitation Institute, Canada
The Organizers and CanWheel (A Canadian Institutes of Health Research Emerging Team in Wheeled Mobility for Older Adults)
Cristina Urdiales Garcia (tentative)
University of Malaga, Spain
Metrics and Benchmarking for Assisted Wheelchair Navigation
Christian Mandel
German Research Center for Artificial Intelligence and Bremen University, Germany
Development and Evaluation of Mobility Assistants and Smart User Interfaces for Electrically Powered Wheelchairs
Francois Routhier
Laval University and Center for Interdisciplinary Research in Rehabilitation and Social Integration, Canada
Evaluating Smart Wheelchair Technology: From the Laboratory to the Real World


Sajay Arthanat
University of New Hampshire, U.S.A.
Exploring Usability of Intelligent Powered Wheelchairs in Multiple Contexts
Karyn Moffatt
McGill University, Canada


Generalizability in Research with Cognitively Impaired Individuals

Speaker abstracts and biographies

Metrics and benchmarking for assisted wheelchair navigation
C. Urdiales, M. Fdez-Carmona, G. Peinado and F. Sandoval

Abstract: Assistive technologies have been of key importance for the last decades. There have been many approaches to assistance and different paradigms have been proposed. However, there is still a lack of standardization in the field, not because no scale or metrics exist, but rather because there are many and they serve different purposes. Besides, since disability is a complex concept, in order to fully characterize a given test, several scales are needed to provide information on issues like ergonomics, user’s condition, technology, etc. It needs to be noticed that not all scales are appropriate for every experiment, so it is important to know which ones to use in terms of what needs to be measured. Finally, it needs to be noted that benchmarking in this context is even more difficult, because there is no such thing as a standard user. Indeed, even people suffering the same condition react differently to assistance depending on many factors. This paper presents a taxonomy of the most common scales used in assistive technologies, indicating what they are typically used for. Then, a review of current benchmarking procedures is presented along with a new benchmarking tool, which is based on data mining techniques to extract a prototype user from tests with a large number of volunteers presenting different degrees of physical and cognitive disabilities. This tool and a chosen set of metrics are tested with a real experiment on wheelchair navigation both in assisted and unassisted mode.

Development and Evaluation of Mobility Assistants and Smart User Interfaces for Electrically Powered Wheelchairs
C. Mandel, PhD

Abstract: This talk gives an introduction to the development and user-centered evaluation of driving- and navigation-support for users with different levels of impairment. The most basic Driving Assistant takes user-commands from a standard hand-joystick, or optionally from a head-joystick that translates the operator’s head movements into driving commands. By evaluating local 2D maps generated from laser range finder data and odometric information, user commands are checked for safety, and automatically modified when necessary. Apart from a look on first test sessions with healthy subjects, this talk focuses on a clinical study in which impaired participants from two groups (with and without driving assistance) operated an electrically powered wheelchair by hand- or head-joystick on a standardized parcours. Evaluation criteria included simple performance metrics, such as speed and driven distance, as well as the entropy rate of controller histograms. It is shown that the latter one can be interpreted as a measure for the cognitive challenges imposed by the user’s task and by the system. In addition to the treatment of the driving assistant’s aspects, this talk will shed light on the development and testing of more sophisticated user-interfaces, e.g. target pose selection for autonomous navigation by speech input, or a brain computer interface (BCI). While the former applies simple keyword spotting and selects targets labeled within the mapped environment, the BCI experiments rely on the detection of steady state visual evoked potentials. The work presented here has been performed in several projects at the University of Bremen in the context of the DFG funded Transregional Collaborative Research Center Spatial Cognition (SFB-TR8), and at the German Research Center for Artificial Intelligence (DFKI). It is currently continued at DFKI in the project Assistants for Safe Mobility (ASSAM), funded under the AAL Joint Programme by the European Commission and the national funding organizations of the member states involved.
Bio: Christian Mandel studied computer science at the universities of Hildesheim and Bremen. His diploma thesis “Trajectory Generation and Control in the Configuration Space of Non-Holonomic Vehicles” dealt with real-time capable planning and control algorithms implemented on the Bremen Autonomous Wheelchair Rolland. For his PHD thesis “Navigation of the Smart Wheelchair Rolland”, he developed further navigation and localization solutions applicable in indoor and outdoor scenarios at the transregional collaborative research center Spatial Cognition in context of the projects A1-RoboMap and T2-Rolland. Beside the continuation of this work, Christian Mandel is currently with the German Research Center for Artificial Intelligence (DFKI) where he develops outdoor navigation solutions for the elderly in context of the project Assistants for Safe Mobility (ASSAM).

Evaluating Smart Wheelchair Technology: from the Laboratory to the Real World
François Routhier, PEng, PhD; Dahlia Kairy, PT, PhD; Robert Forget, PT, PhD; Louise Demers, OT, PhD and Joelle Pineau, PEng, PhD

Abstract: Many people who have to rely on powered wheelchairs can find it difficult to fulfill daily navigation tasks due to their physical and/or cognitive impairments. Smart wheelchair technology can help them minimize the physical and cognitive load required in steering their powered wheelchair. Few smart wheelchairs have been developed in the past and one major challenge faced by the research teams that developed such wheelchairs is the lack of formal tools to evaluate them. What perspective should be used? The developers’ perspective or the users’ perspective? Smart wheelchair can perform very well from an engineering point of view, but the way they perform in the real environment is paramount. What tasks should be considered? What performance metric should be used? How can users’ needs, confidence and satisfaction be addressed? The objective of this presentation is to discuss relevant evaluation methodologies, from laboratory to wheelchair user environment. A focus will be put on how our research team has evaluated the SmartWheeler using the Wheelchair Skills Test. At the conclusion of this workshop, we wish to develop a list of criteria on the properties required of a good evaluation tool for smart wheelchair technologies.
Bio: François Routhier is an Assistant Professor at the Rehabilitation Department (occupational therapy program) at Université Laval (Québec, Canada) since 2011 and researcher at the Center for Interdisciplinary Research in Rehabilitation of Social Integration of the Institut de réadaptation en déficience physique de Québec (Québec, Canada) since 2008 . He obtained his Ph.D. in Experimental Medicine from Université Laval in 2004 and completed postgraduate training in physical medicine, rehabilitation and aging at Dalhousie University and Université de Montréal between 2004 and 2008. His research interests include rehabilitation technology development and assessment, wheelchair skills, wheelchair performance and wheelchair mobility.

Exploring usability of intelligent powered wheelchairs in multiple contexts
Sajay Arthanat, PhD. OTR/L., ATP

Abstract: Usability of assistive technology (AT) devices determines how effectively and efficiently users participate in meaningful occupations and life roles. This presentation will highlight the conceptual definition of AT usability and introduce a framework, the usability scale of assistive technology (USAT), for the measurement of this AT outcomes domain. I will briefly elucidate some past research involving the USAT for wheeled mobility and illustrate findings pertaining to usability of specialized power mobility devices in multiple contexts. The latter part of the presentation will focus on operationalizing usability of intelligent power wheelchairs by identifying potential measurable indicators and effective methodologies that could be applied towards its evaluation.
Bio: Sajay is a faculty in the Department of Occupational Therapy at University of New Hampshire. He received his Bachelor’s Degree in Occupational Therapy from Santosh College of Occupational Therapy in India (1997), an Advanced Certificate in Assistive Technology (2004) and a Ph.D. in Rehabilitation Science from University at Buffalo (2007). Prior to his current position, he worked as a Research Associate at the Rehabilitation Engineering Research Center on Technology Transfer at University at Buffalo. Dr.Arthanat’s research program involves a) developing methods and tools to measure outcomes and benefits of assistive technology devices by people with disabilities; and b) implementing strategies to integrate evidence to optimize the quality of AT service provision. Dr. Arthanat developed the Usability Scale of Assistive Technology (USAT)-a human factors science framework to evaluate the usability of AT devices relative to the user’s participation, skills and context. The USAT-Wheeled Mobility was specifically designed and tested to measure user-perceived usability of wheeled mobility devices from a multi-contextual perspective.

Generalizability in Research with Cognitively Impaired Individuals
Karyn Moffatt, PhD

Abstract: A major challenge in designing technology for users with cognitive impairments is achieving generalizable results. By design, controlled experiments—the gold standard for evaluation—remove individual differences through statistical means. While this is appropriate in many situations, when working with special populations, individual differences are often very pronounced, and, generally, crucial to the success or failure of the design. Custom-built systems circumvent the problem of addressing diverse needs by creating individualized solutions to match the unique needs of each user. However, this approach limits our ability, as researchers, to develop broader, more generalized theories that can be used as the basis for future work. Thus a tension exists between satisfying the need for customized solutions that address the immediate needs of users and identifying generalizable results that can be used to achieve long-term research goals.
In this presentation, I will share my experiences designing technology for individuals with aphasia (an acquired communication impairment limiting the production and comprehension of spoken and written language). In this project, we sought a balance between identifying generalizable results and ensuring that our designs addressed unique needs. We found that designing for one individual or small group, yet testing with a larger set of potential users helped us identify which characteristics of a design are generalizable and which will require customization. We also found that gathering detailed functional assessments of each participant’s abilities aided in interpreting results when individual differences were pronounced. These approaches are by no means a complete solution, but I hope that presenting them can motivate further discussion on these issues.
Bio: Karyn Moffatt is an assistant professor in the School of Information Studies at McGill University. Her research broadly explores the ways in which technology can be employed to meet the human needs, especially in the context of designing for older adults and people with disabilities. Her work has led to a number of publications in top-tier academic venues and has been recognized with awards at ACM ASSETS 2007 and ACM CHI 2009. She was a member of the Aphasia Project throughout her graduate studies, and remains active in this space. At the moment, she is especially interested in exploring the design of assistive tools that are sensitive to the social context in which they will be used and that seek to leverage and support those relationships. Prior to joining McGill University, Karyn was a post-doctoral fellow at the University of Toronto supported by awards from both NSERC and CIHR’s Health Care, Technology, and Place strategic initiative She received her doctorate in computer science from the University of British Columbia, where she studied methods for increasing the accessibility of pen-based interaction for older adults.

For more information, e-mail Pooja Viswanathan or visit the ICORR 2013 website.