PhD Thesis

Title: The Application of Multiple Modalities to Improve Home Care and Reminder Systems.

Abstract: Existing home care technology tends to be pre-programmed systems limited to one or two interaction modalities. This can make them inaccessible to people with sensory impairments and unable to cope with a dynamic and heterogeneous environment such as the home. This thesis presents research that considers how home care technology can be improved through employing multiple visual, aural, tactile and even olfactory interaction methods. A wide range of modalities were tested to gather a better insight into their properties and merits. That information was used to design and construct Dyna-Cue, a prototype multimodal reminder system. Dyna-Cue was designed to use multiple modalities and to switch between them in real time to maintain higher levels of effectiveness and acceptability. The Dyna-Cue prototype was evaluated against other models of reminder delivery and was shown to be an effective and appropriate tool that can help people to manage their time and activities.

Warnock, D. (2013). The Application of Multiple Modalities to Improve Home Care and Reminder Systems. PhD Thesis. College of Science and Engineering, University of Glasgow, UK.

Multiple Notification Modalities and Older Users

Abstract: Multimodal interaction can make home care reminder systems more accessible to their users, most of whom are older and/or have sensory impairments. Existing research into the properties of different notification modalities have used younger participants rather than members of the older popu- lation at which they are aimed. This paper presents the results of a user study with older adults that examined how different notifications modalities affected (a) performance in a card matching game and (b) how effective the different modali- ties were at delivering information. Participants were all aged over 50 and notifications were delivered using textual, picto- graphic, abstract visual, speech, Earcon, Auditory Icon, tac- tile and olfactory modalities while playing the game. The re- sults showed that older users were influenced by the same fac- tors as younger users, yet there were subjective differences. The implications for the design of multimodal reminder sys- tems for home care are discussed.

Warnock, D., McGee-Lennon, M., & Brewster, S. (2013). Multiple Notification Modalities and Older Users. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘13, pp. 1091-1094). Paris, France: ACM Press.

The Application of Multiple Modalities for Improved Home Care Reminders

Abstract: My work aims to contribute to the development of home care reminder systems through the development of dynamic multimodal notification technology, able to select a notification modality based on user, environmental and message factors such as sensory impairment and social context. This work involves (1) investigating different properties of notification modalities, (2) establishing guidelines for their use as notifications in the home, (3) creating a prototype dynamic multimodal home reminder system and (4) evaluating it in home trials. At present, work on part two is being completed and preparations have begun for the third part of this research.

Warnock, D. (2012). The Application of Multiple Modalities for Improved Home Care Reminders. CHI ‘12 Extended Abstracts on Human Factors in Computing Systems (CHI EA ‘12, pp. 951–954). Austin, Texas, USA: ACM Press.

Older Users, Multimodal Reminders and Assisted Living Technology

Abstract: The primary users of assisted living technology are older people and are likely to have one or more sensory impairments. Multimodal technology allows users to interact via non-impaired senses and provides alternative ways to interact if primary interaction methods fail. An empirical user study was carried out with older participants that evaluated the performance, disruptiveness and subjective workload of visual, audio, tactile and olfactory notifications then compared the results to earlier findings with younger participants. It was found that disruption and subjective workload were not affected by modality, although some modalities were more effective at delivering information accurately. It is concluded that although further studies need to be carried out in a real-world settings, the findings support the argument for multiple modalities in assisted living technology.

Warnock, D., McGee-Lennon, M., & Brewster, S. (2012). Older Users, Multimodal Reminders and Assisted Living Technology. Health Informatics Journal, 18(3), 181–190. SAGE Publications.

The Impact of Unwanted Multimodal Notifications

Abstract: Multimodal interaction can be used to make home care technology more effective and appropriate, particularly for people with sensory impairments. Previous work has revealed how disruptive notifications in different modalities are to a home-based task, but has not investigated how disruptive unwanted notifications might be. An experiment was conducted which evaluated the disruptive effects of unwanted notifications when delivered in textual, pictographic, abstract visual, speech, earcon, auditory icon, tactile and olfactory modalities. It was found that for all the modalities tested, both wanted and unwanted notifications produced similar reductions in error rate and task success, independent of modality. The results demonstrate the need to control and limit the number of unwanted notifications delivered in the home and contribute to a large body of work advocating the inclusion of multiple interaction modalities.

Warnock, D., McGee-Lennon, M., & Brewster, S. (2011). The Impact of Unwanted Multimodal Notifications. Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI ‘11, pp. 177–184). Alicante, Spain: ACM Press.

The Role of Modality in Notification Performance

Abstract: The primary users of home care technology often have significant sensory impairments. Multimodal interaction can make home care technology more accessible and appropriate, yet most research in the field of multimodal notifications is not aimed at the home but at office or high-pressure environments. This paper presents an experiment that compared the disruptiveness and effectiveness of visual, auditory, tactile and olfactory notifications. The results showed that disruption in the primary task was the same regardless of the notification modality. It was also found that differences in notification effectiveness were due to the inherent traits of a modality, e.g.olfactory notifications were slowest to deliver. The results of this experiment allow researchers and developers to capitalize on the different properties of multimodal techniques, with significant implications for home care technology and technology targeted at users with sensory impairments.

Warnock, D., McGee-Lennon, M., & Brewster, S. (2011). The Role of Modality in Notification Performance. Proceedings of the 13th IFIP TC 13 International Conference on Human-Computer Interaction - INTERACT 2011 (Lecture Notes in Computer Science Vol. 6947, pp. 572–588). Portugal, Spain: Springer Berlin Heidelberg.

A Subjective Evaluation of Multimodal Notifications

Abstract: The primary users of home care technology often have significant sensory impairments. Multimodal interaction can make home care technology more accessible and appropriate, but most research in the field of multimodal notifications is aimed at office or high-pressure environments instead of the home. Two experiments were carried out that evaluated the subjective workload of responding to visual, auditory, tactile and olfactory notifications (simulating home care reminders) while carrying out a primary task (a card matching memory game). The subjective measurements and observations revealed that participants were open-minded about the possibilities and applications of these modalities, suggesting that home care technology should embrace a much wider range of interaction methods than are currently used.

Warnock, D. (2011). A Subjective Evaluation of Multimodal Notifications. Proceedings of the 5th International Conference on Pervasive Computing Technologies for Healthcare (Pervasive Health ‘11, pp. 461–468). Dublin, Ireland: IEEE Computer Society.

An Exploration of Cursor Tracking Data

Abstract: Cursor tracking data contains information about website visitors which may provide new ways to understand visitors and their needs. This paper presents an Amazon Mechanical Turk study where participants were tracked as they used modified variants of the Wikipedia and BBC News websites. Participants were asked to complete reading and information-finding tasks. The results showed that it was possible to differentiate between users reading content and users looking for information based on cursor data. The effects of website aesthetics, user interest and cursor hardware were also analysed which showed it was possible to identify hardware from cursor data, but no relationship between cursor data and engagement was found. The implications of these results for web analytics and the design of user engagement experiments are discussed.

Warnock, D. & Lalmas, M. (2015). An Exploration of Cursor Tracking Data. arXiv:1502.00317.

Dynamic Affinity Scheduling in Heterogeneous Multi-Core Processors

Abstract: Heterogeneous processors designs are becoming increasingly common, and are likely to continue gaining popularity due to high-profile processors such as the Cell Broadband Engine. Traditionally programming for such a heterogeneous multi-core architecture requires separate design and implementation for each processing core, and programs are strictly bound to the core for which they were created. It is often the case that the tools available for each core are significantly different, and heterogeneous multi-core architectures have earned a reputation for being very difficult to develop for.

Recent work by Ross McIlroy has produced Hera JVM, a Java Virtual Machine which runs on the Cell processor and abstracts the complex hardware. Hera JVM uses annotations to schedule threads to cores, which prevents full hardware abstraction. This project aims to replace the annotation-based scheduler in Hera JVM with a dynamic affinity scheduler, allowing for a more complete abstraction of the heterogeneous processor.

A prototype scheduler was constructed, then its performance was compared to a static scheduler and the previous annotation-based scheduler. The dynamic affinity scheduler was found to be capable of matching or surpassing the performance of the alterantive schedulers while offering a greater degree of hardware abstraction.

Warnock, D. (2009). An Exploration of Cursor Tracking Data. Master’s Thesis. College of Science and Engineering, University of Glasgow, UK.