RAPS

From HCIM

Contact point: Nuno Guimarães

Contents

Summary

The analysis and evaluation of interactions in computing systems has historically been performed through empirical methods, essentially based in observations, scene analysis through recording or annotations, or logging of behavioral data. The growing understanding of the human physiology and the availability of capture technology is making it possible to use experimental methods for the analysis and evaluation of interactions. This project is focused on the use of electroencephalography (EEG) to perform reading detection and analysis.

The importance of building a reading analysis and detection environment is based on the following beliefs:

  • The detection of reading processes and states has a pervasive relevance in the analysis and evaluation of interactive situations that are related with attention, concentration, legibility, or learning,
  • There is a renewed interest in the analysis of reading processes. The introduction of computer screens has promoted a host of studies about the reading of electronic texts which have produced a body of knowledge that is still valid today. The design and development of new devices, from small screens (mobile and smart phones, PDA) to large displays (screens, walls, interactive surfaces) has repositioned the questions on the reading activity and gave the analysis of this process a new relevance.

On the other hand, the use of EEG signals for reading detection and analysis is justified by the following facts:

  • There is a consolidated understanding of the neural basis of reading processes,
  • There is an evolution of the EEG devices that will, in the coming future, create devices that are much easier to use than current EEG systems, through the use of “dry” electrodes and wireless transmission of EEG data,
  • If the previous condition holds, and we have evidence that it will eventually do, the use of EEG will provide minimal limitations (apart from the use of the right caps) and will be more independent from the physical set up and environment where reading occurs (desktop, living room, outdoor activities),
  • The availability of robust EEG reading detection will create a new channel of adaptation between the computer and the user, a capacity that has been designated as a passive Brain Computer Interface.

The objective of this project is therefore to design and build a robust and usable reading detection environment based on EEG capture and processing. The conceptual and technical background for this project is the following:

  • 1.The theoretical understanding of reading processes that discriminates reading styles and situations,
  • 2.The previous experiments carried out at the LASIGE/HCIM research group which have harnessed the basic mechanisms for signal capture and explored the space of feature extraction and classification algorithms,
  • 3.The knowledge and experience of the clinical lab of the Faculty of Medicine, which has demonstrated to be an invaluable reference to validate methods and results produced in common interactive situations,
  • 4.The alternative methods for reading detection, which are also based on physical information from the users, but rely on other techniques like eye-movement detection and can be used as benchmarks for this approach.

The operational objectives and work items of the project are:

  • To identify the optimally discriminating configurations of EEG data for the different types of reading activities. These configurations are sets of brain signal samples determined by cortex regions (electrodes) and frequency bands (rhythms). The best configurations are those that allow for the best precision rates with a minimal set of sources. To meet this objective the project will gather a significant set of data based on a battery of experiments with well defined quantitative and qualitative requirements,
  • To tune the capture procedures, feature extraction and classification algorithms to support real-time execution and integration in common interactive tools and applications. The previous experience has identified the best choices in the algorithm space (feature reduction through PCA, Principal Component Analysis, K-NN (Nearest Neighbor) classification) but tuning and real time feasibility is still to be achieved,
  • The evaluation of the usability and effectiveness of light weight and portable EEG devices, which claim to provide clean and usable information but still need to be integrated in effective tools and applications,
  • The design and test of a fully functional prototype of an EEG-based reading detector.

Administrative Info

Participating Research Groups and Team

Contact point: Nuno Guimarães

LASIGE, Human Computer Interaction and Multimedia (http://hcim.lasige.di.fc.ul.pt)

IMM, Laboratorio de Estudos da Linguagem,Behavioural Neurosciences Group (http://imm.fm.ul.pt)

  • Nuno M. Guimarães (Principal Investigator)
  • Inês Oliveira
  • Isabel Pavão Martins
  • Carla Bentes
  • Inês Mares
  • Pedro Custódio

Time Line

The project started officially on 01-01-2011, with a duration of 24 months.

Funding Source

RAPS is funded by Fundação da Ciência e Tecnologia (PT) (Contract PPTDC/EIA-EIA/113660/2009


Publications of the Project (FINAL)

Papers in Journals (submitted)

Inês Mares,Pedro Custódio, Isabel Pavão Martins, Carla Bentes, Nuno Guimarães, Sónia Frota, Cátia Severino, Neurophysiological Analysis of Continuous Reading: the Choice of Text Segmentation Alternatives, , SUBMITTED TO SCIENTIFIC STUDIES OF READING - Rejected and under reformulation (2014/07), , pdf.gif Download paper


Custódio,P., Mares, I., Martins, I.P., Bentes, C., Guimarães, N., Neurophysiological correlates of continuous reading, , SUBMITTED TO BRAIN RESEARCH, , pdf.gif Download paper


Mares, I., 
Custodio, P., Fonseca, J., Bentes, C., Guerreiro, M., Guimarães, N., Martins, I., To read or not to read: a neurophysiological study, , SUBMITTED TO NEUROCASE, , pdf.gif Download paper

Papers in Conferences (published)

Inês I. Oliveira, Nuno M. Guimarães (2013), Practical Neurophysiological Analysis of Readability as a Usability dimension, doi:10.1007/978-3-642-39062-3_12, SouthCHI 2013 - 1st International Conference on Human Factors in Computing & Informatics, Maribor, Slovenia, 2013, pdf.gif Download paper

Inês I. Oliveira, Nuno M. Guimarães (2013), A Tool for Mental Workload Evaluation and Adaptation, doi:10.1145/2459236.2459260, AH'13, Stuttgart, Germany, March 2013, pdf.gif Download paper

Thesis

Inês Oliveira, Análise de Usabilidade com Integração de Sinais Eletroencefalográficos, DOUTORAMENTO EM INFORMÁTICA (ENGENHARIA INFORMÁTICA), Faculdade de Ciências da Universidade de Lisboa, Abril 2013 Inês I. Oliveira, Análise de Usabilidade com Integração de Sinais Eletroencefalográficos, , FCUL, Abril 2013, pdf.gif Download paper

Other

Nuno Guimarães, Inês Mares, Pedro Custódio, Carla Bentes, Cátia Severino, Sónia Frota, Isabel Pavão Martins, Um estudo sobre a Leitura “RAPS-Reading Analysis with neurophysiological Signs” - Sessão Clínica no Hospital de Sta Maria, abril 2013




Background Knowledge and Related Information

HCI in General

Engelbart, D. (1962), Augmenting Human Intellect: A Conceptual Framework, SRI, US, October 1962, pdf.gif Download paper

Neurosciences and HCI

Minnery, B.S. and Fine, M.S. (2009), Neuroscience and the Future of Human Computer Interaction, Interactions, ACM, March-April 2009, pdf.gif Download paper

Brain Computer Interfaces

Gerven, M. et al (2009), The brain–computer interface cycle, J. Neural Eng., 6, 2009, pdf.gif Download paper

Millan, J. (2003), Adaptive Brain Interfaces, Communications of the ACM, 46(3), March 2003, pdf.gif Download paper

Millan, J. (2008), Non Invasive Brain Machine Interaction, International Journal of Pattern Recognition and Artificial Intelligence, 22(5), 2008, pdf.gif Download paper

Zander, T. and Jatzev, S. (2009), Detecting affective covert user states with passive Brain-Computer Interfaces, ACII 2009, Los Alamitos, CA, 2009, pdf.gif Download paper

Zander, T. et al (2010), Enhancing Human-Computer Interaction with input from active and passive Brain-Computer Interfaces, ZMMS, TUBerlin, DE, 2010, pdf.gif Download paper

Neurosciences and Reading

Oliveira, I. et al (2010), Relevance of EEG Input Signals in the Augmented Human Reader, AH'10, Megéve, FR, 2010, pdf.gif Download paper

Eye Tracking

Loslever, P., Simon,P., Rousseau, F. and Popieul, J.C. (2008), Using space windowing for a preliminary analysis of complex time data in human component system studies. Examples with eye-tracking in advertising and car/head movements in driving, Information Sciences, 178, Elsevier, 2008, pdf.gif Download paper

Kemper, S. and McDowd, J. (2006), Eye movements of Young and Older Adults while Reading with Distraction, Psychology of Aging, PMC, March, 2006, pdf.gif Download paper

Qian, M., Aguilar, M., Zachery, K., Privitera, C., Klein, S. Carney, T. and Nolte, L. (2009), Decision-Level Fusion of EEG and Pupil Features for Single-Trial Visual Detection Analysis, IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, 56(7), IEEE, July 2009, pdf.gif Download paper

Pollatsek, A., Reichle, E. and Rayner, K. (2006), Tests of the E-Z Reader model: Exploring the interface between cognition and eye-movement control, Cognitive Psychology (52), Elsevier, 2006, pdf.gif Download paper

Zelinsky, G. (2008), A Theory of Eye Movements during Target Acquisition, Psychology Review 115(4), NIH PMC, October 2008, pdf.gif Download paper

Sereno, S. and Rayner, K. (2003), Measuring word recognition in reading: eye movements and event-related potentials, TRENDS in Cognitive Sciences, Vol.7 No.11, Elsevier, November 2003, pdf.gif Download paper

Koorneef, A.W. and Can Berkum, J.J.A. (2006), On the use of verb-based implicit causality in sentence comprehension: Evidence from self-paced reading and eye tracking, Journal of Memory and Language 54, Elsevier, 2006, pdf.gif Download paper

Campbell, C.S. and Maglio, P. (2001), A Robust Algorithm for Reading Detection, PUI 2001, Orlando, FL, USA, 2001, pdf.gif Download paper

Appelbaum, L.G. ,Liotti,M., Perez III,R., Fox, S. and Woldorff, M.G. (2009), The temporal dynamics of implicit processing of non-letter, letter, and word-forms in the human visual cortex, Frontiers in Human Neuroscience, 3:56, doi: 10.3389/neuro.09.056.2009, 2009, pdf.gif Download paper

EEGLab and Tools

Delorme, A. and Makeug, S. (2004), EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis, Journal of Neuroscience Methods 134 (2004), 9-21, Elsevier, 2004, pdf.gif Download paper



Software and Information Resouces

EEGLab - http://sccn.ucsd.edu/eeglab

SPM - http://www.fil.ion.ucl.ac.uk/spm

LORETA http://www.uzh.ch/keyinst/loreta.htm


(updated 2013.07.30, nmg)