The NeuroRehabLab Software Tools for Research and Development of Interactive Systems

The NeuroRehabLab has been working for years in the development of Virtual Reality, Interactive and Serious Games applications. Most of this work has been done under the "RehabNet: Neuroscience Based Interactive Systems for Motor Rehabilitation" project (303891 FP7-PEOPLE-2011-CIG). In the process we have developed multiple prototype system applications relying on novel technologies ranging from body movement tracking, interactive surfaces, eye controlled, biosensors, to brain computer interfaces. 

This endeavor, however, is rather technologically complex. For this reason we designed the RehabNet architecture [1], a simple yet scalable infrastructure that proposes an inclusive approach towards an open and distributed architecture for 'in-home' neuro-rehabilitation and monitoring by means of non-invasive ICT. 

Our software solution natively support at the moment of writing this article, the following hardware devices: 

Electrophysiological

• Emotiv EPOC neuro-headset is intergrated for acquiring raw EEG data, gyroscope data, facial expressions and Emotiv’s Expressiv™, Cognitiv™ and Affectiv™ suite;
• Neurosky EEG headset is supported for raw EEG acquisition and eSense™ meters of attention and meditation;
• Myoelectric orthosis mPower 1000 (Myomo Inc, Boston, USA) is supported, providing 2 EMG channels and adjustable levels of assistance;

Kinematics

• Microsoft Kinect v.1 and v.2 is natively supported either by the Microsoft or OpenNI drivers;
• Nintendo Wiimote;
• Android Phones;
• Leap Motion;
• Head tracking with Oculus Rift and Vuzix iWear;

Eye tracking

• Tobii T120;
• Tobii EyeX;
• Eye-Tribe;

Other

• VRPN and OSC protocols are supported for the connection with any device (e.g. Vicon’s tracking, 5DT data gloves) or software supporting it (e.g.OpenViBE BCI software)

This is all achieved by a single software, the RehabNet Control Panel (RehabNet CP). The CP runs on a windows machine and acts as a router for all of these devices and broadcasts their data through a UDP socket connection. This approach allows us to keep the "content", i.e. virtual environments, games, or interactive applications, hardware and platform independent. Thus, the data is streamed from the CP to any application that is able to read the RehabNet protocol. This is a text protocol based on the following structure:

[$]device_type,[$$]device_name,[$$$]data_type,data_name,X,Y,Z,P;

an example for kinect data would be:

[$]tracking,[$$]kinect,[$$$]body,rotation,0,1,0,-1.629207E-07;

Extended functionality and device support is achieved via this UDP protocol that is used by a number of companion applications.

Analysis and Tracking System (AnTS): AnTS is quite simple to use camera based tracking system. It runs on Windows and Linux using Qt and Intel's OpenCV library. The main concept behind it is that you just add a number of filters that will sequentially process the video input, and the tracking process is performed on the resulting filtered images. This makes it extremely versatile and programming free. You can change filters and parameters in real time, so you can play with the system and then save your configuration once it is tuned for your purposes. In short what it does: 

• General purpose image-based tracking;
• Augmented Reality 3D tracking;
• QR code tracking;
• Mouse emulation;
• Supports PlayStation Eye;
• MS Kinect and video for windows compatible cameras;
• Perspective and distortion correction;
• Tracking information is sent over TUIO and UDP sockets using a custom defined protocol or the RehabNet protocol.

RehaMote is a mobile app for smartphones and tablets running Android OS for transmitting (via UDP) the available sensor data of the phone to the RehabNet CP or any other software compatible with the RehabNet protocol. Additionally, RehaMote can receive data via UDP for bidirectional communication of the phone with the CP or a Virtual Environment.

FaceAPI client: A too that provides real-time face tracking (gaze position (on the monitor), head position, head orientation) and data streaming via UDP based on the FaceAPI library.

BITalino client: BITalino is a biosignal acquisition device supporting sensors for electrocardiography (ECG), electromyography (EMG), electrodermal activity (EDA), accelerometer, and ambient light.

Bioplux client: Bioplux is a high end biosignal acquisition device from PLUX supporting sensors for electrocardiography (ECG), electromyography (EMG), electrodermal activity (EDA), accelerometer, BVP and Respiration.

MS Kinect v2 client:  is a MS kinect v2 RehabNet client that sends skeletal information using the RehabNet protocol.

In addition, to all of these sensing technologies, we have developed also some simple tools to ease development and testing. These are the RehabNet Data visualizer and Data logger, simple tools for visualizing graphically, logging and replaying incoming UDP data streamed in the RehabNet protocol.

NeuroRehabLab is making all of this is available to the community in a dedicated website for our tools (http://neurorehabilitation.m-iti.org/tools). Visit and check it out!

[1] Vourvopoulos, A., Faria, A. L., Cameirao, M. S., & Bermudez i Badia, S. (2013). RehabNet: A distributed architecture for motor and cognitive neuro-rehabilitation. In 2013 IEEE 15th International Conference on e-Health Networking, Applications Services (Healthcom) (pp. 454–459).