Multimodal Contactless Architecture for Upper Limb Virtual Rehabilitation

org


I. INTRODUCTION
Limited mobility is a prevalent dysfunction that is observed in patients suffering from neurological diseases such as Stroke, Epileptic Encephalopathy, Cerebral Palsy or Parkinson diseases [1,2,3,4].The importance of upper limb function rehabilitation is emphasized since upper limbs are used to manipulate objects and to interact physically in Activities of Daily Living (ADL) [5,6].
Studies present different types of implementations for rehabilitation that respond to different kinds of patients' needs [7,8,9] however, sometimes medical conditions do not allow the use of wearable devices [10].Contactless devices are an alternative for gesture recognition applications in healthcare [11] and permit the tracking of free and natural movements facilitating the user's mobility [12].Important advances have been made in contactless approaches that grant safety and accuracy [13,14].
Multimodal systems allow Virtual Reality Systems (VRS) to be implemented with multiple sources of information for both input and output so that the participant can have a personalized interaction with the system [15,16].Multimodal systems also consider multimodal feedback that consists of visual, auditory, and tactile feedback that can be combined to increase participant motivation and improve training effectiveness [17].
Literature shows different kinds of focus regarding the use of contactless devices, for example, the analysis of the contactless interactions of new users when they are learning and adapting [18].Some other research focuses on the impact of a specific device in the rehabilitation process such as Leap Motion Controller [19,20,21] or Kinect [22,23,24,25].Some other studies refer to how virtual rehabilitation using contactless devices reinforce motivation [26,27,28,29].
The main objectives of this work are: 1) to propose a multimodal contactless architecture for a virtual rehabilitation system for upper limbs; 2) implement a virtual rehabilitation system based on the proposed architecture; and 3) evaluate the acceptance of the VRS and the kinematic outcomes.
The rest of the paper is organized as follows: Section II presents the related works; Section III presents the architecture and experiments conducted.Section IV presents results and discussion, and finally Section V gives the conclusions.

II. RELATED WORK
Contactless devices for rehabilitation have been used for many years.Early research makes use of Kinect, and experiences have been carried out for various diseases such as stroke, cerebral palsy or Parkinson disease.
Huang [30] implemented recognition on arm movements.Therapists were able to adjust the rehabilitation movements based on the conditions of the participant.Pastor et al. [31] focused on increasing range of motion to improve functional use of the impaired upper extremity.They developed a game that requires patients to control a cursor on the screen by moving their hand.
Other research has focused on ADL, for example, the work in Adams [32] implemented activities for preparing meals with an avatar for recovery of upper extremities combining a virtual world and a Kinect™ sensor.
There is other research that has focused on active movements of the upper and lower limbs using a Kinect-based game system in addition to conventional therapy.The results showed that Kinect may have supplemental benefits for patients [33] [34].
In recent years, other contactless devices have become relevant in the virtual rehabilitation process due to their small size, accuracy, and low cost.Taraki et al. [35] presented the use of the Leap Motion Controller (LMC) for upper extremity rehabilitation to improve the joint range of motion, muscle strength, coordination, and fine motor functions of the hand and wrist in patients.The results showed quantitatively that www.ijacsa.thesai.orgLMC should be used as an effective alternative treatment option in children and adolescents with physical disabilities.
Wang et al. [36] also used six interfaced virtual exercises that are included in the LMC virtual reality system.The games focus on the improvement of dexterity.Their results conclude that LMC facilitates the recovery of the motor function and dexterity of a paretic upper limb.Khademi et al. [37] used LMC to implement the game of Fruit Ninja focusing on finger individuation for stroke patients.The results demonstrated significant correlations between the scores generated from the game and standard clinical outcome measures.
From the review of the literature, it has been observed the importance of the use of contactless systems for certain types of patients who cannot use wearable devices.Likewise, it has been found the effectiveness of therapies that make use of virtual systems as a complement to traditional therapies.Finally, given the different conditions that patients have, it is necessary to adapt the different types of feedback that patients need at the auditory, visual, or tactile level.Even though all of the related works propose different kind of implementations, it had not been found a generic architecture for virtual reality systems oriented to upper limb rehabilitation.

A. Architecture Proposal Methodology
The software architecture of a system is the structure that considers: 1) software components, 2) the externally visible properties of those components, and 3) the relationships between the components.Software architecture is important because it defines a set of constraints on the subsequent implementation and it focuses on component assembly [38].
For the architecture proposal, this work have adapted the methodology proposed by Parisaca et al. [ 39] considering only six steps 1) Identification of system quality attribute requirements; 2) Identification of architecturally significant requirements; 3) Design of architecture components; 4) Classification of components; 5) Validation of design decisions; and 6) Analysis and evaluation of software architecture.
The development of each step is described in section B.

B. Multimodal Contactless Architecture for Upper Limb Virtual Rehabilitation
Step 1: Identification of system quality attribute requirements.Upper limbs rehabilitation is important since it allows the use of hands to interact physically in ADL [5].Sometimes different kinds of medical conditions do not allow the use of wearable devices for virtual rehabilitation.This reason makes it necessary to consider contactless devices for gesture recognition.
Step 2: Identification of architecturally significant requirements.The functional requirements related to the architectural components are shown in Table I.Step 3 Design of architecture components.
Based on Dumas, Lalanne and Oviatt [40] work, we propose an architecture for contactless virtual rehabilitation systems that focus on multimodal feedback.The architecture has four components (Fig. 1): i) The Input Modality; ii) The Integration Committee; iii) The Output Modalities; and iv) The Virtual Reality System.The Integration Committee has three elements: i) Dialog Management, ii) Context User Model History, and iii) Output Modalities Fission.Input Modality refers to the contactless device that communicates to the Dialog Management.Dialog Management is in charge of identifying the dialog state that takes place in the VRS (the actions to communicate to the VRS and/or the messages to return through the Output Modalities Fission).Output Modalities refers to the auditory or visual feedback that gets the user.
Output Modalities Fission is in charge of returning the feedback to the user through a combination of modalities, depending on the Context User Model History (visual, auditory or other).
Step 4: Classification of components.From the functional requirements described in Table I we have considered: component A (Input Modality) as unimodal and component D (Output Modalities) as multimodal.
Step 5: Validation of the Design Decisions.The validation of the architecture design can be done with different techniques such as scenarios, questionnaires, simulations, www.ijacsa.thesai.orgmathematical models, or prototypes [38].In this work we decided to validate the architecture with a prototype.
For the prototype, the proposed VRS focuses on performing the coordinated actions of handling objects: picking up, manipulating, and releasing them in order to perform exercises to recover hand dexterity [41].The task of the VRS is to preparing a pizza.The participant must pick up a highlighted ingredient, (one by one) and drop them onto the pizza dough.For each ingredient placed correctly, the participant receives a point (visual feedback).The VRS shows whether it is a hit or a failure (visual feedback).Auditory feedback is also provided, in the case of a hit, a bell rings, and in case of a miss, an error horn sounds (Fig. 2).The VRS allows configuring the hand to be used (left or right) and the number of ingredients to be displayed: 5, 10, 15, or 20.
The VRS was developed in Unity with the contactless optical tracking device Leap Motion Controller (LMC).This is a small optical device with sub-millimeter precision oriented to gestural hand movement [13].
The architecture of the VRS is shown in Fig. 3.The Input Modality considers the contactless device LMC.The Output Modalities are implemented with visual and auditory feedback.We also show the interaction of the VRS states.From the user perspective the Decision state represents the person's attention, intention, and emotions.The Action state represents the hand movements.The Perception state is the recognition of the gestures and movements controlled by the LMC.In the Interpretation state, the data captured from the device is processed.
From the perspective of the VRS, the Computation state performs the fission process that allows the VRS to generate the feedback messages based on the context of the user.The Action state refers to the response to the user action in the form of visual and audio cues.The Perception state refers to what the user hears and sees in the VRS.Finally, the Interpretation state refers to new decisions that the participant will make.Step 6 Analysis and evaluation of software architecture.Two experiments have been performed in order to evaluate the acceptance and technical effectivity of the implementation of the proposed VRS.

Experiment 1
The participants were 106 healthy university students.There were sixty-eight males and thirty-eight females.The mean age of the participants was 20 ± 1.4 years old.The instrument was the User Experience Questionnaire (UEQ) [42,43], which measures six dimensions: (a) Attractiveness: attractive, pleasant, friendly, and enjoyable; (b) Efficiency: to perform tasks quickly, efficiently and pragmatically; (c) Perspicuity: easy to understand, clear, simple, and easy to learn; (d) Reliability: interaction should be predictable, safe and meet user expectations; (e) Stimulation: interesting, exciting, and motivating; (f) Novelty: innovative, inventive, and creatively designed.The scale ranges from -3 to +3.Values between -0.8 and 0.8 represent a more or less neutral evaluation of the corresponding scale, values greater than 0.8 represent a positive evaluation, and values lower than -0.8 represent a negative evaluation [43].
With regards to the protocol, first, the researchers explained the instructions for interacting with the VRS.Then each participant interacted with their dominant hand.Each participant interacted with 20 elements without a time limit.Finally, participants filled out the UEQ questionnaire (Fig. 4 shows the interaction of one student).www.ijacsa.thesai.orgThe second intervention was to evaluate the kinematic outcomes of the study.It focused on the number of hits the participants had when they dropped the ingredients onto the dough.With regards to the protocol, the children interacted with the VRS in ten sessions, first with five elements using their right hand and then with their left hand.Then they interacted with the VRS with ten elements using their right hand and then their left hand.The VRS recorded the number of hits in each interaction.

Experiment 1
Table II shows the values of the 6 UEQ scales, all of which have positive results: < 0.8.Fig. 5 shows that the highest value is Perspicuity, with an average of 2.068, which is considered to be Excellent.The perception of clarity is based on the comments obtained from the open-ended questions, which highlight the interactivity and ease of use, the intuitiveness of the game, and the feedback channels (visual and audio cues) that are available to the participants.The second highest value was Attraction, with an average of 1.987, which is considered to be Excellent.According to the opinions of the participants, they found the system to be friendly because, by having both visual and audio feedback, they could achieve the objective of the game.A second aspect mentioned was the use of the LMC for hand recognition, which has no physical contact with the user.
The Stimulation obtained a value of 1.703, which is a value between Excellent and Good.The participants found the interaction with the LMC interesting for moving virtual objects.They also highlighted the importance of sounds for hits and faults.In addition, since it is a scoring game, the users want to improve their performance.
Dependability obtained 1.649, which places it on the borderline between Excellent and Good.A requirement to meet this value is the correct functioning of the input and output modalities, which allows the user's expectations to be met in terms of the virtual hand behaving in the same way as the real hand (no delay in execution time).Safety has been guaranteed since the Leap Motion Controller is certified as being compliant with safety and electrical regulatory standards and has no contact with the user.
Efficiency scored a value of 1.517.The system was considered to be fast and efficient because the user's hand movements are reflected in real-time.However, during the tests, the users identified some occasions in which the hand was not visualized and the pizza topping did not end up falling onto the dough.
Finally, Novelty obtained a value of 1.401 which is considered to be as good.Novelty is given by the creative design of the game that refers to an activity of daily life such as preparing a pizza.The novelty is also given by the fact that the game seeks to be applied as a complement to motor skills rehabilitation therapies.The system captures the history of each participant using the time it takes to move each of the ingredients and the number of success/failures.This information allows performance over time to be evaluated.

Experiment 2
Fig. 6 shows the kinematic outcomes.The study analyzed hand dexterity by counting the number of hits per session.
For the interaction using both the right hand and the left hand with five virtual objects (Fig. 6(a) and Fig. 6(b)), there are different performance measures for each participant.We observed better performance in the interaction with the right hand.This is explained by the fact that, for the four participants, the dominant hand was the right hand.www.ijacsa.thesai.orgThese results show physical therapist quantitative data in manual dexterity regarding accuracy to capture the dexterity required to complete ADL.This work presents an adapted version of the architecture proposed by Dumas, Lalanne and Oviatt [40] by including only one device for the Input Modality: a contactless device.This decision is proposed based on the context of the type of patients with upper limb disabilities.Likewise, today, contactless devices have shown greater accuracy for data capture.This decision also reduces the complexity described in [40] for the fusion processes in multimodal inputs: data-level fusion and feature-level fusion.
The simplified architecture proposal considered only one input contactless device.This prevents having to deal with problems of noise when dealing with multiple signals.This work considers that upper limb gesture recognitions could be done with only one device since different studies have demonstrated its accuracy [13] [44].Feature-level fusion handles noise better but needs numerous data training sets before satisfactory performance can be achieved.

V. CONCLUSION
This work has proposed a Multimodal Contactless Architecture for upper limb Virtual Rehabilitation.This work has also implemented a Virtual Reality System based on the architecture for upper limb rehabilitation using the contactless device LMC.
The use of the proposed architecture has allowed the orchestration of four components: 1) the use of a contactless device for gesture recognition; 2) audio and visual cues for multimodal feedback; 3) an integration committee that performs the orchestration between the four components; and 4) the virtual rehabilitation system: Pizza-Game.The architecture focused on multimodal feedback.This study shows that this architecture could be useful for developers for VRS that do not require the use of complex and multiple devices for input modalities.The proposed architecture has considered specific constraints such as the use of contactless devices for patients that can not use wearable devices due to their medical conditions.This study had a few limitations.First, the study has only focused on commercial devices that have demonstrated their accuracy such as the Leap Motion Controller device.But the use of custom-made contactless devices whose accuracy has not been proven has not been considered, in that case, an architecture with more input devices is required.Second, since the sample size is small, one must be cautious with respect to the kinematics results obtained at a statistical level.Future studies should be carried out with a representative sample size at a statistical level.
As future work, it is planned to experiment with other contactless devices in order to compare accuracy and user experience.We also plan to incorporate some other devices for multimodal feedback.

Fig. 4 .
Fig. 4. VRS Pizza Game -Interaction.Experiment 2 The participants were four children with moderate hand disabilities.There were two males and two females.The mean age of the children was 12.25 ± 3.4 years old.The study followed the guidelines of the Declaration of Helsinki.It was approved by the Ethical Committee (reference number: 2008-0234).

Fig. 5 .
Fig. 5. Results of the six main areas of user experience according to UEQ, in qualitative intervals.

TABLE II .
RESULTS OF THE UEQ SCALES (MEAN AND VARIANCE)