Augmented Reality Application for Hand Motor Skills Rehabilitation

—The paper presents an augmented reality based solution for hand movement rehabilitation using visual and tactile feedback. The proposed approach to the rehabilitation process combines the effects on visual, auditory and tactile channels of perception and simulation scenarios. In order to develop a hand movement model capable of solving rehabilitation tasks there was studied the concept of immersive virtual reality. Original 3D models and scenes have been developed to simulate the basic hand positions and motor functions. To provide efficient hand motion fixation it is recommended to implement a mechanical position tracking system based on a sensor glove improved by using the resistive transducers. Augmented reality is implemented to inspire and motivate the user to perform the required exercises by generating the corresponding pictures. Personalized comparative analysis of the dynamics of the patient's initial condition and rehabilitation results help to study the motor function and restore everyday skills. The proposed solution allows achieving accuracy and adequacy of fingers movements sufficient to satisfy the requirements of medical rehabilitation applications.


I. INTRODUCTION
Application of Virtual Reality and Augmented Reality (AR/VR) technologies is a promising area of e-Health development in medical and social rehabilitation. Modern user interfaces are capable of simulating realistic scenes supporting the disabled people in the process of adaptation and health recovery. This is especially required for psychological rehabilitation aimed at overcoming the idea that it is impossible to achieve a positive result due to any treatment.
Despite high popularity of AR/VR technologies their practical application in the area of medical and social rehabilitation remains yet challenging. In order to provide stable and reliable results medical applications should consider the variety in perception of mixed reality scenes by different people. Therefore information technologies (IT) solutions for rehabilitation should consider the human factor and provide adaptability at the level of user interfaces.
In this paper there is presented a software and hardware complex based on implementing AR/VR technology to solve the problem of restoration of the hand and fingers function. It is proposed to improve the adequacy of the whole solution and resulting rehabilitation treatment efficiency by providing tactile feedback using a specifically designed sensor glove. A specialized solution was implemented to inspire and motivate the user to perform the required exercises by generating the corresponding 3D scenes.
In this paper the second section contains an overview of related recent developments; Section 3 describes hand visualization features and special aspects of hand movement simulation that require an original approach. Considering this motivation there was developed a model introduced in Section 4, which implementation by a sensor glove and probation are illustrated by Sections 5 and 6 correspondingly.

II. RELATED WORKS
Movement disorders observed in most cases of stroke consequences are one of the important causes of permanent disability, and are one of the global goals of continuous neurorehabilitation [1,2]. Moreover, as a rule, restoration of the function of the upper limb occurs at a later date, often remaining the only cause of the patient's disability.
Rehabilitation of the hand is a labor-intensive process. Often months and years of purposeful work lead to the restoration of only global movements, while functioning of the fingers and especially fine motor skills remain impossible, leading to a serious limitation of daily activity. An everincreasing amount of evidence shows that repeated, intense, skill-focused training increases recovery of the upper limb [3]. However, the transition to skills training is sometimes impossible due to the barrier of insufficient level of movements in the hand [4].
Introduction of new automated and robotic devices, game strategies for hand recovery has opened up new prospects for the recovery of paresis of the hand, thanks to the use of computer, virtual strategies, interactive visualization and activation of biological feedback [5,6]. At the same time the opportunities to solve this problem remain limited due to the high cost of specialized equipment and the lack of training methodology that considers the individual features of patients and medical cases.
In addition, among the existing robotic technologies for restoration of the upper limb, there are not enough methods aimed at restoring the distal parts of the arm, hand, and fingers.
The use of a sensory glove for the correction of fine motor skills of patients is a highly effective method in comparison (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 11, No. 4, 2020 52 | P a g e www.ijacsa.thesai.org with standard therapy after a stroke, affecting not only the level of disturbances, but also expanding the possibilities of everyday use of a paralyzed hand. In addition, this method is interesting for the patient, improves motivation for study through involvement of the patient's personality. Follow-up observation data suggest that the restoration of fine motor skills of the hand dramatically increases the level of use of the paretic limb, which in turn improves functional recovery.
Hand motion capturing and simulation system refer to a well-known family set of position tracking systems, which are one of the most innovative developments related to virtual reality [7,8]. Positional tracking captures the location of a hand in space and at the same second changes the scene of virtual reality in AR/VR goggles, depending on the person's posture.
Tracking human movements is usually implemented using built-in sensors and markers. Markers are attached to the body of a participant in immersion in virtual reality and transmit information about his movements with signals.
Mechanical tracking methods [9,10] are systems for tracking and displaying the position of the human body when moving it using mechanical means for tracking the position of the head and body elementsgoniometers, which are designed to measure the rotation angles of the joints and to determine the final positions, for example, the fingertip relative to the hand or the point in space in which the goniometer is placed.
The advantages of mechanical tracking systems include high accuracy, technical simplicity, simplicity of the calibration procedure, stability of the received data. Weaknesses of mechanical tracking systems include possible problems with reliability, possible discomfort when using (the need to wear gloves with a set of sensors and wires). The most prominent example of mechanical tracking systems is an exoskeleton.
The problems of AR/VR technologies application in practice including the sphere of rehabilitation are fully studied in [11,12]. To study the processes of social and domestic rehabilitation, virtual reality is recommended to be used with tactile feedback, as well as optical and resistive tracking systems. Combination of the process of social rehabilitation and tactile biological feedback allows achieving a positive rehabilitation result: restoration of domestic self-service skills.
Implementation of AR and VR scenes provide new opportunities to involve the user into the process. Benefits from virtual reality application to improve medical rehabilitation are described in [13 -16]. Using low cost VR devices such as head-mounted displays, special virtual scenes can be designed to assist patients in the process of re-training their brain and reorganizing their functions and abilities. It is noted that using virtual reality helps increasing the degrees of immersion and interaction.
Besides, movement skills obtained by training in the augmented reality can be transferred to the real-world environment [17]. AR/VR based rehabilitation systems demonstrate high efficiency by improving the user engagement and exercise performance outcomes [18]. Vibrotactile feedback is also well accepted by patients [19], and helps improving the solution efficiency.
Authors in [20,21] present the results of deep study of AR technologies application for shoulder rehabilitation. It has been observed that the use of AR technologies is promising and deserves future attention, but today the number of clinical studies conducted is low. In addition to this there is identified a request for advanced metrics for patient performance analysis. Both problems require new solutions for AR rehabilitation implementation in practice.
Contextual data visualization [22,23] is provided to combine several data sets to analyze multiple layers of a biological system at once. This approach is widely used for medical data processing, but can be easily disseminated for cyber-physical rehabilitation. It provides continuous interacting with the system, which helps optimizing the learning behavior of both humans and algorithms.
Another one trend of AR based rehabilitation improvement is concerned with gamification [24,25]. The design of wearable and tangible, game-based equipment, 3D models and software are a subject of a multidisciplinary study. Additional efforts are required to provide personalized game based rehabilitation techniques.
Considering the features of AR/VR user's perception is one of the key functional requirements of modern solutions in rehabilitation. Some ideas of monitoring and analysis of the user activity are presented in [26 -28]. Analysis of the event flows that trace the user activity can give the system a useful feedback indicating the rehabilitation efficiency. At the same time this information can be used to capture a combination of user's focus and context and provide personalization of medical care.
In order to develop a hand movement model capable of solving rehabilitation tasks there was studied the concept of immersive virtual reality. Immersive virtual reality, or virtual reality of immersion, is a non-material reality with such properties that allow subjects to clearly identify it [29,30].
This reality is generated in the human mind in the process of interaction with complex technical systems, e.g. virtual reality environments. It has its own unique space and time, logic, exists only while the user is "present" in this reality, and provides interactivity as the ability to respond to user actions. This reality presupposes a sensually-shaped space in which the human will acts, embodied in one of the images of virtual reality. In addition to this, virtual objects can interact with the real ones and overlap each other, which effect is identified as mixed reality.
Computer immersive virtual reality is characterized by a number of properties, such as intensity, interactivity, immersion, illustrative, and intuitiveness. The intensity of computer immersive virtual reality allows the user to concentrate and focus on a large amount of diverse information. Illustrative virtual environment depends on the quality and visibility of the information provided. The visual range, as in a computer game, should inspire and arouse interest. Intuitive property of computer reality characterizes the ease of perception of information.
Based on the described analysis there were identified the main challenges and principles of AR/VR solution www.ijacsa.thesai.org development for hand movement rehabilitation. The basic gap is concerned with a necessity to personalize the 3D scene and its dynamics to the concrete patient.
The scientific novelty of the proposed approach lies in the development of new approaches to the rehabilitation process in terms of combining the effects on visual, auditory and tactile channels of perception and playing-home scenarios, as well as an individual analysis of the dynamics of the patient's condition before the rehabilitation course, during the course of the rehabilitation and a comparative analysis of the achieved results.

III. HAND MOVEMENTS SIMULATION
Tracking of a hand movement is a subdomain of pattern recognition used to capture the spatial positions of the fingers and build their high-precision three-dimensional mappings in real time. A full display of the hand in virtual space is achieved by combining fingers tracking with determination of a spatial position of the hand. The main requirement for tracking systems is high intuitiveness and naturalness of the interaction, close to that of real-world objects.
As an assessment of the functional state of the hand, as a rule, some resulting indicators of movement are used. This can be, for example:  the distance between the tips of the fingers and the palm at the maximum possible flexion of the fingers;  the distance between the tips of the thumb and forefinger when they are closed,  the resulting grip force, etc.
However, the distance of the fingertips to the palm is the result of flexion of the fingers in individual joints, and the grip force is the result of the development of force moments in these joints.
Registration of movements in individual finger joints in order to obtain numerical estimates of their pathological condition has recently been increasingly used in clinical practice. Biomechanical parameters for research include:  angles of abduction of the thumb and little finger;  range of motion in the wrist and metacarpal joint of the thumb;  range of motion in the metatarsophalangeal joints;  bending angles of all fingers;  the angle of flexion of the wrist joint.
In the framework of this work, we consider a kinematic model of the hand, built according to the mathematical laws, while the action that induces the patient to move is familiar to him, necessary to restore skills due to the high immersiveness of the virtual environment.
The following features should be studied as a part of rehabilitation hand motion capturing system.
Hand positions should be mapped in VR scene. With partial visualization, virtual hands are the user's avatar and create the effect of being in a virtual environment, through a connection is created between the user and the setting, as well as the story itself.
Despite the apparent ease of creation, there are a number of nuances in the development of hands simulation in the AR/VR scene. Currently here are two main approaches to the process of taking an object with a virtual hand. In one case, when taking an object, this object is placed in a virtual hand in the desired position. The second option is to replace the virtual hand with the object itself.
It is assumed that the first option provides a greater presence effect, but it can be caused by the ergonomics of concrete AR/VR devices. In case of replacement the hand image by the grabbed object the focus is transferred to the object itself. After the virtual hand disappears the avatar suffers losing contact with it. Still this option is much easier to develop, since it does not require realistic visualization of the states of taking an object by hand and animation of multiple fingers positions.
When trying to create a photorealistic hand with partial visualization, it is still possible to create a "wax" model that will be unpleasant for human use. In this regard, the hands are often shown either in the form of peculiar gloves, or somehow stylized as a part of a setting (e.g. like the hands of a robot). Cutting the geometry is not a solution, even with animated styling.
To solve this problem there is introduced an "empty gloves" effect with visualizing a glove that takes the shape of a hand, but there is nothing inside if you look from the side of the cuff. The developers portray hands as transparent to solve the problem of the ephemerality of the virtual hand, justifying it visually. Such a solution practically does not affect the immersion.
While using virtual reality goggles, it is impossible to disrupt the synchronization between the position of the real and virtual hands. This feature affects the feeling of possessing the fingers, so if a virtual hand crosses any geometry in the scene, then it will go through it. If the hand is visually translucent, then this does not cause dissonance at the logical level.
Visualization of the shadow from the hand helps in determining its position in space, and also creates a connection between the hand and the virtual environment. Since the user associates virtual hands with him, this strengthens his involvement. This also applies to the active use of controller vibration as feedback when crossing geometry, taking objects, and other similar situations.
To assess fine motor skills and various types of grip, the Action research arm test (ARAT) scale is used, which consists of several sections: ball grip, cylindrical grip, pinch grip and large arm movements.
In order to implement it in rehabilitation training there were developed the corresponding 3D models of virtual scenes that contain primitive bodies and hand positions (see Fig. 1 to 3). www.ijacsa.thesai.org   On the basis of these models there was formulated and adopted the problem statement for a sensor glove design, which includes the requirements and limitations specific for hand movement rehabilitation.
Kinematic and dynamic parameters of the movement give a complete description of the mechanics of the motor function of the hand. Therefore it was proposed to implement the tracking system that is built on interdependent coordinates obtained from sensors. The sensors should be located on each phalanx of each finger of the hand, on the metacarpal bones and on the wrist joint, making up a single sensor system. As a result there was developed a hand kinematic model capable of simulating the separate finger phalanges and their coordinated movement for typical hand actions. To improve the realistic visualization and build the immersive virtual reality there was designed a set of 3D models of various hand positions associated with the kinematic model.
The resulting virtual environment can be implemented in AR and VR scenes of rehabilitation solutions.

IV. TIME MODELLING AND DYNAMICS SIMULATION
On order to provide meaningful and valuable 3D modeling of human body parts in medical applications it is necessary to provide the simulation of their dynamics characteristics. Dynamic processes include heart beating and breathing that introduce changes in human body organs positions, size and shapes. Liquids like blood, lymph, bile and water usually need additional efforts to make them look realistic.
Simulation of dynamic changes of human body parts are also required in rehabilitation solutions based on AR/VR technologies. Rehabilitation process usually consists in performing a number of exercises and IT solution is used to force certain patient actions and receive the feedback. In this case VR should be introduced to inspire the user by generating the corresponding picture.
Let us present the scene objects by i w that typically refer to the human body parts like finger phalanx. Each i w at time , ij t can be described by geometrical parameters of size and shape , ij g . Simplistically , ij g can be represented by a sphere. Therefore users' manipulation can be represented by an event (Boolean variable): This model allows formalization of the following problem statement: The system should motivate the user m u to make certain actions and thus generate the event flow , mn e , which leads to The target sequence of the required positions can be generated either by an expert or automatically on the bases of the current patient condition and health status. The quality of manipulation is characterized by minimum total deviation in position and time: Considering this goal there can be formulated a typical (target) event flow: As a consequence the quality of manipulation can be transformed to.
,. This means that we need to introduce the target event flow that inspires the user to perform the required actions at proper time and location that will lead to the proper result.
The example is given for a hand movement recovery in Fig. 4, 5. Possible spatial position of each hand fingertip can be described by a spherical surface with radius determined by patient capability. This sphere is simulated in 3D scene and visualized by VR device. The patient movement's feedback is captured by the sensor glove.
Initial capability is identified as a result of a simple exercise. Later the patient is inspired to make further movement by the sphere radius decrease in virtual scene. Looking at this process he tries to follow it and thus forces deeper finger movements. Periodical trainings demonstrate positive results. The proposed approach allows improving the patient involvement and introducing game strategies in rehabilitation process.
The resulting 3D models can describe all possible positions of fingers normal and with deviations. Comparing and coordination of these positions in VR scene and reality is performed by software. Getting feedback from the sensor glove allows calculating the necessary changes.
The interactivity of virtual reality suggests that in the process of interaction between the user and the computersimulated environment, the program and the person retain their state in anticipation of the response of the other party. The interactivity of immersive virtual reality clearly reveals the limitations of the virtual environment, which directly depend on the specifics of the reality with which it was copied.
The proposed approach allows incorporating game mechanics. Game logic analyses a consequence of user's performed manipulation, considering the history of previous actions on the case and real time events. It is proposed to calculate the total deviation time between the moments of expected actions and the factual actions executed by the user.  This indicator can be utilized either as a component of the summarizing grade, or as a separate parameter used by the rehabilitation system to adapt the exercise complexity.

V. SENSOR GLOVE DESIGN
From the point of view of 3D modeling, hands can be represented as a mechanism from the system of interconnected bodies with certain degrees of freedom of movement. Within the framework of this model, each bone can be represented by its rotation relative to those associated with it, regardless of the position and spatial orientation of the hand.
Various sensors were studied analyzed and tested. Inertial measurement units presented acceptable accuracy up to 2 and were selected for implementation. The developed prototypes are presented in Fig. 6. The proposed solutions are capable with the described above 3D models.
Several solutions for transmitting tactile feedback to the user were considered. A patient in designed gloves with biological feedback takes a virtual object in virtual space. At the same time, vibration sensors located at the fingertips of the gloves signal with a vibration that the simulated hand in virtual reality has reached the outer surface of the virtual object. www.ijacsa.thesai.org This means that the patient has applied sufficient force to further work with this object. The force applied by the user is a value derived by calculating from readings from resistive sensors.
In the event that the patient has not reached the external surface of a given object, vibration is not applied. In the event that the patient in virtual reality reaches the outer surface of the virtual object and continues further movement to compress the palm, the vibration will be given by an increasing effect until the compression stops.
Thus, the patient is given a signal that the exercise was performed incorrectly.
Comparing to the devices available on the market the proposed solution is capable of tracking the movement of each finger phalanx. This feature is critical for medical and social rehabilitation programs.

VI. IMPLEMENTATION RESULTS
To implement this concept in practice considering the features and possible functionality of the described above models and sensor gloves it was proposed to implement a training methodology of hand motion recovery.
To perform rehabilitation using the developed complex there was provided a specific procedure that contains two tests. All the actions are performed using the specifically designed stand (see Fig. 7) with EEG recording on going. The subject is sitting right in front of the table (hands with palms down on the  table).
First test is performed for check-up analysis and contains the operations carried out without the help of automated complex.
The work is done without the virtual reality (VR) device: the subject sees physical prototypes of objects. Motion algorithm requires grabbing the object from the stand and putting it to the "square" zone.
Second test supposes rehabilitation application. The work is carried out in a VR helmet: the subject sees the 3D models of the objects placed on the stand. The proposed solution allows achieving accuracy and adequacy of movements sufficient to satisfy the requirements of medical rehabilitation applications.

VII. CONCLUSION
AR/VR technologies application for hand movements' rehabilitation increases the efficiency of regular exercises due to higher user involvement in rehabilitation process and a possibility of procedures personalization.
Immersive virtual reality is implemented to inspire and motivate the user to perform the required exercises by generating the corresponding pictures. The proposed solution considers the specifics of hand simulation for better restoration of fine motor skills. www.ijacsa.thesai.org The proposed prototype of a hardware-software complex for hand rehabilitation using tactile feedback demonstrates the efficiency of AR technologies application. To provide efficient hand motion fixation it is recommended to implement a mechanical position tracking system improved by using the resistive transducers.
Next developments are planned to solve the problems of motion recording and patient support in the virtual reality environment. In addition to it the proposed solution is probated and tested in rehabilitation practice.