Intelligent Haptic Virtual Simulation for Suture Surgery

The aim of this study is to develop an intelligent haptic virtual simulation for suture surgery, enabled with an AI assistant. This haptic VR suture simulation mainly composed of three parts: visuo-haptic rendering for surgery, replica training for surgery, and AI assistant for surgery. In the simulation, a trainee surgeon can practice the suturing procedure using tactile touches in the stereoscopic 3D virtual environment. The simulation adopts a “precise haptic collision detection method using subdivision surface and sphere clustering” developed in the previous studies of authors. In addition, an expert surgeon's suture operations can be replicated to the medical trainee's simulation to experience it as it is. This method has the advantage of reproducing the experience of the expert’s ideal movements of surgical procedures. The data recording of the skilled surgeon's motions and operations are normalized into a form suitable for AI learning. The AI assistant distinguishes five typical types of suture surgery methods and learns the proper methods for various wounds using deep learning techniques, then it suggests the most appropriate suture method. The suture simulation can reduce the cost and time required for surgical training and eventually provide safe and accurate physical surgery. Keywords—Haptic; virtual simulation; suture surgery; artificial intelligence; assistant


I. INTRODUCTION
It is predicted that the advancement of AI (artificial intelligence) will lead to the disappearance of the medical profession, but rather the surgical AI techniques will become more popular. The recent technology trend is in the evolutionary combination of VR (virtual reality), AR (augmented reality), MR (mixed reality) and AI. Especially, AI winged haptic surgical simulation has the potential to revolutionize the healthcare industry.
One of the pioneers in the field of virtual surgical simulation is the MSim® 1 of Mimic Technologies. Other prominent products are LabSim®, EndoSim®, and TeamSim® 2 of Surgical Science. They provide a significant level of realistic virtual surgery simulations with haptic feedback. It would be also notable the CardinalSim 3 of Stanford University which is an immersive simulation environment developed for the preoperative rehearsal of complex surgical procedures. The recent trend is to combine artificial intelligence technology with these virtual surgical simulations to make them more convenient and useful. However, those commercial products are very expensive and require specific hardware.
The goal of this study is to develop a low cost and efficient suture surgery simulation system equipped with an intelligent assistant for surgeons to practice sufficiently prior to actual procedures. The system provides a realistic medical training environment for suturing operation in a 3D virtual reality environment. It also allows for experiencing the actual suturing procedure repeatedly using haptic devices. In addition to tactile sensation, the usage of a 3D VR headset gives the visual part feel realistic. Moreover, the trainee can acquire the surgical skills of the expert by reproducing the expert's surgical motions. If the training results are not satisfactory, the trainee can profit the calibration and the feedback from the surgical assistant AI. This paper will describe the core technology for the development of an intelligent suture surgery training simulation using haptic VR technology. In order to develop a suture simulation system, precise graphics simulations that realistically reflect the physical phenomena applied to the virtual body are required. There is also a need for precise haptic simulations that provide realistic tactile sensation when the surgical tool makes contact with virtual tissue. To deliver smooth, sophisticated tactile feedback to users, an update rate of 1000 Hz or higher should be maintained.
Note that the 3D (3-dimensional) stereoscopic visualization can affect the performance of intuition, accuracy, and immersivenesss of the surgical training simulation [1]. In addition, it would be very effective if a trainee surgeon could acquire the skills of an expert surgeon by mimicking an expert's surgical behavior exactly. Moreover, if the trainee can receive personalized corrections and feedbacks from the AI assistant that helps to learn the suturing, the educational improvement effect will become amazing.
The rest of this paper proceeds with a short survey of related works. Then, a brief description of the haptic virtual simulation for suture surgery is presented. Next, the details of each part of the simulation are elaborated. Lastly, the conclusion and future work are followed.

II. RELATED WORK
Haptics technology allows users to perceive tactile sensations regarding force and movement. Haptics plays an important role in simulations in virtual environments, such as 55 | P a g e www.ijacsa.thesai.org in virtual surgical training or haptic gaming [2]. When a haptic device collides with a virtual object, the haptic device performs a haptic rendering process by calculating the reaction force of the moment [3]. Then it provides haptic feedback to users, thereby enabling users to feel the touch. Nevertheless, the challenge of accurate and realistic haptic feedback requirements for high-frequency updates (1000 Hz) is always an obstacle to overcome.
Recently, many studies have been conducted applying haptic virtual reality technology, however, the study of training systems using haptic virtual reality technology goes back more than 20 years [4]. Various core technologies for effective haptic virtual reality simulations are being studied in order to provide training systems for practicing before actual operations. One of the early studies is a simulation developed to perform an elaborate work using a haptic device. This study developed an educational program for the telemanipulation of carbon nanotubes using haptic feedback and a 3D display [5].
The 3D visualization method using a VR headset is expected to produce superior results in user experience than the 3D visualization method using a flat monitor which is commonly used. The HMDs (head-mounted display) are also advantageous for acquiring cognitive skills related to remembering and understanding spatial and visual information [6]. There is also a telehaptics application for haptic probing of remote objects [7].
In the field of medicine, diverse haptic virtual reality surgery simulations are being studied in order to provide medical students and surgeons practicing before the actual surgery. One of the remarkable early studies for surgery training simulations using haptic devices and virtual reality technology can be found in 2007. It is a simulation-based training in minimally invasive surgery (MIS) which allows the trainee touch, feel, and manipulate virtual tissues and organs while viewing images of tool-tissue interactions on a monitor as in real laparoscopic procedures [8]. Also, a dental anesthesia training simulation is developed based on anatomical data. In this study, haptic technology is used for dental treatment technology education [9]. One of the recent studies on haptic virtual simulations for surgery proposed a solution to incorporate a surgeon's sensations (haptic factors, visual factors, and hearing factors) during practical surgery into the training system to achieve a higher level of immersion. This study demonstrated that haptically enabled training simulations combining these multisensory data would be able to provide a more immersive and effective training environment [10]. Relevant to cognitive science, a study on the HVDT (haptic visual discriminant test) [11] using haptic virtual reality technology can be noted. This work presents a possible integration of vision and haptic perception of physically or mentally disabled children [12].
There is no doubt that haptic virtual simulations have a great impact on preoperative medical training. However, even if presenting and dealing with 3D objects in a virtual environment represented by a 2D monitor, the position of the object on the 2D monitor can be different from what it is in real. The reasons are that there can be distortion on the 2D screen or a lack of perspective. Because of this problem, there is a limit to recognizing the position of objects through a simple 2D plane monitor. In order to practice the operation of the correct movement, intuitive, accurate, and immersive awareness of the operation position in the VR environment is needed [1].

III. INTELLIGENT HAPTIC SURGICAL SIMULATION
In the simulation, instead of interacting only with the suture site, it provides a simulation that can enter the suture site from the actual human model. Therefore, the practitioner can perform the suture surgery experience more realistically. In this way, the simulation can be easily extended to other parts of the human body. Note that the simulation can allow trainees not only perform the suture surgery experience more realistically but also provide extensibility to surgical training of various parts of the human body. In addition, the simulation of the suture surgery can also provide the ability to change the surgical tool to the shape of the surgical tool used in actual suture surgery.  The design and implementation of each part of the haptic virtual simulation will be discussed in detail in the following sections.

IV. VISIO-HAPTIC RENDERING FOR SURGERY
The suturing is one of the basic techniques needed for patients with open wounds. There exist so many suturing methods. In this study, we distinguished five types of suture methods. They are:  A suture operation can be broken down into detailed steps. There is an excellent study on the analysis of a suturing and knot-tying task [13]. This work adopted for the design and implementation of the suturing simulation. Fig. 2 illustrates the process of suture surgery, step by step, and the process of each step. By learning the detailed process of suture surgery as shown in Fig. 1, it is possible to gain experience in the suture operation process of practitioners and induce effective learning effects.
It is notable that the most important implementation issue for visuo-haptic rendering is "precise collision detection". In order to prevent the progress of abnormal experiments due to the phenomenon of unrealistic penetration of rigid objects during the movement process, all of the objects must be set up to be a "collision object". In addition, the simulation adopts the "haptic collision detection method using subdivision surface and sphere clustering" developed in the author's previous studies [14], [15]. This algorithm can provide more precise haptic sensations. The implemented "precise collision detection" algorithm considers the following parameters: 1) The graphical tightness of bounding spheres.
a) The number of triangles.
b) The average area of triangles.
c) The number of bounding spheres. d) The collision detection time.
e) The haptic update rate. In order to give surgeons a realistic and immersive experience of surgical simulations, the Touch haptic devices (former PHANToM OMNI® haptic devices) 5 and Oculus Rift VR headsets 6 are used.

A. Replica Training through File
After recording the surgical procedure of the skilled medical expert in a specific file, it is restored to the simulation of the trainee so that it can be experienced as it is. This method allows the practitioner to learn the surgical motion step by step by reproducing the data recorded all the motions of the most ideal surgical simulation. This method has the advantage of allowing repeated training by imitating the expert's know-how.

B. Replica Training in Real-Time over the Network
This method allows the trainee surgeon to replicate and experience the surgical procedures of the expert surgeon in real-time. Professional work such as surgical operations can be www.ijacsa.thesai.org applied with the knowledge and sensations or know-how from the accumulated experience of a skilled person. Things are difficult to teach by word or writing and have characteristics that must be learned through hands-on experience. In this study, the surgical operations of a skilled expert can be imitated by medical trainees through network transmission. If the accumulated experience and know-how of skilled practitioners can be delivered through haptic feedback using haptic devices, the training effect of the practitioners will be much higher.
In order to implement real-time simulation via network transmission, client-server socket programming was used to improve communication between the trainee and the expert for the immediate copying of remote operations. The real-time synchronization through the network takes more time to be transmitted and received from a remote place. Therefore, the protocol selection at the transport layer should be made in consideration of the problem of latency of the response to haptic feedback. Protocols in the IP based transport layer include TCP and UDP. TCP is a connection-oriented protocol, which includes establishing a connection and performing flow control and retransmission protocols. The UDP protocol has a disadvantage that data loss is not known because retransmission and flow control is not performed. However, the TCP protocol is not suitable for applications where high speed is important. It is well-known that using UDP rather than TCP is suitable for real-time systems. Therefore, the real-time networked experience copying system uses UDP protocol instead of TCP because transmission speed is very important.
We developed a prototype system for replicating an expert surgeon's surgical sutures to a trainee surgeon. Fig. 3 presents an example of the replica training.

VI. AI ASSISTANT FOR SURGERY
We are developing a haptic VR simulation system that allows users to learn the suture technique appropriate for the characteristics of wounds. In order to make the simulation intelligent, we designed an AI (artificial intelligence) assistant which can recommend an adequate suture method for each specific wound. In addition, T system is designed to feedback on the results of previous training, and recommend an appropriate suture technique using AI technology.
In order to provide correct surgical information according to the level and characteristics of the trainee (left-handed, surgical-related physical characteristics, etc.) based on the proposed suture method through AI for a given wound in the surgical environment.
The method of suture depends on:  the type of wound,  the thickness of the skin,  the degree of tension,  the degree of suture scars,  the degree of bleeding, etc.
After arranging data recorded by the skilled surgeon based on the above five criteria, a deep learning model was designed for distinguishing a variety of suture surgery methods for recommending an optimal suture method.
An evaluation standard is established to measure the trainee's proficiency based on:  the accuracy judgment and  the execution time of the suturing for evaluating the results of the trainee's suture.
The system measures the proficiency of the trainees according to the evaluation criteria, feeds back the parts deemed insufficient in the simulation and recommends additional suture training for further study. Fig. 4 is the wound image processing model that predicts the kind of wound. A collection of wound data is normalized into a form suitable for the deep learning model for the classification. The wound image data are preprocessed through dimension reduction and color correction. The model also includes some traditional feature extraction and labeling by assigning a unique index to each label using the one-hot encoding.
The AI assistant recognizes the incision wound image and recommends the optimal suture for the desired closure of the incised wound. Among many machine learning techniques, the best performing machine learning technology for image recognition is CNN (convolution neural network). The CNN network is constructed by extracting features through convolutional computation in computer vision, applying activation functions to the configured feature maps, and constructing the active maps and passing the pooling layers through the active maps. Fig. 4. Wound Image Processing Model. www.ijacsa.thesai.org An AI learning model was developed for the proper classification of wounds. Part of the data preprocessing and optimization for proper classification of wound images are the most important core goal of the research. In order to achieve this core goal, data analysis on the incision wound images should extract the characteristics of the wounds for classification. In addition, it is required to build an AI model for wound image training and control overfitting and hyperparameters that appear when the real data is trained and verified. Finally, the optimization process will generate a model that is best suited to accurately classify the cause of the wound for new wound image data input.
A brief description of the AI modelization is presented in the following three steps:  Data collection and analysis for wound images  Preprocessing for wound images  AI model optimization for wound images

A. Data Collection and Analysis for Wound Images
The first step in developing an AI application is to collect good data. In addition, special attributes or feature information that can be extracted from the wound image will be secured through the cooperation of a university hospital.  what the level of wound contamination is, and  how the patient's age or health is.
The classification of detailed features and attributes of the data is directly related to increasing the accuracy of wound data classification by extracting and reflecting detailed features when the data is actually input.

B. 4.2 Preprocessing and Correction for Wound Images
Based on the collected wound data analysis, preprocessing can be applied for the training data. The training data are preprocessed and further classified into detailed attributes in the appropriate form for the AI model.
Many studies are performed for recognizing wound images [16], [17], [18], [19], [20]. The correction of medical images is the most important process to improve the accuracy of the model.

C. 4.3 AI Model Optimization for Wound Images
The most important part is to build an optimal AI model for classifying suture methods. The AI model should play an appropriate role in extracting features from the image data, and the final result label should be one of the types of suturing methods illustrated in Table I. During the model construction process, the components such as hyperparameters and parameters that make up the AI model are adjusted and optimized to best fit the training data. In addition, the model optimization is performed by additionally applying a technique for eliminating the influences caused by similar images or a technique for preventing overfitting. Through this process, the AI model will be optimized for the wound images, which is the training data, and it can classify the type of wound data with high accuracy.
After the preprocessing, the wound image data are entered into the CNN model presented in Fig. 5.

VII. DISCUSSION
Surgical simulations have been studied for over 20 years and now fully demonstrate their effectiveness. There is no doubt that by presenting a virtual surgical simulation as a stereoscopic 3D objects using a VR headset. the intuitive operation performance has been improved and especially operational accuracy has been greatly enhanced [1]. The most difficult part of virtual simulations is the graphical rendering of it. Precise rendering of graphics results in precise haptic rendering. Advances in graphics hardware enable realistic visual rendering. However, haptic rendering is still very different from realistic touch, and the precise and realistic haptic rendering has room for improvement.
Recently, mixed reality technology has been in the spotlight. I would like to emphasize that the simulation will become much more useful if it is converted to a mixed reality environment. In this case, the suture simulation can be useful not only for suture training but also for actual surgical procedures. In particular, the AI assistant for surgery can recognize the wound in real-time, suggest an appropriate suture method for the wound, and guide the suture procedure, etc.
VIII. CONCLUSION In this study, a simulation for learning and evaluating suture surgery using haptic VR and AI technology is developed. In order to provide medical trainees with realistic visual and haptic effects, it is very important to obtain accurate physical properties of the body and the precise and stable presentation of touch. Therefore, a "precise collision detection algorithm using subdivision surface and sphere clustering" that was proposed in the author's previous work is implemented. www.ijacsa.thesai.org Moreover, in the simulation, an expert surgeon's surgical procedure can be replicated to the trainee surgeon's simulation for mimicking the suturing. This method allows the trainee to learn the ideal motions of the expert surgeon.
The AI assistant for suturing learns the wound dataset and corresponding labels of typical suturing methods in consideration of the criteria of wounds, such as the type of wound, the thickness of the skin, the degree of tension, the degree of suture scars, and the degree of bleeding, etc. After arranging the wound data recorded by the skilled surgeon in those criteria, the AI model can learn the wound image data and their corresponding suture methods using deep learning techniques. Then, the AI assistant can recommend an optimal suture technique for a given wound.
It is expected that the simulation can allow for reducing the cost and time required for suture training and ultimately provide safe and accurate actual surgery. It is also advantageous that the learning curve of the trainee can be sharply increased by allowing them to experience the ideal surgical movements and procedures. In addition, the system can be improved for collaborative training, remote surgery, cooperative surgery, and remote cooperative surgery, etc. The results of this study can be used not only in the medical field, but also in all fields such as education, games, entertainment, and industry, etc.
Future studies will be focused on the evaluation of the developed suture simulation as well as the AI model for improving the efficiency of the suturing and on the provision of personalized feedback for enhancing the proficiency of the surgeon. In addition, the transformation of this intelligent haptic virtual simulation for suture surgery into a mixed reality environment will be conducted.