Visual Exploration of Complex Network Data Using Affective Brain-Computer Interface

This paper describes the current state of the work aimed towards an affective application of BCI to the task of complex data visual exploration. The developed technological approach exploits the idea of supporting tacit and complex domain-specific knowledge acquisition during the examination of visual images built using large input data sets. The presented experimental research on the complex network data exploration process shows the capabilities of the presented approach through the analysis of a user's affective state estimation. explicit attention to all the aspects of the analysed scene by using some tacit knowledge and their own experience. In this case it is very difficult to identify all the aspects of the decision even by interviewing the expert. Both issues aggravate in cases of multiple "points of interest" to be discovered and need to be taken into account in order to make a correct conclusion. Moreover, considering these issues from the point of view of knowledge acquisition, both of them deal with the loss of some parts of knowledge. In the first case, there are pieces of knowledge which can be discovered by analysing visualised data. In the second case, a part of expert tacit knowledge is lost. To overcome these issues, a visualisation system should have information about the objects within the visual scene which are most important for the user within a particular task. Nowadays, brain-computer interface (BCI) appears to be a technology which allows for the augmenting of virtual reality with brand new input from the user (3). Often this technology


I. INTRODUCTION
Contemporary scientific tasks deal with a huge amount of complex data.These data sets present results of observations, analysis and simulation.The approaches to the handling of large and complex data sets are important issues of e-science [1].Usually, the main attention of an e-science project is focussed on the development of software tools for the processing of this data (simulation, data mining, etc.).Nevertheless, the final processed results are often quite complex and require the development of special user-centric data-analysis tools.The common approach to the development of such tools explores the idea of data visualisation within a 2D or 3D space.There are a lot of techniques of complex data presentation with a long history of development (see, for example, the brilliant works of Edward R. Tufte [2]).A strong push for the development of visual analysis technologies was given by the appearance of virtual reality, which enables advanced an interactive visual exploration of complex spatiotemporal datasets.However, the complexity of visualised data still makes cognitive analysis (i.e.coming to a particular conclusion(s) during analysis) a difficult task to perform.
Usually, cognitive analysis is focussed on several "objects of interest" to be analysed with greater accuracy (e.g.unusual structure or behaviour).Nevertheless, in the case of data complexity there occur several important issues with regard to the visual exploration of complex data structures.Firstly, it is often rather difficult to automatically identify the importance of particular elements of complex visual scenes.As a rule, experts in a particular problem domain usually fulfill this task with ease, but non-expert users often face this problem during visual data analysis.Secondly, even for experts it is sometimes quite difficult to focus consciously on every important object within the scene.Frequently, experts can perform analysis without explicit attention to all the aspects of the analysed scene by using some tacit knowledge and their own experience.In this case it is very difficult to identify all the aspects of the decision even by interviewing the expert.Both issues aggravate in cases of multiple "points of interest" to be discovered and need to be taken into account in order to make a correct conclusion.Moreover, considering these issues from the point of view of knowledge acquisition, both of them deal with the loss of some parts of knowledge.In the first case, there are pieces of knowledge which can be discovered by analysing visualised data.In the second case, a part of expert tacit knowledge is lost.To overcome these issues, a visualisation system should have information about the objects within the visual scene which are most important for the user within a particular task.
Nowadays, brain-computer interface (BCI) appears to be a technology which allows for the augmenting of virtual reality with brand new input from the user [3].Often this technology is used for navigation or similar purposes.Though considering the presented issues, one of the interesting abilities of BCI is the estimation of the affective state of the user that include, for example, engagement and excitement levels, or even high-level characteristics like arousal and valence [4], or workload and vigilance [5].This allows the use of BCI as one of the tools available within an affective computing approach [6].In this article the idea of using the affective state of experts, estimated by brain-computer interface for support of complex scientific data exploration, is discussed.We believe that such implementation can significantly enrich the virtual reality technology by enabling tacit knowledge acquisition and using them to make the process of data analysis more effective.

II. VISUAL EXPLORATION OF SCIENTIFIC DATA
In considering scientific data visualisation, several abstraction levels of data can be defined.
1) Raw data.These data present the initial values to be visualised.In particular, analysis of this data is the main goal of visual data exploration.
2) Semantic objects.Initial data can be arranged according to the structure of the investigated system, its elements and properties.The structure can be defined using terms specific to a particular problem domain.
3) Visual objects.Each object can have several visual representations.These representations may present images of a related natural object or visualise some characteristics of the www.ijacsa.thesai.orgdata.The data can also influence the behavior of the visualised object.
4) Visual scene.A complete dynamic visual scene which shows all the images related to the scene, analysed objects and presented data.
While constructing a visual representation of the data it is required to pass all the levels from 1 to 4. This can be done automatically (by software) or semi-automatically (e.g. by implicit identification of semantic objects during visualisation of software development).Nevertheless, the result which users can see and explore is a complete visual scene.It is interesting that, while analysing the scene, the user passes almost the same levels but in reverse.Firstly he/she perceives the scene (level 4) using visual stimuli.Then the scene is decomposed into the particular visual objects (level 3).They are classified and analysed according to the experience of the user.The experience of the user includes knowledge of a particular problem domain which allows them to recognise the correct semantic meaning of each object (level 2).Finally, the analysis of the visual images in accordance with their semantic meaning gives the user an insight into the initial data (level 1).And here, having this insight, the user understands the meaning of the data that was initially presented.That leads him/her to a particular conclusion, or way of reasoning or decision making.The idea of the presented work is to analyse the affective state of the user and to link it with the process of data exploration which is performed within these four levels of abstraction.
There are several scenarios that benefit from the implementation of the presented approach.Here are two examples which show a possible use: 1) Knowledge acquisition.Retrieving knowledge often becomes a difficult task as the knowledge may be very complex and contain tacit parts.For example, some experts make a decision using their experience without explicit attention or consideration of some statements that are considered as obvious.Analysing of domain-specific data could involve a lot of expert knowledge of different kinds.So it could be possible to track the interest of the expert and turn their minds to particular features of the visual scene during the following interview.
2) Experts' and specialists' training.By analysing the affective state of the expert during the exploration of training data sets, it might be possible to identify features which require attention.These features can be explicitly marked for educational purposes.Studying experts, decision makers, or even students could see which particular features are usually important within domain-specific data.

A. Virtual Reality Semantic Structure
Today, virtual reality technology can be considered as a powerful tool for interactive data visualisation.We try to exploit the idea of complex scientific data presentation within virtual reality (interactive 3D scenes), but the approach can be mapped with ease onto any 2D or 3D data representation.
Within the presented approach the core structures are semantically described as domain-specific objects which organise incoming data.Frequently within e-science tasks [1] the data represents the result of a computer simulation.In this case, the process of data exploration is tightly coupled with the process of simulation.The proposed approach uses VSO (Virtual Simulation Objects) concept and technology [7] which allows the organisation of the simulation and data analysis process around the domain-specific semantic structure of the investigated system.Within the VSO concept, the system is described as a set of objects which interact with each other.Each of the VSOs is related to some real-world object, which forms a real system.So the set of VSOs can be considered as an image of the system to be investigated using simulation.This concept claims the following requirements: 1) System's description.This should be considered as a structural model of the investigated system, containing the objects which are interacting with each other and with the environment.The environment can be described as a separate VSO or as a set of parameters.
2) VSO can be considered as a distributable set of knowledge which can be integrated within a processing system to make it support the simulation of particular objects.This allows us to make the system's description interpretable in two ways: a) the system should have sense within a problem domain (for the user); b) the description should be machineinterpretable (for the simulation process and data analysis).
3) Typical e-science tasks consist of three stages: modelling, simulation and result analysis.VSO concepts should present continuous technological and informational support for all three stages of this process.
The structure of the virtual simulation object can be considered as a tuple: where B is a set of available bases (spatial, temporal or group [8]); V is a set of values which can be defined on the bases from B ; Q is a set of quality metrics for values from V ; M is a set of models, which operates with values from V ; E is a set of interconnections between models.The system's description includes a set of VSOs.
A set of visual objects (images) can be associated with the VSOs set: This maps bases of the VSO onto 3D space and 1D time.Semantically, the system is described as a set of interactive objects . So the visual scene can be www.ijacsa.thesai.orgdescribed as a set () . For the purpose of scene analysis the following function can be defined: This allows us to identify the semantic object for any visual object.
During the process of visual exploration the user is characterised by the position and gaze direction 33 () pt  within the 3D scene.But, within a consideration of the task, a much more interesting problem is to identify the attention position (focus) The presented formal description allows mapping of abstraction levels 2-4 with an arrangement of user's attention during data exploration.Thus, it allows us to trace the user's exploration process through space, and visual and semantic objects.This trace forms the initial data set for an affective state analysis which is performed using BCI.
One important issue here is that in many cases it is quite difficult to estimate a single position of attention (especially concerning 3D space).There are several reasons which make this task more complex.Firstly, it is difficult to estimate the correct gaze direction precisely as there are errors in measurements.Secondly, even if the gaze direction could give a single point on the screen, the viewport of the user is somehow wider.Finally, if there are a lot of small objects on the screen, it is difficult to guess which one the user is looking at.This can be more troublesome if the 3D space is considered as the distance to the object to be taken into account.To overcome these issues several techniques can be used: where 0 d allows us to tune "wideness" of attention.1) 3D analysis.The process of interactive exploration within virtual reality gives us additional capabilities for gaze position estimation.For example, if the viewport of the 3D scene is spinning around some point and the user is looking at one object from different directions, it gives a better precision of attention position estimation as there exists a point of intersection of gaze lines.If the affective track is recorded and then aggregated using the weight function, the position or object at the intersection place will gather more scores.

B. Affective State Estimation and Mapping
Today, brain-computer interface allows us to estimate different parameters of a user's affective state [4,5].The popular approach is to use electroencephalography (EEG) devices for this purpose as they are easy to use and provide quite good opportunities for building BCI.The devices which implement this technology become lighter and easier to use (see e.g.Axio project [9]).
Within the proposed approach we try to join affective state estimation with the process of visual data exploration.During the exploration, a set of characteristics ( ) : tA   can be tracked.Here A is a set of available attention aims (possible focus),  is a set of BCI-based affective characteristics (engagement, excitement, etc.).It is possible to associate characteristics with different attention aims: spatial attention - .
The function () t  can be used to construct 3D spatial maps of affective state and estimation of attitude towards different visual and semantic objects.As the goal of affective state processing within the considered task is to analyse the user's attitude towards the data structures, the estimations (7) are usually more important.
One of the important things about the affective state processing is aggregation of the tracked states  to estimate affective characteristics of visual and semantic objects in accordance with attention track gathered as it was described earlier.Within the proposed approach, two types of analytical aggregation procedures are used.www.ijacsa.thesai.org 1) Weighted summation.This used equations ( 6) for every affective parameter.This allows us to score almost every object within the scene.So, the resulting score can be defined as: Where s is an object for score estimation; () G a t s  is a distance from gaze (as a 2D point or 3D line) to the estimated object which is used in weight function ( 6); () t  is a characteristics vector (7) tracked by BCI.

A. Hardware facilities
For the purpose of experimental research of the proposed approach, several pieces of specific hardware were used.The core hardware within the experiments was the Emotiv EPOC BCI [10] which uses EEG technology (see fig. 1a).Among other available features, this device allows us to estimate the affective state of the user including the values of the following characteristics: excitement, engagement, meditation and frustration.To perform the experiments two test beds were used: 1) A virtual reality environment was built using a 3D-wall (fig.1b) which allows us to observe the virtual environment using polarised glasses.To interact with the environment, the Kinect [11] interface was used to support movement within the 3D space.The wide screen of the 3D-wall allows us to use gyroscope sensors of the epoc to track the head position as a basic gaze-tracking source.
2) A PC-based environment was built using a generalpurpose PC.The 3D environment mouse was used for interaction.To track the gaze of the user, an eye tracking system was built using a Genius iSlim 321r camera (fig.1c), taking as an example the experience described in [12].To track the gaze direction, the ITU Gaze Tracker [13] was used.

B. Software implementation
The presented approach was implemented within the application for a visual analysis of complex networks (such as social networks, organisational structures, etc.).The software was developed using the Microsoft .NET Framework and uses the capabilities of the CLAVIRE cloud computing environment [14] developed at the e-Science Research Institute.The environment allows us to perform the complex simulation process in an automatic way and can present the resulting data simultaneously to the simulation by the use of Interactive Workflow (IWF) technology.This approach allows the joining of external data services or clients dynamically.Within the considered task external client is presented by the developed visual data exploration software.
The developed visualisation software presents the network in the 3D space.The network can be painted (both edges and nodes) and laid out according to predefined rules.The user is allowed to move "through" the network space and to zoom it in and out.Also, the user is allowed to switch the layout and colouring scheme, and to obtain further information on selected nodes.

C. Test case
For the purpose of the experiments, a network of sexual contacts used for HIV simulation [15] was visualised using developed software.The network for that case contains individuals (nodes) and their contacts (edges).During the experimentation process the user explores presented data within the 3D space by moving and zooming in and out of the presented visual structure.Fig. 2a shows a sample screenshot of the visualised complex network.Here, the network has a connection-based layout (nodes with more connections are displayed on the top).While the user was exploring the visual scene, the user's gaze was tracked.Fig. 2b shows the gaze track that was recorded while exploring the static view of the visual scene presented in fig.2a.Then, gaze track was transformed into a viewing heat map (see fig. 2c for the presented gaze track).This technique is often used within usability researches.It is supposed to show the places on the screen which attract most of the user's attention.The gaze track and heat map can be also constructed for a 3D space.Within the research they are implicitly constructed during the score computation.www.ijacsa.thesai.orgSimultaneously, to gaze tracking the values given by BCI are also tracked.Within the test case, engagement value is mostly used as it seems to be semantically closer to the interest level.Averaging these values can give an additional affective heat map (see fig. 2d).The affective heat map resembles the view heat map in shape, but it has differences in the heat value (as they are more than just a sum).
More interesting results can be obtained while analysing the affective state of the user within the relationship to the particular objects.In this case network nodes are considered as such objects.Figure 2e shows the distribution of affective state www.ijacsa.thesai.orgvalues (engagement) recorded in relationship to particular nodes (mentioned by id).The experiment shows that: a) there are "sessions" within the exploration process, because the user can return to the already viewed node showing a different value of affective characteristics (fig.2e shows 3 nodes with a single view session, which have very narrow distributions, and one node with a multiple sessionit has quite a smooth distribution); b) the distribution of BCI-tracked values within a session can be distinguished from one another.Results of affective state analysis allow scoring of viewed objects (nodes): fig.2f shows a visualisation of the same network with the scores for different affective characteristics (excitement, engagement, meditation and frustration).The value of characteristics is shown by the number of triangles of different colour orientated in four directions.

IV. DISCUSSION
Contemporarily, affective BCI is quite a new and developing technology which gives us the ability to interact with the user on a completely new level.While initially BCI was mainly used with medical issues or to support disabled people, today there are tasks where it supports, optimises, and even enhances the general user's ability [16].Moreover, this technology forces the appearance of completely new areas of research like neuromarketing [17] or augmented cognition [18].
On the other hand, as the amount of information available for analysis today is increasing more rapidly, new forms of data analysis are required.One of the approaches towards overcoming the problem is the visualisation of complex data using specific artistic techniques (see [2] and examples at [19]).However, the analysis process still remains quite complicated, as within domain-specific tasks the expert's knowledge or specific data-mining solutions are required to recognise some important features.
The idea that lies behind the developing approach is to apply affective BCI technology to support the data analysis process (e.g. to support the expert knowledge acquisition process).As there is a lot of tacit or complicated knowledge in the wide area of problem domains, the idea promises to be fertile and it can be used immediately: during the exploration or by analysing recorded tracks; within experts' interviews or within the study process; with data-based aggregation or for analysing the user state and skills.
Yet, the analysis of the idea uncovers a set of questions.Which part of the user's impression is caused by the data analysis, and which part is caused solely by the visual expressiveness of the image?What influences do the domain skills and experience of the user have?How should the user's current state and mood be taken into account?We are trying to answer these and other questions within our current research.

V. CONCLUSIONS AND FUTURE WORK
The work presented in this paper is devoted to the development of technology which augments the visual exploration of complex scientific data with affective BCI.The scope of the research lies between a set of developing areas such as BCI, affective computing, augmented cognition, human-computer interaction, complex data visualization, virtual reality etc.Within the solution the significant role is played by BCI technology.Today the area of applied BCI is presented by many solutions (see, for example, works [3][4][5][16][17][18]).Still the issue of affective complex data exploration using this technology is uncovered.
The presented approach is aimed at the implementation of a new of presenting complex data by leading the user to the most significant objects of visualised data.The developed technological solution and experimental research performed using the complex network visualisation application show the ability to use affective BCI within this task.
The work is still in progress.Upcoming future plans include a deeper analysis of time-based and session-based measuring of affective user state, development of general technology for virtual reality augmentation on a basis of performed experimental technology, and further experimental research for a more complete understanding of the affective data exploration process (including the questions mentioned earlier).
visual image of the VSO; 3D V S -set of visual images for values; 3D C S set of visual cognitive structure that extend basic VSO representation.The position of visual images can be described as the following function:

Fig. 1 .
Fig.1.Experimental facilities: a) Emotiv EPOC BCI; b) 3D-Wall and Microsoft Kinect; c) DIY eye-tracking system analysis could show affective characteristics induced by the object with more details.

Fig. 2 .
Fig.2.Experimental data: a) initial data visualization; b) view track; c) view heat map; d) affective heat map (engagement); e) distribution of measured value for top-viewed nodes (engagement); g) node scores visualization