Mobile Devices Based 3d Image Display Depending on User's Actions and Movements

—Method and system for 3D image display onto mobile phone and/or tablet terminal is proposed. Displaying 3D images are changed in accordance with user's location and attitude as well as some motions. Also 3D images can be combined with real world images and virtual images. Through implementation and experiments with virtual images, all functionalities are verified and validated. One of applications of the proposed system is attempted for demonstration of virtual interior of the house in concern. User can take a look at the inside house in accordance with user's movement. Thus user can imagine how the interior of the house looks like.


INTRODUCTION
There are strong demands of 3D image display onto mobile devices and or tablet terminals of which virtual 3D images are changed in accordance with users' movements.For instance, virtual interior (in house) 3D images are changed according to users' movement when users wear Head Mount Display: HMD on which the created 3D images are displayed.Then users can imagine interior images designed by architects.
OS of iPhone: iOS is well described [1], [2] together with 3D programming [3].On the other hand, game and graphics programming under iOS and Android OS environment for OpenGL of computer graphics software is available [4].There are a variety of software tools and library software for developing programs under Apple OS environment [5].Also video programming can be done on iTune [6].Motion detection and capturing can be done through sensor fusion on Android devices [7].Thus mobile phone, tablet terminal based 3D image display depending on user's actions and movements can be created.
The following section describes the proposed method and system for displaying 3D images which depend on users' action and movement onto mobile devices and tablet terminals followed by implementation and experiments.Then conclusion is described with some discussions.

A. Proposed System
Illustrative view of the proposed system is shown in Figure 1.Illustrative view of the proposed system User wears Head Mount Display: HMD.User also has mobile device or tablet terminal which has GPS receiver, WiFi communication capability, and accelerometer together with gyro and magnetic compass.Therefore, users may move and may change their attitude together with some actions.Users can see 3D images displayed onto screen of mobile devices or tablet terminals as the users in the virtual studio which is shown in Figure 2. If users move in the virtual studio, then the displayed 3D image is changed in accordance with the movement.Also when users look to the right and left as well as up and down, then 3D image is changed according to the look angles defined in the Figure 3.  Users' action and movement can be detected with gesture recognition in the input controller which consists of accelerometer and gyro sensors.Recognize user movement as the input for controlling the view of 3D objects.Accelerometer sensors are using to detect user's body move and front camera for detecting user's head movement.
Indoor and outdoor 3D images can be created in accordance with the users' action and movements.These data are controlled by the main controller which includes front camera, GPS receiver, WiFi of equipments.
Main controller has video output capability.Therefore, created 3D images can be displayed onto mobile phones and or tablet terminals together with HMD.Output Video is displayed into the screen of a HMD attached on user glasses.The view is changed base on the user's movement.

A. Attitude Detection
Accelerometer data, usually, too rapidly changed.Therefore, some low pass filter is needed for detect relatively smoothly changed attitude as shown in Figure 5.If the iPhone and iPad is inclined 90 degrees, then x and y axis is exchanged as shown in Figure 6.If users change their attitude in pitching direction as shown in Figure 8, then heading angle can be estimated with accelerometer data.
Triple Axis Accelerometer (x, y, z) data can be transferred from iPhone to LIS302DL with communication through Low Level Accelerometer API: LLAAPI.These data should be filtered to remove shaking effect (using low pass filter).Acquired and processed attitude data (Red: x, Green: y and Blue: z directions, respectively) Fig. 6.
Pitching motions of users' attitude changes www.ijarai.thesai.orgBody turn around, head look to the left-right can be detected with direction sensors (azimuth).Also head look down-up can be detected with direction of gravity (altitude).
Although direction of gravity did not tell the direction, direction can be determined by using Gyro Compass or GPS for outdoor case.For indoor case, we can use Compass (magneticHeading).Head movement (left-right) can be detected by using captured face image from rear camera, then using image processing for determine the movement.

B. Detection of Walking
The following three methods are attempted for detection of the linear movement: 1) Double integration of the acceleration to measure speed and then the distance. 2

) Combination of Gyro Acceleration, Compass and Accelerometer to find Linear Movement 3) Using DeviceMotion Data (DeviceMotion is part of CoreMotion framework, a new framework on iOS as combination of Accelerometer & Gyr0)
There are four types of motion on DeviceMotion: Each type of motion has different response for 3 different states (Stop, Walking, Rotating) Using the response of deviceMotion.rotationRatethen we develop our new method for user walking detection.
Walking can be detected by estimating cycle on accelerometer sensors signal.Figure 8 shows typical accelerometer output signals when users stay (Stable), move forward (Walking) and change their attitude in roll, pitch and yaw directions (Rotation).In the figure, left column shows the motion of users' acceleration while right column shows the motion of users' rotation rate (the first derivative).It is much easier to detect their motions by using rate data rather than the acceleration itself.
Rotation can be detected by using the motion of users' rotation rate in directions of roll, pitch and yaw angles.Rotation angles in direction of roll, pitch and yaw directions are estimated with the motion of users' acceleration data.Meanwhile, walking can be detected by using the motion of users' rotation rate.All of rotation rate in directions of roll, pitch and yaw directions is relatively small in comparison to that of the motion of users' rotation.
When users do not move, then accelerometer data shows stable while if users change their attitude in roll direction, then accelerometer shows relatively large changes in the corresponding direction.On the other hand, when users are walking, the accelerometer data shows small changes.Therefore, it is possible to detect users' walk.

C. Control Camera Position, Orientation of 3D Space in Computer Graphics
In order to create 3D Computer Graphics: CG images which depends on users' action and movement by frame by frame basis, users' position, action and face angle (head pose) has to be detected together with determination of camera position and looking angle, users' orientation of line of sight, and relation between camera and 3D object.Figure 9 shows the process flow of controlling camera position, orientation of 3D space in Computer Graphics: CG.With the GPS receiver (Outdoor) and wireless LAN access point or WiFi access point (Indoor), users' location can be estimated.Using acquired users gesture with front camera mounted on mobile devices and/or tablet terminal, camera orientation in 3D space of CG can be estimated.Also users' head pose can be estimated by using the acquired image with the front camera.www.ijarai.thesai.orgIV.IMPLEMETATIONS

A. Development Environment
Development environment is shown below, • MacPro with OS X (Mountain Lion)  Due to the fact that iPhone and iPad has a capability of gyro and magnetic compass and also has a capability to output 3D images to HMD easily, iPhone and iPad is used for the proposed system.Figure 10 shows the definition of coordinate system of the iPhone and iPad.As is mentioned in the previous section, iPad is selected for the proposed system.The following HMD which is shown in Figure 11 can be attached to the iPad for displaying the created 3D images.Major specification of HMD is shown in Table 2.  Figure 12 shows outlook of the proposed system.User wears the HMD which is attached to the iPhone in this case.In Figure 12, there is TV monitor for demonstration of the created 3D images by the iPhone.

C. Application Software
Two application software, with and without walking detection are created.With walking detection of application software allows display 3D images in accordance with users' action and movement, changing users' head pose and users' walking together with mobile devices orientations.Figure 13 shows the start display image onto mobile device.

A. Examples of 3D Dispalyed Images
When user looks to the right direction, displayed image is changed to the right image.That is the same thing for left, up, and down directions.Also when he or she walks forward, then displayed 3D image is changed to the front image.This is the same thing for backward direction.
Figure 14 shows the displayed 3D image when user looks up and down, rotate in left and right directions, and step in forward and backward directions, respectively.www.ijarai.thesai.orgIn accordance with users' motion together with mobile phone movement, displayed 3D images are changed by frame by frame.Accelerometer data can be acquired 24 data a second.Taking every 6 data out of 24 data a second, users' attitude, roll, pitch, and yaw angle is estimated.Therefore, 4 times a second of attitude data are used for updating the 3D images which have to be created.

B. Example of 3D Display onto iPhone
Figure 16 shows the example of 3D image display onto iPhone.Method and system for 3D image display onto mobile phone and/or tablet terminal is proposed.Displaying 3D images are changed in accordance with user's location and attitude as well as some motions.Also 3D images can be combined with real world images and virtual images.Through implementation and experiments with virtual images, all functionalities are verified and validated.One of applications of the proposed system is attempted for demonstration of virtual interior of the house in concern.User can take a look at the inside house in accordance with user's movement.Thus user can imagine how the interior of the house looks like.

Fig. 1 .
Fig. 1.Illustrative view of the proposed system

Fig. 8 .
Fig. 8.Typical accelerometer output signals (Motion of users' acceleration (left hand side and motion, rotation rate (right hand side)

Fig. 13 .
Fig. 13.Start home display image onto mobile device V. EXPERIEMNTS Fig. 14.Displayed 3D images when user look up and down, rotate in left and right directions, and step in forward and backward directions

Fig. 15 .
Fig. 15.Definitions of three axis coordinate and roll, pitch, and yaw rotation axis

Fig. 16 .
Fig. 16.Example of 3D image display onto mobile phone C. Walking Detection Figure 17(a) shows the motion of rotation rate when user walks while Figure 17 (b) shows that when user rotating in roll direction.

Fig. 18 .
Fig. 18.Acceleration and rotation rate when user walks

Fig. 20 .
Fig. 20.Application of the proposed system to AR

TABLE I .
TRADEOFFTABLE FOR POSSIBLE DEVICES FOR THE PROPOSED SYSTEM.www.ijarai.thesai.org