A Comparative Study of Forensic Tools for WhatsApp Analysis using NIST Measurements

One of the popularly used features on Android smartphone is WhatsApp. WhatsApp can be misused, such as for criminal purposes. To conduct investigation involving smartphone devices, the investigators need to use forensic tools. Nonetheless, the development of the existing forensic tool technology is not as fast as the development of mobile technology and WhatsApp. The latest version of smartphones and WhatsApp always comes up. Therefore, a research on the performance of the current forensic tools in order to handle a case involving Android smartphones and WhatsApp in particular need to be done. This research evaluated existing forensic tools for performing forensic analysis on WhatsApp using parameters from NIST and WhatsApp artifacts. The outcome shows that Belkasoft Evidence has the highest index number, WhatsApp Key/DB Extractor has superiority in terms of costs, and Oxygen Forensic has superiority in obtaining WhatsApp artifact. Keywords—Whatsapp; acquisition; NIST parameters; artifact


I. INTRODUCTION
Smartphones with the Android operating system were introduced to the public in 2007; and it became the most popular operating system in 2011, judging from the sales [1].In the fourth quarter of 2016 the number of smartphone sales with android operating system is 379.98 million units, as shown in Fig. 1.Some popular smartphone features are messaging (88%), email (70%), Facebook (62%), camera (62%), and WhatsApp (51%) [2].In [3], authors conducted a survey on instant messaging application, WhatsApp, Viber, and Telegram.From the survey results, WhatsApp tops the list at 60%.In terms of the user number, WhatsApp has increased significantly from year to year [4].As of July 2017, the number of WhatsApp users has as many as 1.3 billion users as in Fig. 2. WhatsApp has various features, for instance sending and receiving text messages, pictures, videos, and documents.WhatsApp also comes with phone call and video call features.WhatsApp has been equipped with end-to-end encryption technology that serves to secure sent messages.With end-to-end encryption, the messages sent can only be read by senders and recipients [5].
It is impossible to separate WhatsApp from misuse.The large number of users and the end-to-end encryption technology used can be a magnet for someone with a criminal purpose such as drug trafficking, cyber-bullying, trafficking, and so on.There are some cases involving IM or WhatsApp applications [6].In a case involving smartphone devices, the investigator needs to do mobile forensics.Mobile forensics is one of the forensic digital branches that learn on how to perform evidence recovery from a smartphone device.The investigator will perform forensic analysis of smartphone devices using forensic tools with a forensically-tested methodology, thus the analysis results are valid before the law and can be used as means of evidence [7].According to [8], there are three forensic acquisition techniques: manual, physical, and logical.In the manual acquisition, the investigator will manually create the acquisition by directly looking at the contents of the smartphone device to find evidence.The advantage of manual acquisition is that investigators do not require forensic tools to create acquisitions.Manual acquisition has constraints in terms of the integrity of the evidence as investigators will directly examine the evidence which may result in the possibility of data changes.In physical acquisition, the investigator will clone a smartphone device.The cloning results will then be analyzed using forensic tools.In logical acquisition, the investigator will perform the data acquisition found in the smartphone device to be subsequently analyzed.
In [9], authors performed forensic analyzes using Oxygen Forensic and MOBILedit.The researchers argue that every forensic tool has its own advantages and disadvantages.It can be handled using several forensic tools that have different capabilities in addressing cases related to smartphone devices.MOBILedit has advantages in terms of run time, while Oxygen Forensic has an advantage in terms of artifact analysis.In other research conducted by [10] using Oxygen Forensic tools managed to find artifacts of call logs, text messages, media files (photos, video, audio), internet data, geolocation, applications, and social media data.In [10], authors also explained that mobile forensic has several challenges, such as: malicious programs, lack of availability of tools, password recovery, accidental reset, and anti-forensic technique.
In [11], authors performed comparisons and analysis of commercial forensic and open source tools.The tools put into comparison are TSK Autopsy, SIFT, MOBILedit, and Cellebrite UFED.The researcher believes that No. 1 forensic tool is perfect for performing all processes.Open source forensic tools have advantages in the number of users, flexibility in terms of use with console commands or GUIbased applications, logging capability, and good in tolerating errors.Meanwhile, commercial forensic tools are superior in terms of process speed, data extraction accuracy, and analytical skills.Commercial forensic tools also have the ability to restore deleted data.In [12], authors also conducted mobile forensic analysis using Celebrite UFED in order to determine the extent of forensic tool performance.In [12], authors obtained information on IMSI and ICCID.The artifacts, such as call logs, social media chat, contact list, email, SMS, and media files (audio, documents, image, video) also retrieved.In [12], authors added that each of the forensic tools has the possibility to produce different outputs.Therefore, an investigator should know what forensic tool he should use for a case.
The development of mobile technology and the large number of smartphone devices on the market become a challenge for investigators.One of the challenges of mobile forensics is the lack of resources in the sense that the rapid development of mobile technology and the growing number of smartphone devices are not put in a balance by the development of forensic mobile technology and the existing forensic tools [13].
NIST released a test plan to measure the performance of a forensic tool in a publication entitled "Mobile Device Tool Test Assertions and Test Plan ver.2" and "Mobile Device Tool Specification ver.2" [14], [15].NIST argues that increasing the number of smartphone devices each year gives problems in forensics cases.Therefore, a method is needed to measure the ability of forensic tools on the market.NIST provides 42 measurement parameters and methods to measure the performance of forensic tools based on the results of each test plan.
Judging from the development of mobile technology and WhatsApp technology, WhatsApp's popularity, the possibility of cases involving WhatsApp, and previous research, the researcher conducted a comparative evaluation of forensic tools for WhatsApp analysis on Android-based smartphones.The forensic tools used are WhatsApp DB/Key Extractor, Belkasoft Evidence, and Oxygen Forensic.The performance and ability to perform WhatsApp forensic analysis from each forensic tool will be evaluated using the NIST forensic tool parameter and additional parameters from the researcher.The research' results will be used as a recommendation for investigators when handling cases related to WhatsApp.

II. METHODOLOGY AND TOOLS
The objective of this study was to evaluate forensic tools.WhatsApp DB/Key Extractor, Belkasoft Evidence and Oxygen Forensic will be evaluated based on parameters from NIST and additional parameters from researchers in terms of the ability to perform WhatsApp's forensic analysis on Android.

A. Research Methodology
The research used the steps as in Fig. 3.The steps of the research are divided into four: experiment simulation, forensic analysis, analysis result, and conclusion.
• Experiment Simulation: Fig. 4 shows the experimental simulations performed.User A's smartphone device will be used to communicate with User B and simulates the daily use of WhatsApp, such as sending messages, making calls, receiving pictures.User A's Smartphone then will be used for forensic analysis in the next step.User A's smartphone used in the research is Samsung Galaxy S4 GT-I9500 with Android Lollipop 5.0.1 operating system and it has been rooted.WhatsApp version used in this research is version 2.17.351.
• Forensic Analysis: The researches will perform forensic analysis on smartphone devices using the WhatsApp DB/Key Extractor, Belkasoft Evidence, and Oxygen Forensic.The forensic analysis will be conducted under closed conditions in the sense that smartphone devices will be converted into Airplane Mode to maintain data integrity.
• Result Analysis: The performance of each forensic tool will then be analyzed using NIST parameters and additional parameters from the researcher.The parameters used are adjusted to the objective of the research, namely, WhatsApp analysis.• Conclusion: The evaluation of forensic tools using NIST parameters and additional parameters are presented.

B. Research Tools
The research tools used in this research are divided into two: Experimental tools and forensic tools.Here, the researcher used parameters from NIST as on Table III.NIST lists the measurement parameters of forensic tools on two written reports entitled "Mobile Device Tool Specification" and "Mobile Device Tool Test Assertions and Test Plan".The measurement parameters are divided into cores and optional.The division is done based on the type of acquisition made.Core leads to logical acquisition features and capabilities.Meanwhile, optional leads more to physical acquisition features and capabilities.In this research, the researcher does not include the parameters of MDT-CA-10 and the parameters on Universal Integrated Circuit Card (UICC) because the data on WhatsApp application are in the internal memory, not on UICC.
Researcher adds several additional measurement parameters as shown in Table IV.The additional parameters are more focused on the abilities of forensic tools to extract artifacts from WhatsApp for logical acquisition and physical acquisition.Additional parameters listed are essential for investigator during investigation related to WhatsApp.

A. WhatsApp DB/Key Extractor
WhatsApp Key/DB Extractor can only conduct logical acquisition.Fig. 5       WhatsApp Key/DB Extractor managed to get the image artifact with its metadata as shown in Fig. 9. Image artifacts can be zoomed but the zoomed image will blur out.WhatsApp/DB Key Extractor image artifact has a weakness in terms of resolution.The image artifact obtained has a small image resolution according to the thumbnail size in WhatsApp.The video and document artifacts cannot be obtained using WhatsApp Key/DB Extractor.

B. Belkasoft Evidence
Belkasoft Evidence has the ability to perform logical acquisition and physical acquisition.From the experimental results, Belkasoft Evidence almost meets all criteria of core parameters and optional NIST.Belkasoft Evidence provides information on smartphone devices, such as IMEI in accordance to NIST MDT-CA-06 parameters.Belkasoft Evidence is also accompanied by an option to select the data to be acquired individually or as a whole, as shown in Fig. 10 in accordance to the parameters of MDT-CA-01, MDT-CA-02, and MDT-CA-03.Investigators can choose what data needed for acquisition and will reduce acquisition run time.Fig. 11 shows the notification when there is a disruption to the acquisition process using Belkasoft Evidence.Notification feature during connection interruption is in accordance with MDT-CA-04 NIST parameter.In logical acquisition, Belkasoft Evidence failed to retrieve contact list artifact of WhatsApp, WhatsApp call log, and text messages.The researcher only managed to find images, video, and document artifacts.Video and document artifact files can be opened, simplifying the analysis process.Fig. 12 shows image artifact obtained using Belkasoft Evidence.The image artifact obtained comes with considerably large pixel resolution, so that it not blurred when being zoomed in.
Text message artifact is successfully obtained using the physical acquisition Belkasoft Evidence as in Fig. 13.The timestamp information, the contact number of the sender and the recipient of the message can be found.Japanese letter used for experiment and successfully read by Belkasoft Evidence in accordance with NIST MDT-AO-06 parameter.

C. Oxygen Forensic
Just like Belkasoft Evidence, Oxygen Forensic has the ability to perform logical acquisition and physical acquisition.Oxygen Forensic successfully obtains smartphone device information as shown in Fig. 14.Information regarding IMEI and IMSI is able to obtain according to NIST MDT-CA-06 parameter.Oxygen Forensic only has one feature to choose entire data acquisition according to NIST parameter MDT-CA-01, and does not have feature to individually select the data to be acquired.Fig. 15 shows WhatsApp contact list artifacts obtained using Oxygen Forensic.Contact name and contact number information can be determined and can be used to assist in the investigation process.From the logical acquisition and physical acquisition, text message artifact can be generated as in Fig. 16.Information such as message sender, message recipient, message content, and timestamp is successfully obtained.Text messages in non-Latin writing are also successfully read by Oxygen Forensic according to NIST parameters MDT-CA-08 and MDT-AO-06.
Oxygen Forensic also manages to obtain document, image, and video artifacts.Fig. 17 indicates the image artifacts successfully obtained by Oxygen Forensic.The image artifacts obtained have sufficiently good quality that they do not blur when the image is zoomed in to see them more clearly.
Oxygen Forensic managed to retrieve video artifacts.The video artifacts obtained by using Oxygen Forensic can be played so that it can help the investigation process if video file content is necessary to be viewed as in Fig. 18.Information regarding file names, video format, and video size can be found.

D. Discussion
The researcher used calculations with index numbers to determine the performance of each forensic tool in accordance with the experiment results.The calculation of index number used is unweighted index as shown in (1).Table V indicates the evaluation results of forensic tools using NIST measurement parameters and additional parameters from the researcher.
Equation ( 1) used to calculate the index number from each forensic tool.WhatsApp Key/DB Extractor has an index number of 23.52%.Belkasoft Evidence has an index number of 88.23%.Oxygen Forensic has an index number of 82.35%.
WhatsApp Key/DB Extractor is only capable of conducting logical acquisition and requires another tool to read WhatsApp Key/DB Extractor acquisition result.However, WhatsApp Key/DB Extractor successfully obtained WhatsApp Contact List Artifacts, WhatsApp call log, text messages, and images.The image artifacts obtained have a thumbnail size pixel resolution, which will blurry when being zoomed in.
From experimental results using Belkasoft Evidence, all core parameters and optional NIST are almost met entirely.Belkasoft Evidence did not meet the parameters of MDT-CA-08 for failing to get message artifacts in non-Latin writing.Logical acquisition using Belkasoft Evidence cannot successfully get the contact list artifact on WhatsApp, WhatsApp call log, and text messages.However, document, image and video artifacts are successfully obtained and can be opened to assist the investigation process.Similar to Belkasoft Evidence, Oxygen Forensic is able to perform logical acquisition and physical acquisition.Oxygen Forensic has disadvantage in terms of options for selecting data to be acquired.Oxygen Forensic cannot individually select the data to be acquired.From the experiments, Oxygen Forensic did not have any notification feature to notify investigators when connection problem occurs during acquisition process.Oxygen Forensic successfully obtained all artifacts according to parameters, either with logical acquisition or physical acquisition.
shows the acquisition process conducted using WhatsApp Key/DB Extractor.WhatsApp Key/DB Extractors have many shortcomings in terms of Core Assertions and Optional Assertions.Looking from experiment results, WhatsApp Key/DB Extractor did not get any information regarding smartphone devices, such as (International Mobile Equipment Identity) IMEI or (International Mobile Subscriber Identity) IMSI.From the NIST parameters used, WhatsApp Key/DB Extractor only succeeded in meeting the criteria of MDT-CA-07, MDT-CA-08, MDT-CR-01 A, and MDT-CR-03 A.

Fig. 6
Fig. 6 shows the acquisition results located in the WhatsApp-Key-DB-Extractor/extracted folder.WhatsApp DB/ Key Extractor can only do data acquisition alone, thus opening the acquisition results need to use other tools.In the present research, Belkasoft Evidence is used to open the acquisition result of WhatsApp Key/DB Extractor.The wa.db file contains the WhatsApp's contact list.Contact information, for example contact names and contact numbers, can be found as shown in Fig. 7.Meanwhile, msgstore.dbfile contains the communication logs that are performed using WhatsApp.WhatsApp Key/DB Extractor manages to get the text message artifact as shown in Fig.8.Message information such as message content, sender and recipient of message, timestamp, and file attachment can also be found.WhatsApp Key/DB Extractor can also do text acquisition in non-latin writing in accordance to MDT-CA-08.In this research, the researcher successfully retrieved Japanese letter for the experiment.
Table I describes the experimental tools used in the research.Table II describes the forensic tools used in the research.

TABLE III .
NIST FORENSIC TOOL PARAMETERS