Accession Number : AD1046778

Title :   Scaling Bulk Data Analysis with Mapreduce

Descriptive Note : Technical Report

Corporate Author : Naval Postgraduate School Monterey United States

Personal Author(s) : Andrzejewski,Timothy J

Full Text :

Report Date : 01 Sep 2017

Pagination or Media Count : 133

Abstract : Between 2005 and 2015, the world population grew by 11% while hard drive capacity grew by 95%. Increased demand for storage combined with decreasing costs presents challenges for digital forensic analysts working within tight time constraints. Advancements have been made to current tools to assist the analyst, but many require expensive specialized systems, knowledge and software. This thesis provides a method to address these challenges through distributed analysis of raw forensic images stored in a distributed file system using open-source software. We develop a proof-of-concept tool capable of counting unique bytes in a 116 TiB corpus of drives in 1 hour 41 minutes, demonstrating a peak throughput of 18.33 GiB/s on a 25-node Hadoop cluster. Furthermore, we demonstrate the ability to perform email address extraction on the corpus in 2 hours 5 minutes, for a throughput of 15.84 GiB/s, a result that compares favorably to traditional email address extraction methods, which we estimate to run with a throughput of approximately 91 MiB/s on a24-core production server. Primary contributions to the forensic community are: 1) a distributed, scalable method to analyze large datasets in a practical timeframe, 2) a MapReduce program to count unique bytes of any forensic image, and 3) a MapReduce program capable of extracting 233 million email addresses from a 116 TiB corpus in just over two hours.

Descriptors :   COMPUTATIONAL FORENSICS , data mining , High Performance Computing , software tools

Subject Categories : Information Science

Distribution Statement : APPROVED FOR PUBLIC RELEASE