Accession Number : AD1046778


Title :   Scaling Bulk Data Analysis with Mapreduce


Descriptive Note : Technical Report


Corporate Author : Naval Postgraduate School Monterey United States


Personal Author(s) : Andrzejewski,Timothy J


Full Text : https://apps.dtic.mil/dtic/tr/fulltext/u2/1046778.pdf


Report Date : 01 Sep 2017


Pagination or Media Count : 133


Abstract : Between 2005 and 2015, the world population grew by 11% while hard drive capacity grew by 95%. Increased demand for storage combined with decreasing costs presents challenges for digital forensic analysts working within tight time constraints. Advancements have been made to current tools to assist the analyst, but many require expensive specialized systems, knowledge and software. This thesis provides a method to address these challenges through distributed analysis of raw forensic images stored in a distributed file system using open-source software. We develop a proof-of-concept tool capable of counting unique bytes in a 116 TiB corpus of drives in 1 hour 41 minutes, demonstrating a peak throughput of 18.33 GiB/s on a 25-node Hadoop cluster. Furthermore, we demonstrate the ability to perform email address extraction on the corpus in 2 hours 5 minutes, for a throughput of 15.84 GiB/s, a result that compares favorably to traditional email address extraction methods, which we estimate to run with a throughput of approximately 91 MiB/s on a24-core production server. Primary contributions to the forensic community are: 1) a distributed, scalable method to analyze large datasets in a practical timeframe, 2) a MapReduce program to count unique bytes of any forensic image, and 3) a MapReduce program capable of extracting 233 million email addresses from a 116 TiB corpus in just over two hours.


Descriptors :   COMPUTATIONAL FORENSICS , data mining , High Performance Computing , software tools


Subject Categories : Information Science


Distribution Statement : APPROVED FOR PUBLIC RELEASE