SiLK: A Tool Suite for Unsampled Network Flow Analysis at Scale

reportActive / Technical Report | Accession Number: ADA610679 | Open PDF

Abstract:

A large organization can generate over ten billion network flow records per day, a high-velocity data source. Finding useful, security-related anomalies in this volume of data is challenging. Most large network flow tools sample the data to make the problem manageable, but sampling unacceptably reduces the fidelity of analytic conclusions. In this paper we discuss SiLK, a tool suite created to analyze this high-volume data source without sampling. SiLK implementation and architectural design are optimized to manage this Big Data problem. SiLK provides not just network flow capture and analysis, but also includes tools to analyze large sets and dictionaries that frequently relate to network flow data, incorporating higher-variety data sources. These tools integrate disparate data sources with SiLK analysis.

Author(s):
Thomas, Mark ; Metcalf, Leigh ; Spring, Jonathan ; Krystosek, Paul ; Prevost, Katherine
Author Organization(s):
CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERING INST
Descriptive Note:
Conference paper
Supplementary Note:
Published in the IEEE 3rd International Congress on Big Data, 27 Jun-2 Jul 2014, Anchorage, AK.
Pagination:
0021

Security Markings

DOCUMENT & CONTEXTUAL SUMMARY

Distribution:
Approved For Public Release
Distribution Statement:
Approved For Public Release; Distribution Is Unlimited.

RECORD

Collection: TR
Identifying Numbers
Report Number(s):
CERTCC-2014-24
Contract/Grant Number(s):
FA8721-05-C-0003
Monitor Series:
DOD
 Subject Terms
Joint Capability Areas:
JCA_6.1.3_Switching and Routing; JCA_6_Net Centric; JCA_6.1_Information Transport; JCA_5_Command and Control; JCA_5.3_Planning; JCA_4.6_Engineering; JCA_4.6.1_General Engineering; JCA_5.5_Direct; JCA_1_Force Support; JCA_5.5.1_Communicate Intent and Guidance; JCA_6.4.1_Secure Information Exchange; JCA_6.2.1_Information Sharing; JCA_6.2.2_Computing Services; JCA_1.2_Force Preparation; JCA_6.2_Enterprise Services; JCA_6.4_Information Assurance; JCA_1.2.3_Educating; JCA_1.2.5_Lessons Learned; JCA_1.3_Human Capital Management; JCA_3.2_Engagement; JCA_3_Force Application
Modernization Areas:
Cyber
Communities of Interest:
Sensors
Descriptor(s):
*DATA PROCESSING, COMPUTER NETWORK SECURITY, DATA ACQUISITION, INFORMATION PROCESSING, MONITORING
Field(s)/Group(s):
Information Science, Computer Systems Management and Standards
Keyword(s):
NETWORK FLOW ANALYSIS, NETWORK FLOW DATA, DATA SETS, DATA CAPTURE, DATA ANALYSIS
Report Date:
2014 Jun 01
Creation Date:
2014 Dec 10