Anomaly-Based Intrusion Detection Systems Utilizing System Call Data

This research aims at the enhancement of computer defenses by making them invulnerable to new, mutating and obfuscated malware. It offers a semantic approach to system behavior analysis, centered on the concept of functionality. Functionality is the highest level of the behavior semantics, it is defined by the specific goal of computer operations, not by its software realization. This allows for identifying some classes of malware achieving the same specific malicious operations. Colored Petri nets are proposed as a basis for behavioral signatures representing particular functionalities, both legitimate and malicious. Special techniques are proposed to address three interrelated aspects signature expressiveness, behavioral obfuscation and run-time signature matching efficiency. A signature based approach for detecting malicious functionalities in the system call domain is developed. It has been implemented in a prototype software and tested. It is superior to existing behavior based techniques in addressing behavioral obfuscations and multiple functionality realizations. The experiments results indicate low rate of false positives and negatives, and low execution overhead. Such results suggest that detecting malicious functionality presents a sufficiently dependable and efficient method for distinguishing malware from benign software.