Enhancing Malware Mitigation using Graph Neural Networks, Adversarial Retraining and Conformal Prediction

Abstract

Cybersecurity defences encompass strategies, tools, and practices designed to protect digital systems, networks, devices, and data from unauthorized access, attacks, and other cyber threats. The constant evolution of attack techniques necessitates ongoing advancements in cybersecurity measures. Recently, deep learning and machine learning have been increasingly used to detect cyber threats due to their ability to generalize across various instances of malware based on learned patterns, offering a dynamic and proactive defence mechanism.
However, integrating advanced machine learning technologies introduces vulnerabilities, notably susceptibility to adversarial examples and dataset shifts. Adversarial inputs, created with subtle perturbations, cause models to misclassify inputs, allowing malware to evade detection. Another critical issue is dataset shift, which occurs when the statistical properties of the training data differ from those of the operational data, thereby degrading model performance. Despite advancements in adversarial machine learning and dataset shift mitigation, the battle between defenders and attackers persists. Additionally, insider threats pose a unique challenge, as they originate from within the organization, leveraging legitimate access to inflict harm. This thesis explores advanced machine learning and deep learning methodologies to enhance cybersecurity defences across Android applications, Windows platforms, corporate networks, and insider threats.
The first significant contribution of this thesis is the development of VGAE-MalGAN, an adversarial malware generation algorithm targeting Graph Neural Network (GNN) based Android malware classifiers. VGAE-MalGAN preserves the original structure and semantics of the malware graph while adding adversarial edges to fool classifiers successfully. First, we implemented multiple classifiers with API graph, Permission and Intent as features. In comparison to state-of-the-art methods, the GNN with CNN ensemble classifier achieves 98.68% accuracy on the Drebin dataset and 98.33% on CICMaldroid, outperforming existing methods by significant margins. Retraining classifiers with adversarial samples generated by VGAE-MalGAN substantially improves their resilience to attacks. Before retraining, the classifier’s recall dropped to 47.72% on CICMaldroid and 65.64% on Drebin when attacked using VGAE-MalGAN. However, after retraining, the recall improved to 98.46% on CICMaldroid and 91.01% on Drebin. Unlike existing methods such as Grabnel, which modify graph structures, our technique ensures that the integrity of the original malware graph is maintained while effectively bypassing detection models.
The second contribution addresses botnet detection within organizational networks, where traditional Network Intrusion Detection Systems (NIDS) often fall short. Machine learning algorithms fine-tuned through Genetic Algorithms (GA) and Particle Swarm Optimization (PSO) are examined, alongside attacks on individual features using Carlini &Wagner (C&W) and Generative Adversarial Network (GAN) techniques. Finally, conformal prediction was integrated within NIDS to enhance network security by providing valid measures of prediction certainty and rejecting uncertain instances. It improves detection by rejecting incorrect predictions, thus reducing false positives and increasing overall confidence in the model’s decisions. Compared to state-of-the-art methods, our approach achieves an F1-score of 95.52% on the ISCX dataset and 99.92% on the ISOT dataset. The conformal prediction layer boosts recall from 92.3% to 99.93% on the ISOT dataset and 96.57% to 98.02% in the ISCX dataset.
The third contribution of the thesis is to reduce incorrect predictions using uncertainty estimation in Windows PE malware classification. The thesis proposes using uncertainty estimates from an NN Ensemble as the Non-Conformity Measure (NCM) in Inductive Conformal Evaluation (ICE) to improve prediction acceptance and rejection decisions during dataset shifts. In experiments under dataset shifts, our approach rejected 73,724, 72,517, and 72,862 incorrect predictions out of 87,881 incorrect predictions, outperforming the state-of-the-art ICE with probability estimates, which rejected 67,884.
Finally, the thesis addresses malicious insider activities by combining graph-based and temporal methods to detect insider threats. By leveraging both explicit and implicit graph structures and applying a Bi-LSTM to node embeddings from Graph Convolutional Networks (GCN) with an attention mechanism, the research captures relational and temporal dynamics of user activities, improving anomaly and insider threat detection accuracy. Our model was evaluated using metrics such as Area Under the Curve (AUC), Detection Rate (DR), and False Positive Rate (FPR) on the r5.2 and r6.2 datasets. The results show that our model achieved the highest performance with an AUC of 98.62 on the r5.2 dataset and 88.48 on the more challenging r6.2 dataset, outperforming other state-of-the-art models. In particular, the model achieved a 100% DR and a 0.05 FPR on the r5.2 dataset. On the r6.2 dataset, while the task was more challenging than the r5.2 dataset, our model still outperformed other methods with a DR of 80.15% and an FPR of 0.15.

Date of Award	5 Nov 2024
Original language	English
Awarding Institution	Northumbria University
Supervisor	Biju Issac (Supervisor) & Longzhi Yang (Supervisor)