FOR710: Reverse-Engineering Malware: Advanced Code Analysis

Malware authors employ evasion techniques and in-memory execution to hide data, hinder analysis, and thwart detection. First, we discuss the use of malicious steganography. Then we cover key steps in program execution, identifying how code is launched and labeling functions accordingly. Finally, we analyze shellcode with the support of WinDbg Preview.

Section 2 tackles the use of encryption in malware. Adversaries use cryptography for many reasons: to encrypt files, protect keys, conceal configuration settings, and obfuscate command and control (C2) communications. Reverse engineers must be prepared to investigate and articulate the purpose of routines in high-impact malware that implement encryption.

Section 3 covers automating malware analysis. We introduce Python, writing scripts to decrypt configuration data, deobfuscate strings, and extract payloads. We cover a Dynamic Binary Instrumentation (DBI) framework, injecting and executing code within a process to examine its internals. We write Python scripts to automate debugging and dump unpacked code.

In this section, we continue discussing approaches to automating malware analysis. We introduce Ghidra’s API and write Python scripts to accelerate static code analysis. We also examine the value of binary emulation frameworks and use the Qiling framework to simulate execution and deobfuscate code and data.

Section 5 allows students to flex new skills in an independent competitive environment. With extended access to a capture the flag (CTF) platform, students must recall key concepts and perform workflows discussed in class to succeed. This is an opportunity to analyze real-world, complex malware samples and reinforce your new advanced code analysis skills.