Practical Memory Forensics: Hunting Malware with Volatility 3
Memory analysis is often treated as an advanced discipline of incident response, reserved for malware reverse engineers or specialized threat hunters. In practice, memory forensics provides one of the fastest ways to validate compromised hosts, pull unencrypted Command and Control (C2) configuration data, and detect stealthy persistence mechanisms that disk-based scanners completely miss.
When modern malware executes, it frequently uses techniques like process hollowing, reflective DLL injection, or memory-only payloads (fileless malware) to evade traditional Endpoint Detection and Response (EDR) agents. Disk forensics will not show you a file that was never written to disk, but RAM holds the execution context, decrypted strings, injected code blocks, and active network sockets.
In this guide, we will walk through practical analysis of a Windows memory dump using Volatility 3, focusing on threat hunting techniques you can apply immediately during incident response engagements.
Why Volatility 3?
For years, Volatility 2 was the gold standard for memory forensics. However, it relied heavily on static OS profiles that required manual symbol creation for custom or updated Windows builds. Volatility 3 completely redesigned the framework architecture. It automatically downloads and parses PDB symbols from Microsoft symbol servers, making Windows analysis seamless across patch levels.
Volatility 3 is faster, modular, and written in modern Python 3. If you are still relying on older tools, transitioning to Volatility 3 will drastically reduce your setup overhead during time-critical investigations.
Setting Up Your Environment
To get started, install Volatility 3 via Python package managers. Ensure you have Python 3.9 or newer installed.
Installation command: python3 -m pip install volatility3
To verify your installation and ensure symbol downloading works correctly, run the help command: vol -h
Volatility 3 automatically caches Windows symbol tables in your home directory (~/.cache/volatility3/symbols). If your analysis machine is air-gapped, you can pre-download these symbol packages from the official Volatility symbol repository and place them in your local symbol cache.
Step 1: Mapping Process Trees and Spotting Anomalies
When analyzing an unknown memory image, your first objective is to build a high-level timeline and structure of execution. The windows.pslist plugin outputs active process structures from the doubly linked list in kernel memory (EPROCESS blocks).
Run the pslist plugin: vol -f suspect_memory.raw windows.pslist
Key columns to analyze:
- PID and PPID: Verify parent-child relationships.
- ImageFileName: Look for typosquatting (e.g., svchost.exe spelled as scvhost.exe).
- CreateTime: Look for processes created during suspicious anomaly windows.
However, sophisticated malware often unlinks itself from the EPROCESS active process list to hide from standard OS APIs. To detect hidden processes, run the windows.pstree and windows.psscan plugins.
The psscan plugin scans pool memory for EPROCESS objects, finding processes that were terminated or intentionally unlinked from the active process list. Compare the results of pslist and psscan. If psscan reveals an EPROCESS structure for a process missing from pslist, you likely identified a rootkit or hidden process.
Legitimate parent-child relationships to memorize:
- smss.exe (Session Manager) creates csrss.exe and wininit.exe, then terminates.
- wininit.exe spawns services.exe, lsass.exe, and lsass-related core processes.
- services.exe spawns svchost.exe instances with the -k flag.
- Any instance of svchost.exe spawned by cmd.exe, powershell.exe, or user applications is a major red flag.
Step 2: Detecting Code Injection and Process Hollowing
Fileless malware and loaders frequently rely on Process Injection (such as VirtualAllocEx combined with WriteProcessMemory) or Process Hollowing (creating a suspended legitimate process like svchost.exe, unmapping its memory, and writing shellcode over it).
To uncover injected executable code regions, use the windows.malfind plugin: vol -f suspect_memory.raw windows.malfind
Malfind scans process memory regions for pages allocated with executable permissions (PAGE_EXECUTE_READWRITE or PAGE_EXECUTE_READ) that are not mapped to a file on disk.
Look specifically for:
- Memory regions starting with the ‘MZ’ header (0x4D 0x5A), indicating an injected PE binary in heap or stack memory.
- Shellcode instructions at the beginning of the memory page (such as NOP sleds or assembly setup instructions like push ebp / mov ebp, esp).
When malfind flags a PID, note the base virtual address. You can dump the specific memory range using the windows.dumpfiles plugin or the specialized malfind dump option to extract the raw executable for dynamic analysis in a sandbox.
Step 3: Uncovering Rogue Handles and Loaded DLLs
Malware must interact with the system using operating system primitives: file handles, registry keys, named pipes, and loaded modules.
If a process looks suspicious, list its loaded dynamic link libraries (DLLs) using windows.dlllist:
vol -f suspect_memory.raw windows.dlllist –pid
Look for DLLs loaded from unusual paths like AppData\Local\Temp or C:\Users\Public. Legitimate system DLLs should consistently load from C:\Windows\System32 or C:\Windows\SysWOW64.
Next, inspect open handles using windows.handles:
vol -f suspect_memory.raw windows.handles –pid
Filter the handles output for Key, File, or Mutant (mutex). Named mutants/mutexes are often hardcoded by malware authors to prevent multiple instances from running simultaneously. Identifying a unique mutex name gives you an instant Indicator of Compromise (IOC) to sweep across your endpoint telemetry.
Step 4: Extracting Network Connections and Strings
Even if malware uses memory-only execution, it must communicate back to Command and Control servers.
Run the windows.netscan plugin to extract active network sockets, bound ports, and listening endpoints: vol -f suspect_memory.raw windows.netscan
The output maps local IP addresses, foreign IP addresses, port numbers, connection states, and associated process IDs. If you find an active connection tied to a hollowed svchost.exe process reaching out to an untrusted external IP over port 443 or 8443, you have confirmed a live C2 session.
To pull actionable threat intelligence (such as C2 domain names, IP addresses, or user-agent strings) without decompiling, dump the process memory and run string extraction:
vol -f suspect_memory.raw windows.memmap –pid
Filter strings using standard Linux CLI utilities: strings -a -e l process_dump.dmp | grep -i ‘http’
The -e l flag instructs strings to parse 16-bit Little Endian Unicode characters, which are standard for Windows memory structures.
Building an Automated Triage Workflow
During an incident, manually typing individual Volatility commands slows down initial scoping. Create a rapid triage script that runs pslist, pstree, psscan, malfind, and netscan sequentially, piping output into structured text or JSON files for rapid reviewing.
Volatility 3 also supports a Python API, allowing you to build custom python pipelines that ingest raw memory images, execute plugins programmatically, and trigger automated alerts when high-confidence signatures (such as malfind hits combined with external netscan entries) are detected.
By adding memory analysis to your defensive toolkit, you bypass the evasion tricks modern adversaries use to blind host-based monitoring, giving you full visibility into what is actually executing in system memory.