Building a Custom eBPF File Integrity Monitor for High-Throughput Linux Systems

Traditional file integrity monitoring (FIM) tools on Linux rely heavily on auditd or the inotify subsystem. While inotify is simple to set up, it hits structural limits when monitoring vast, deeply nested directory trees, requiring thousands of watches that consume kernel memory. On the other hand, auditd provides comprehensive system-wide visibility, but its context-switching overhead severely degrades performance under heavy file I/O workloads. To eliminate these bottlenecks in my lab infrastructure, I built a low-overhead FIM using eBPF kernel tracepoints and a userland consumer daemon.

Intercepting Path Transitions via Kernel Tracepoints

Instead of attaching kprobes to unstable kernel functions that may change across minor kernel releases, tracepoints offer a stable API surface. For file integrity monitoring, the primary tracepoints of interest are syscalls:sys_enter_openat, syscalls:sys_enter_unlinkat, and syscalls:sys_enter_renameat2.

When an application opens a file with write flags (such as O_WRONLY, O_RDWR, or O_CREAT), our eBPF program captures the process ID, user ID, binary name, and target file path. To pass this event stream to userland efficiently, we use a BPF ring buffer (BPF_MAP_TYPE_RINGBUF). Ring buffers share memory pages directly between kernel space and userland, avoiding per-CPU array copying and reducing event dropping under sudden burst traffic.

Kernel-Side eBPF Implementation

Below is the kernel-side C code attached to the sys_enter_openat tracepoint. It intercepts the syscall arguments, screens for write access, and pushes relevant event details directly into the shared ring buffer:

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

struct event_t {
    u32 pid;
    u32 uid;
    char comm[16];
    char filename[256];
    u32 flags;
};

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

SEC("tracepoint/syscalls/sys_enter_openat")
int handle_openat(struct trace_event_raw_sys_enter *ctx) {
    u64 flags = ctx->args[2];
    
    // Filter out read-only access early in kernel space
    if ((flags & (1 | 2 | 64 | 512)) == 0) {
        return 0;
    }

    struct event_t *event;
    event = bpf_ringbuf_reserve(&events, sizeof(*event), 0);
    if (!event)
        return 0;

    u64 id = bpf_get_current_pid_tgid();
    event->pid = id >> 32;
    event->uid = bpf_get_current_uid_gid();
    bpf_get_current_comm(&event->comm, sizeof(event->comm));
    
    const char *filename_ptr = (const char *)ctx->args[1];
    bpf_probe_read_user_str(&event->filename, sizeof(event->filename), filename_ptr);
    event->flags = (u32)flags;

    bpf_ringbuf_submit(event, 0);
    return 0;
}

char _license[] SEC("license") = "GPL";

Userland Processing and Context Enrichment

The userland component loads the compiled BPF ELF object, binds the program to tracepoint hooks, and continuously polls the ring buffer. Because initial filtering occurs inside the kernel driver, userland worker threads only wake up when a file modification attempt actually takes place.

When an event is consumed, the daemon enriches the payload by calculating cryptographic file hashes (SHA-256) if the target path matches monitored sensitive paths like /etc/, /usr/bin/, or /var/www/. Pushing path comparison maps into the eBPF kernel space keeps userland memory allocation near zero, preventing queue backpressure during heavy disk write bursts.

Architectural Trade-offs and Edge Cases

While this approach drastically reduces overhead compared to standard audit subsystems, a few operational edge cases require careful handling:

1. Relative Paths: System calls like openat frequently pass relative file paths alongside a directory file descriptor (dirfd). Reading raw user strings only gives relative path context. Resolving full paths requires utilizing bpf_d_path on supported kernels or maintaining a file descriptor cache in userland.
2. Short-Lived Processes: Rapid process execution where a process opens a file and terminates in milliseconds can cause race conditions when userland attempts to read additional process metadata from /proc. Capturing complete process provenance (e.g., namespace IDs, parent PID, start time) inside the kernel payload ensures accurate event attribution.
3. Ring Buffer Sizing: Tune ring buffer size according to system write throughput. A 256KB buffer effortlessly handles spikes up to 50,000 write events per second in local testing with zero dropped samples.

Photo by Conny Schneider on Unsplash.