LEVELING UP

Command-Line Log Analysis: Uncovering Attacks Without a SIEM

Advertisement

Enterprise Security Information and Event Management (SIEM) platforms like Splunk, Microsoft Sentinel, or Elastic are invaluable during security operations. However, during incident response, live triage, or jump-box analysis, you will not always have a multi-million-dollar SIEM ready at your fingertips. Sometimes, all you have is SSH access to an compromised machine and raw text files in /var/log.

Mastering native Linux text-processing tools allows you to filter gigabytes of log data in seconds, extract malicious activity, and map out attacker timelines directly from the terminal.

In this guide, we will cover practical command-line techniques using grep, awk, cut, sort, uniq, and jq to hunt down threat actors in web logs and system authentication files.

The Core Command-Line Toolkit

Before diving into specific scenarios, let us review the workhorses of command-line log analysis:

  • grep / ripgrep (rg): Filters text using literal strings or regular expressions.
  • awk: Extracts specific fields (columns) from structured text.
  • cut: Simple field delimiter tool, lighter than awk for basic operations.
  • sort: Orders output alphabetically or numerically (-n).
  • uniq -c: Groups adjacent identical lines and outputs occurrence counts.
  • jq: A lightweight command-line JSON processor.

Combining these tools using Unix pipes (|) creates powerful, real-time data pipelines.

Scenario 1: Analyzing Nginx and Apache Web Logs

Web servers are constant targets for automated scanners, path traversal probes, and web application attacks. Standard web log formats (such as Combined Log Format) usually structure lines as follows:

192.168.1.105 - - [14/Oct/2023:14:32:10 +0000] "GET /index.php HTTP/1.1" 200 1420 "-" "Mozilla/5.0"

1. Identifying the Top Requesting IP Addresses

An unusually high volume of requests from a single IP address often indicates scanning, directory bruteforcing, or layer-7 DDoS attempts.

To list the top 10 IP addresses by request volume:

awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -n 10

Breakdown:

  • awk '{print $1}': Extracts the first column (the client IP address).
  • sort: Sorts IPs alphabetically (required before running uniq).
  • uniq -c: Counts consecutive identical occurrences.
  • sort -nr: Sorts numerically (-n) in reverse (-r) order (highest counts first).
  • head -n 10: Limits output to the top 10 entries.

2. Detecting Web Application Exploitation Probes

Attackers frequently attempt SQL Injection (SQLi), Local File Inclusion (LFI), or Path Traversal. We can run pattern matching across the HTTP request strings (column 6 through 8).

Hunting for Path Traversal:

grep -Ei "(\.\./|\.\.\\|/etc/passwd)" access.log

Hunting for SQL Injection attempts:

grep -Ei "(UNION%20SELECT|SELECT.*FROM|OR%201=1|--)" access.log

3. Filtering by HTTP Response Status Codes

If an attacker is directory fuzzing (e.g., using ffuf or Gobuster), you will see thousands of 404 Not Found or 403 Forbidden responses followed by a single 200 OK when they discover a secret endpoint.

To count status codes (column 9 in standard formats):

awk '{print $9}' access.log | sort | uniq -c | sort -nr

To quickly find requests that successfully returned content (200 OK) to sensitive endpoints:

awk '$9 == "200" {print $1, $6, $7}' access.log | grep -E "(admin|config|backup|\.env|\.git)"

Scenario 2: Uncovering SSH Brute Force Attacks

On Debian/Ubuntu systems, SSH authentication attempts are recorded in /var/log/auth.log (or /var/log/secure on RHEL/CentOS systems).

1. Finding Failed Login Attempts

When threat actors automate SSH password spraying, they leave behind thousands of failed authentication events.

Extract failed login attempts along with the associated usernames and IP addresses:

grep "Failed password" /var/log/auth.log | awk '{if ($9 == "invalid") print $11, $13; else print $9, $11}' | sort | uniq -c | sort -nr | head -n 15

This pipeline normalizes username positions (accounting for invalid user strings) and prints the most targeted accounts along with the attacking source IP addresses.

2. Spotting Successful Logins After Brute Force

A critical step during incident response is determining whether an attacker successfully authenticated after multiple failed attempts.

Run this command to check for successful SSH logins:

grep "Accepted password\|Accepted publickey" /var/log/auth.log

Correlate the timestamp and source IP of a successful login with your list of brute-forcing IPs. If IP 198.51.100.44 had 4,000 failed password attempts and then generated an Accepted password for root log at 03:14:22, you have a confirmed compromise.

Scenario 3: Wrangling Structured JSON Logs with jq

Modern applications, microservices, and log shippers (like Vector or Fluentbit) output logs in structured JSON format rather than plaintext strings. Parsing JSON logs with traditional text tools like awk can be messy. This is where jq excels.

Suppose you have an application log (app.json) formatted like this:

{"timestamp": "2023-10-14T15:00:12Z", "client_ip": "203.0.113.5", "status": 500, "endpoint": "/api/v1/user", "user_agent": "python-requests/2.28.1"}

Extracting Specific Fields

To pull only the timestamp, client IP, and endpoint for requests that returned a 500 Internal Server Error:

jq -r 'select(.status == 500) | [.timestamp, .client_ip, .endpoint] | @tsv' app.json
  • -r: Outputs raw strings (removes JSON quotes).
  • select(.status == 500): Filters entries where the status field equals 500.
  • @tsv: Formats the array as tab-separated values for clean command-line reading or piping into other commands.

Finding Rare User-Agents

Attackers often use uncommon or default automation user-agents (e.g., sqlmap, python-requests, Go-http-client).

Summarize all unique user-agents and their counts:

jq -r '.user_agent' app.json | sort | uniq -c | sort -n

Outliers appearing only once or twice at the top of the output deserve immediate investigation.

Building an Effective On-the-Fly Analysis Workflow

When performing live log analysis during an incident, follow this structured process:

  1. Establish a Baseline: Identify high-volume normal traffic (e.g., internal health checks, monitoring agents) and filter them out early (grep -v).
  2. Identify Outliers: Group events by IP, endpoint, status code, or user account to find statistical anomalies.
  3. Pivot and Zoom In: Once an suspicious IP or user account is found, grep specifically for that IOC across all available log files to construct an chronological timeline of activity.
  4. Document Timestamps: Record exact UTC timestamps of critical events for inclusion in incident response reports.

Command-line fluency converts raw, unorganized log data into actionable threat intelligence without waiting for indexing pipelines or complex dashboard configurations. Next time you encounter a suspicious host, skip the web UI and do your initial triage straight from the shell.

Photo by Luke Chesser on Unsplash.