logutil

# LogUtil - Real-time Nginx Log Analyzer

Overview

LogUtil is a powerful, real-time log analysis tool written in Rust that provides an interactive terminal interface for monitoring and analyzing Nginx access logs. It offers comprehensive log parsing, real-time monitoring, and detailed analytics with a beautiful TUI (Terminal User Interface).

Key Features

🔍 Real-time Log Monitoring

• Live tail functionality that monitors log files as they grow
• Automatic detection and processing of new log entries
• Real-time updates without manual refresh

📊 Comprehensive Analytics

• IP Address Analysis: Track requests by IP addresses with detailed statistics
• URL Analysis: Monitor most accessed URLs and their patterns
• Request Type Tracking: Categorize requests by HTTP methods (GET, POST, etc.)
• Domain Analysis: Track requests by domain names
• Time-based Analytics: Analyze request patterns over time

🎨 Interactive TUI Interface

• 9 Main Tabs: Overview, Requests, Detailed, Sparkline, Heatmap, Security, Performance, Errors, and Bots
• Overview Tab: Shows top IPs and URLs with real-time statistics
• Requests Tab: Searchable log entries with pagination
• Detailed Tab: Drill-down view for specific IP addresses
• Sparkline Tab: Real-time request timeline visualization
• Heatmap Tab: Hourly request patterns across multiple days
• Security Tab: Detection of suspicious IPs and attack patterns
• Performance Tab: Response time analysis and slow request identification
• Errors Tab: HTTP error code analysis and categorization
• Bots Tab: Bot and crawler detection and classification

⚡ Performance Optimizations

• Efficient memory management with automatic cleanup
• Optimized for large log files (handles 10,000+ entries)
• Minimal CPU usage during real-time monitoring
• Configurable cleanup policies

🔧 Flexible Configuration

• Customizable regex patterns for different log formats
• Support for various date formats
• Configurable top N entries display
• Optional automatic cleanup of outdated entries

📁 Interactive File Selection Mode

When you run LogUtil without specifying a log file, it launches an interactive file selection mode:

File Selector Features:

• 📂 Directory Navigation: Browse through folders with intuitive navigation
• 📄 File Browser: View all files with clear icons (📁 for folders, 📄 for files, ⬆️ for parent directory)
• 🔍 Visual Selection: Highlighted current selection with clear visual feedback
• ⚡ Quick Access: Navigate with arrow keys and select with Enter

Settings Configuration: After selecting a log file, you'll see an interactive settings screen where you can configure:

• 📊 Analysis Parameters: Enable/disable specific analysis tabs (Security, Performance, Errors, Bots, Sparkline, Heatmap)
• 🔧 Processing Options: Set count limits, regex patterns, date formats
• ⚙️ Real-time Settings: Configure cleanup policies and monitoring options
• ▶️ Start Analysis: Launch the analysis with your custom configuration

Usage:

# Launch interactive mode
logutil

# Or specify a file directly
logutil /path/to/access.log

🎮 Keyboard Shortcuts

Navigation:

• Tab / T: Switch to next tab
• Shift+Tab / Shift+T: Switch to previous tab
• ↑/↓: Navigate through lists and options
• ←/→: Switch between panels (in tabs with multiple panels)
• Enter: Select item or toggle boolean values
• Esc: Go back or cancel

File Selector:

• ↑/↓: Navigate through files and folders
• Enter: Select file or enter folder
• Esc: Go back to parent directory or exit

Settings:

• ↑/↓: Navigate through settings
• Enter: Edit setting or toggle boolean values
• Esc: Cancel editing or go back

TUI Controls:

• Q / Ctrl+C: Quit application
• Enter: Copy selected item to clipboard (Overview tab)

📋 Command Line Examples

Interactive mode with pre-configured parameters:

# Launch file selector with all analysis tabs enabled
cargo run "" --enable-security --enable-performance --enable-errors --enable-bots --enable-sparkline --enable-heatmap --count=1000

# Launch with custom settings
cargo run "" --enable-security --enable-performance --top=20 --count=500

# Launch with specific analysis tabs only
cargo run "" --enable-security --enable-errors --count=2000

Direct file analysis with all features:

# Analyze with all tabs enabled
logutil access.log --enable-security --enable-performance --enable-errors --enable-bots --enable-sparkline --enable-heatmap --count=1000

# Security-focused analysis
logutil access.log --enable-security --enable-errors --top=50

# Performance monitoring
logutil access.log --enable-performance --enable-sparkline --enable-heatmap

Installation

Quick Install (Recommended)

Download the latest release for your platform:

Linux (x86_64):

curl -L -o /usr/local/bin/logutil https://github.com/s00d/logutil/releases/latest/download/logutil-x86_64-unknown-linux-gnu
chmod +x /usr/local/bin/logutil

Linux (ARM64):

curl -L -o /usr/local/bin/logutil https://github.com/s00d/logutil/releases/latest/download/logutil-aarch64-unknown-linux-gnu
chmod +x /usr/local/bin/logutil

macOS:

curl -L -o /usr/local/bin/logutil https://github.com/s00d/logutil/releases/latest/download/logutil-x86_64-apple-darwin
chmod +x /usr/local/bin/logutil

Build from Source

1. Install Rust:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rust-lang.org | sh

2. Clone and Build:

git clone https://github.com/s00d/logutil.git
cd logutil
cargo build --release

3. Install:

sudo cp target/release/logutil /usr/local/bin/

Usage

Basic Usage

Monitor a log file in real-time:

logutil /var/log/nginx/access.log

Analyze the entire log file:

logutil /var/log/nginx/access.log --count=-1

Show only the last 1000 lines:

logutil /var/log/nginx/access.log --count=1000

Advanced Usage

Custom regex pattern:

logutil /var/log/nginx/access.log --regex='^(\S+) - - \[(.*?)\] "(\S+) (\S+) HTTP/\d+\.\d+" (\d+) (\d+) "([^"]*)" "([^"]*)"$'

Show top 20 entries:

logutil /var/log/nginx/access.log --top=20

Disable automatic cleanup:

logutil /var/log/nginx/access.log --no-clear

Custom date format:

logutil /var/log/nginx/access.log --date-format="%d/%b/%Y:%H:%M:%S %z"

Load regex from file:

logutil /var/log/nginx/access.log --regex=/path/to/regex.txt

Tab Management

By default, only the core tabs (Overview, Requests, Detailed) are enabled. Additional tabs can be enabled using command-line flags:

Enable Security tab (detect suspicious activity, attacks, etc.):

logutil /var/log/nginx/access.log --enable-security

Enable Performance tab (monitor response times, slow requests):

logutil /var/log/nginx/access.log --enable-performance

Enable Errors tab (track error codes and failed requests):

logutil /var/log/nginx/access.log --enable-errors

Enable Bots tab (detect bot traffic and crawlers):

logutil /var/log/nginx/access.log --enable-bots

Enable Sparkline tab (real-time request rate visualization):

logutil /var/log/nginx/access.log --enable-sparkline

Enable Heatmap tab (hourly traffic patterns visualization):

logutil /var/log/nginx/access.log --enable-heatmap

Enable all tabs:

logutil /var/log/nginx/access.log --enable-security --enable-performance --enable-errors --enable-bots --enable-sparkline --enable-heatmap

Console Output Mode

Show top URLs in console:

logutil /var/log/nginx/access.log --show-urls --top=10

Show top IPs in console:

logutil /var/log/nginx/access.log --show-ips --top=10

Interactive Interface

Navigation

• Tab or t: Switch between tabs (Overview → Requests → Detailed → Sparkline → Heatmap)
• ↑/↓: Navigate through lists
• ←/→: Switch between panels or pages
• Enter: Copy selected item to clipboard (in Overview tab)
• q or Ctrl+C: Quit the application

Tabs Overview

1. Overview Tab

• Left Panel: Top IP addresses with request counts and last update times
• Right Panel: Top URLs with request types, domains, and statistics
• Bottom Panel: Full URL display for selected URL
• Navigation: Use arrow keys to switch between panels

2. Requests Tab

• Search Functionality: Type to filter log entries
• Pagination: Navigate through large result sets
• Real-time Updates: New requests appear automatically

3. Detailed Tab

• IP List: Select an IP address to view its details
• Request Details: View all requests from the selected IP
• Drill-down Analysis: Understand traffic patterns per IP

4. Sparkline Tab

• Real-time Timeline: Visual representation of request patterns
• Statistics: Min, max, average, and current request counts
• Time Range: Shows the time span of the data

5. Heatmap Tab

• Hourly Patterns: Color-coded request intensity by hour
• Multi-day View: Track patterns across multiple days
• Legend: Blue (low) → Green → Red (high) intensity

6. Security Tab

• Advanced Threat Detection: SQL Injection, XSS, Path Traversal, Command Injection, Brute Force
• Log Analysis: View highlighted suspicious patterns in log entries with visual indicators
• IP Blocking: Block/unblock suspicious IP addresses with Enter key
• Threat Levels: Visual indicators (🔴🟡🟢) for threat severity assessment
• Pattern Highlighting: Suspicious patterns are highlighted with icons in log details
• Security Summary: Comprehensive overview of all security threats and violations
• Log Detail View: Press Enter to view detailed logs with highlighted suspicious patterns
• Real-time Monitoring: Continuous monitoring and detection of security threats

7. Performance Tab

• Response Time Analysis: Tracks average, min, and max response times
• Slow Request Identification: Highlights requests taking longer than 1 second
• Requests Per Second: Real-time RPS calculation and monitoring
• Performance Tracking: Detailed tracking of slow requests with timestamps
• Performance Metrics: Real-time performance statistics
• Resource Usage: Total response size and throughput analysis

8. Errors Tab

• HTTP Error Analysis: Categorizes and counts error codes (4xx, 5xx)
• Error Pattern Detection: Identifies common error sources
• Error Distribution: Shows which URLs and IPs generate most errors
• Error Trends: Tracks error patterns over time

9. Bots Tab

• Bot Detection: Identifies crawlers, scrapers, and automated traffic
• Bot Classification: Categorizes different types of bots (Googlebot, Bingbot, etc.)
• Bot Activity Analysis: Tracks bot behavior patterns
• Bot Traffic Statistics: Quantifies bot vs human traffic

Configuration

Command Line Options

Option	Description	Default
file	Path to the log file	Required
--count	Lines to read from end (0=tail, -1=all)	0
--regex	Regex pattern or file path	Nginx default
--date-format	Date parsing format	%d/%b/%Y:%H:%M:%S %z
--top	Number of top entries to show	10
--no-clear	Disable automatic cleanup	false
--show-urls	Output top URLs to console	false
--show-ips	Output top IPs to console	false
--log-to-file	Enable logging to app.log	false
--enable-security	Enable Security tab	false
--enable-performance	Enable Performance tab	false
--enable-errors	Enable Errors tab	false
--enable-bots	Enable Bots tab	false
--enable-sparkline	Enable Sparkline tab	false
--enable-heatmap	Enable Heatmap tab	false

Supported Log Formats

1. Nginx Access Log (Default)

^(\S+) - ".+" \[(.*?)\] \d+\.\d+ "(\S+)" "(\S+) (\S+?)(?:\?.*?)? "

Date Format: %d/%b/%Y:%H:%M:%S %z

2. Apache Access Log

^(\S+) \S+ \S+ \[.*?\] "\S+ (\S+?)(?:\?.*?)? \S+" \d+ \d+

Date Format: %d/%b/%Y:%H:%M:%S %z

3. Custom Format Example

^(\S+) - - \[(.*?)\] "(\S+) (\S+) HTTP/\d+\.\d+" (\d+) (\d+) "([^"]*)" "([^"]*)"$

Date Format: %d/%b/%Y:%H:%M:%S %z

Examples

Basic Monitoring

# Monitor nginx access logs in real-time
logutil /var/log/nginx/access.log

# Analyze entire log file
logutil /var/log/nginx/access.log --count=-1 --top=20

Custom Log Formats

# Apache access logs
logutil /var/log/apache2/access.log --regex='^(\S+) \S+ \S+ \[.*?\] "\S+ (\S+?)(?:\?.*?)? \S+" \d+ \d+'

# Custom application logs
logutil /var/log/app/access.log --regex='^(\S+) \[(.*?)\] (\S+) (\S+)'

Console Output

# Get top URLs for reporting
logutil /var/log/nginx/access.log --show-urls --top=10

# Get top IPs for security analysis
logutil /var/log/nginx/access.log --show-ips --top=20

Advanced Configuration

# Custom regex from file
logutil /var/log/nginx/access.log --regex=/etc/logutil/patterns.txt

# Disable cleanup for long-term analysis
logutil /var/log/nginx/access.log --no-clear --count=-1

# Custom date format
logutil /var/log/nginx/access.log --date-format="%Y-%m-%d %H:%M:%S"

Performance Considerations

Memory Management

• Automatic Cleanup: Removes entries older than 20 minutes when over 10,000 entries
• Configurable: Use --no-clear to disable automatic cleanup
• Efficient: Minimal memory footprint even with large log files

Processing Speed

• Real-time: Processes new lines as they appear
• Optimized: Efficient regex matching and data structures
• Scalable: Handles high-traffic logs without performance degradation

File Handling

• Smart Reading: Only processes new lines when tailing
• Error Recovery: Gracefully handles file rotation and truncation
• Progress Tracking: Shows loading progress for large files

Troubleshooting

Common Issues

1. "No match for line" errors

• Check your regex pattern with --regex option
• Verify log format matches the expected pattern
• Use --log-to-file to debug parsing issues

2. High memory usage

• Enable automatic cleanup (default behavior)
• Use --count to limit initial processing
• Consider using --no-clear only for short-term analysis

3. Slow performance with large files

• Use --count=1000 to limit initial processing
• Ensure regex pattern is optimized
• Check system resources (CPU, memory)

Debug Mode

# Enable debug logging
logutil /var/log/nginx/access.log --log-to-file

# Check the generated app.log file for errors
tail -f app.log

Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

Development Setup

git clone https://github.com/s00d/logutil.git
cd logutil
cargo build
cargo test