| Crates.io | vault-audit-tools |
| lib.rs | vault-audit-tools |
| version | 0.11.2 |
| created_at | 2025-10-21 05:08:21.054995+00 |
| updated_at | 2025-12-02 23:07:34.109212+00 |
| description | High-performance command-line tools for analyzing HashiCorp Vault audit logs with intelligent ephemeral entity detection |
| homepage | https://github.com/trenner1/hashicorp-vault-audit-analysis |
| repository | https://github.com/trenner1/hashicorp-vault-audit-analysis |
| max_upload_size | |
| id | 1893282 |
| size | 612,276 |
High-performance command-line tools for analyzing HashiCorp Vault audit logs, written in Rust.
.gz and .zst files without manual decompressioncd vault-audit-tools
cargo install --path .
This installs the vault-audit binary to ~/.cargo/bin/.
Download from the Releases page.
After installation, enable tab completion for your shell:
# Bash (Linux) - single command
sudo mkdir -p /usr/local/etc/bash_completion.d && \
vault-audit generate-completion bash | sudo tee /usr/local/etc/bash_completion.d/vault-audit > /dev/null && \
echo "Completion installed. Restart your shell or run: source /usr/local/etc/bash_completion.d/vault-audit"
# Bash (macOS with Homebrew) - single command
mkdir -p $(brew --prefix)/etc/bash_completion.d && \
vault-audit generate-completion bash > $(brew --prefix)/etc/bash_completion.d/vault-audit && \
echo "Completion installed. Restart your shell or run: source $(brew --prefix)/etc/bash_completion.d/vault-audit"
# Zsh - single command
mkdir -p ~/.zsh/completions && \
vault-audit generate-completion zsh > ~/.zsh/completions/_vault-audit && \
grep -q 'fpath=(~/.zsh/completions $fpath)' ~/.zshrc || echo 'fpath=(~/.zsh/completions $fpath)' >> ~/.zshrc && \
grep -q 'autoload -Uz compinit && compinit' ~/.zshrc || echo 'autoload -Uz compinit && compinit' >> ~/.zshrc && \
echo "Completion installed. Restart your shell or run: source ~/.zshrc"
# Fish - single command
mkdir -p ~/.config/fish/completions && \
vault-audit generate-completion fish > ~/.config/fish/completions/vault-audit.fish && \
echo "Completion installed. Restart your shell."
# PowerShell (Windows/Cross-platform) - single command
$profileDir = Split-Path $PROFILE; New-Item -ItemType Directory -Force -Path $profileDir | Out-Null; vault-audit generate-completion powershell | Out-File -Append -FilePath $PROFILE -Encoding utf8; Write-Host "Completion installed. Restart PowerShell or run: . `$PROFILE"
# Elvish - single command
mkdir -p ~/.config/elvish/lib && \
vault-audit generate-completion elvish > ~/.config/elvish/lib/vault-audit.elv && \
grep -q 'use vault-audit' ~/.config/elvish/rc.elv || echo 'use vault-audit' >> ~/.config/elvish/rc.elv && \
echo "Completion installed. Restart your shell."
Git Bash users need special handling since ~ doesn't expand in output redirection:
# Single command installation for Git Bash
mkdir -p "$HOME/.bash_completions" && \
vault-audit generate-completion bash > "$HOME/.bash_completions/vault-audit" && \
grep -q 'source "$HOME/.bash_completions/vault-audit"' ~/.bashrc || echo 'source "$HOME/.bash_completions/vault-audit"' >> ~/.bashrc && \
echo "Completion installed. Restart Git Bash or run: source ~/.bashrc"
Troubleshooting:
$HOME variable instead of ~ for paths in Git Bashls -la "$HOME/.bash_completions/vault-audit"grep vault-audit ~/.bashrcsystem-overview - High-level overview of all operations, entities, and auth methods (parallel processing)entity-gaps - Identify operations without entity IDs (no-entity operations) (parallel processing)path-hotspots - Find most accessed paths with optimization recommendations (parallel processing)k8s-auth - Analyze Kubernetes/OpenShift authentication patterns and entity churn (parallel processing)token-analysis - Unified token operations analysis with abuse detection and CSV export (parallel processing)
entity-analysis - Unified entity lifecycle analysis (recommended)
churn - Multi-day entity lifecycle tracking with ephemeral detectioncreation - Entity creation patterns by authentication pathpreprocess - Extract entity mappings (auto-generated by default)gaps - Detect activity gapstimeline - Individual entity operation timelineclient-activity - Query Vault for client activity metrics by mountentity-list - Export complete entity list from Vault (for baseline analysis)kv-mounts - Enumerate KV secret mounts with optional depth-based tree traversal
--depth N)vault-audit kv-mounts --format stdoutvault-audit kv-mounts --depth 2 --format csv --output kv-inventory.csvauth-mounts - Enumerate authentication mounts with role/user discovery
--depth > 0)vault-audit auth-mounts --format stdoutvault-audit auth-mounts --depth 0 --format json (mounts only, no roles)kv-analysis - Unified KV secrets analysis (recommended)
analyze - Analyze KV usage by path and entity (generates CSV) (parallel processing)compare - Compare KV usage between two time periods (CSV comparison)summary - Summarize KV secret usage from CSV exports (CSV analysis)kv-analyzer - DEPRECATED: Use kv-analysis analyze insteadkv-compare - DEPRECATED: Use kv-analysis compare insteadkv-summary - DEPRECATED: Use kv-analysis summary insteadMost commands analyze audit log files and do not require any Vault API access. The following commands interact with Vault's API and require a token with specific permissions.
These commands only read audit log files:
system-overview, path-hotspots, entity-gapstoken-analysis, k8s-auth, airflow-pollingentity-analysis (all subcommands)kv-analysis (all subcommands)kv-mounts CommandEnumerates all KV secret mounts and optionally lists their contents in a tree structure.
Required ACL Policy:
# List and read secret mounts
path "sys/mounts" {
capabilities = ["read"]
}
# List KV v2 secrets (for each mount discovered)
path "+/metadata/*" {
capabilities = ["list"]
}
# List KV v1 secrets (for each mount discovered)
path "+/*" {
capabilities = ["list"]
}
auth-mounts CommandEnumerates all authentication mounts and optionally lists roles, users, and groups within each mount.
Required ACL Policy:
# List and read auth mounts
path "sys/auth" {
capabilities = ["read"]
}
# List roles for kubernetes, approle, jwt/oidc auth methods
path "auth/+/role" {
capabilities = ["list"]
}
# List users for userpass and ldap auth methods
path "auth/+/users" {
capabilities = ["list"]
}
# List groups for ldap auth method
path "auth/+/groups" {
capabilities = ["list"]
}
entity-list CommandExports complete entity list from Vault for baseline analysis.
Required ACL Policy:
# Read entity information
path "identity/entity/id" {
capabilities = ["list"]
}
path "identity/entity/id/*" {
capabilities = ["read"]
}
# Read auth mount configuration
path "sys/auth" {
capabilities = ["read"]
}
client-activity CommandQueries Vault's activity log API for client usage metrics.
Required ACL Policy:
# Export client activity data
path "sys/internal/counters/activity/export" {
capabilities = ["read"]
}
# Read mount configuration (secret engines and auth methods)
path "sys/mounts" {
capabilities = ["read"]
}
path "sys/auth" {
capabilities = ["read"]
}
Option 1: Separate policies for each command
# For entity-list command
vault policy write vault-audit-entity-list - <<EOF
path "identity/entity/id" {
capabilities = ["list"]
}
path "identity/entity/id/*" {
capabilities = ["read"]
}
path "sys/auth" {
capabilities = ["read"]
}
EOF
vault token create -policy=vault-audit-entity-list
# For client-activity command
vault policy write vault-audit-client-activity - <<EOF
path "sys/internal/counters/activity/export" {
capabilities = ["read"]
}
path "sys/mounts" {
capabilities = ["read"]
}
path "sys/auth" {
capabilities = ["read"]
}
EOF
vault token create -policy=vault-audit-client-activity
Option 2: Combined policy for all API commands
vault policy write vault-audit-tools - <<EOF
# Entity list access
path "identity/entity/id" {
capabilities = ["list"]
}
path "identity/entity/id/*" {
capabilities = ["read"]
}
# Client activity access
path "sys/internal/counters/activity/export" {
capabilities = ["read"]
}
# Mount information (used by both commands)
path "sys/mounts" {
capabilities = ["read"]
}
path "sys/auth" {
capabilities = ["read"]
}
EOF
vault token create -policy=vault-audit-tools
Option 3: Use existing token
If you already have a Vault token with appropriate permissions (e.g., root token for testing, or admin token), you can use it:
export VAULT_ADDR="https://vault.example.com:8200"
export VAULT_TOKEN="hvs.your-token-here"
vault-audit entity-list --output entities.csv
vault-audit client-activity --start-time 2025-10-01T00:00:00Z --end-time 2025-10-31T23:59:59Z
Commands that interact with Vault API respect standard Vault environment variables:
VAULT_ADDR - Vault server address (e.g., https://vault.example.com:8200)VAULT_TOKEN - Authentication token for API accessVAULT_NAMESPACE - Vault namespace for API requests (e.g., tenant1, admin/team-a)VAULT_SKIP_VERIFY - Skip TLS certificate verification (set to 1, true, or yes) - USE ONLY FOR TESTINGVAULT_CACERT - Path to CA certificate for TLS verificationYou can also provide these via command-line flags:
# Query entities from a specific namespace
vault-audit entity-list \
--vault-addr https://vault.example.com:8200 \
--vault-token hvs.xxxxx \
--vault-namespace tenant1 \
--output entities.csv
# Client activity for a namespace
vault-audit client-activity \
--start 2025-10-01T00:00:00Z \
--end 2025-10-31T23:59:59Z \
--vault-namespace admin/security \
--output activity.csv
# Skip TLS verification (dev/test only)
vault-audit entity-list --insecure --output entities.csv
Vault Enterprise supports namespaces for multi-tenant isolation. This toolset provides comprehensive namespace support for both API commands and audit log analysis.
Commands that query Vault's API (entity-list and client-activity) support the --vault-namespace flag (or VAULT_NAMESPACE environment variable) to target a specific namespace:
# Set namespace via environment variable
export VAULT_NAMESPACE="tenant1"
vault-audit entity-list --output tenant1-entities.csv
# Or use command-line flag
vault-audit client-activity \
--start 2025-10-01T00:00:00Z \
--end 2025-10-31T23:59:59Z \
--vault-namespace admin/security
Audit logs from namespaced Vault clusters include namespace information in each entry. Use the --namespace-filter flag to analyze logs from a specific namespace:
# Analyze only operations in the "prod" namespace
vault-audit system-overview audit.log --namespace-filter root
# Show system overview for specific namespace
vault-audit system-overview logs/*.log.gz --namespace-filter tenant1
# All other audit log commands can filter by namespace using similar patterns
vault-audit token-analysis audit.log --namespace-filter admin
vault-audit kv-analysis analyze audit.log --namespace-filter myapp --output kv-usage.csv
Note: Namespace filtering for audit log commands currently supported:
system-overview - Full support with --namespace-filterrequest.namespace.id field--namespace-filter root for operations in the root namespaceView the full API documentation with detailed module and function descriptions:
# Generate and open documentation in your browser
cd vault-audit-tools
cargo doc --no-deps --open
The documentation includes:
Once published to crates.io, the documentation will be automatically available at docs.rs/vault-audit-tools.
Get detailed help for any command:
# General help
vault-audit --help
# Unified command help
vault-audit entity-analysis --help
vault-audit token-analysis --help
vault-audit kv-analysis --help
# Subcommand-specific help
vault-audit entity-analysis churn --help
vault-audit kv-analysis analyze --help
airflow-polling - Analyze Airflow secret polling patterns with burst rate detection (parallel processing)generate-completion - Generate shell completion scriptsAll commands automatically detect and decompress .gz (gzip) and .zst (zstandard) files:
# Analyze compressed files directly - no manual decompression needed
vault-audit system-overview vault_audit.log.gz
# Mix compressed and uncompressed files
vault-audit entity-churn day1.log.gz day2.log day3.log.zst
# Glob patterns work with compressed files
vault-audit path-hotspots logs/*.log.gz
# Streaming decompression - no temp files, no extra disk space needed
vault-audit token-analysis huge_file.log.gz # processes 1.79GB compressed → 13.8GB uncompressed
Performance: Compressed file processing maintains full speed (~57 MB/s) with no memory overhead thanks to streaming decompression.
When analyzing token operations, it's important to understand the difference between entities and accessors:
Entity (User/Service Identity):
Accessor (Individual Token):
When to use each view:
--export): Shows per-accessor detail for token lifecycle analysis, rotation patterns, and identifying specific problematic tokens# See entity-level summary (6,091 entities with totals)
vault-audit token-analysis vault_audit.log
# Export accessor-level detail (907 individual tokens with timestamps)
vault-audit token-analysis vault_audit.log --export tokens.csv
# Filter to high-volume tokens only
vault-audit token-analysis vault_audit.log --export tokens.csv --min-operations 1000
# Get system overview (works with plain or compressed files)
vault-audit system-overview vault_audit.log
vault-audit system-overview vault_audit.log.gz
# Analyze multiple days without concatenation
vault-audit system-overview logs/vault_audit.2025-10-*.log
# Find authentication issues
vault-audit k8s-auth vault_audit.log
# Detect token abuse across multiple compressed files
vault-audit token-analysis day1.log.gz day2.log.gz day3.log.gz --abuse-threshold 5000
All audit log commands support multiple files (compressed or uncompressed) for historical analysis:
# Week-long system overview with compressed files
vault-audit system-overview vault_audit.2025-10-{07,08,09,10,11,12,13}.log.gz
# Month-long entity churn tracking (auto-preprocesses entity mappings)
vault-audit entity-analysis churn october/*.log.gz
# Multi-day token operations analysis with mixed file types
vault-audit token-analysis logs/vault_audit.*.log --export token_ops.csv
# Path hotspot analysis across 30 days of compressed logs
vault-audit path-hotspots logs/vault_audit.2025-10-*.log.zst
Enumerate and discover all mounts, roles, and secrets without needing to know mount names in advance:
# Discover all KV mounts and their complete tree structure
vault-audit kv-mounts --format stdout
# List only KV mount points (no traversal into secrets)
vault-audit kv-mounts --depth 0 --format csv
# Traverse 2 levels deep and save to CSV
vault-audit kv-mounts --depth 2 --format csv --output kv-inventory.csv
# Get complete KV structure as JSON for further processing
vault-audit kv-mounts --format json --output kv-tree.json
# Discover all auth mounts with their roles and users
vault-audit auth-mounts --format stdout
# List only auth mount points (no role enumeration)
vault-audit auth-mounts --depth 0 --format json
# Export auth configuration with roles to CSV
vault-audit auth-mounts --format csv --output auth-config.csv
Example Output - KV Mounts (stdout format):
KV Mounts:
================================================================================
Path: kv/
Mount Type: kv
Version: 2
Description: key/value secret storage
Accessor: kv_f1c7d8b2
Children (11 paths):
kv/
└── dev/
└── apps/
├── backend-service/
│ ├── config
│ └── example
├── frontend-app/
│ ├── config
│ └── example
└── mobile-app/
├── config
└── example
Example Output - Auth Mounts (stdout format):
Auth Mounts:
================================================================================
Path: kubernetes/
Type: kubernetes
Description:
Accessor: auth_kubernetes_e954d6e1
Roles/Users (5):
├── backend-service
├── cache-service
├── database-operator
├── frontend-app
└── monitoring
Path: approle/
Type: approle
Description:
Accessor: auth_approle_6a0e0046
Roles/Users (5):
├── ansible
├── automation
├── ci-pipeline
├── monitoring-agent
└── terraform
Commands automatically use parallel processing when analyzing multiple files:
# Single file - uses sequential processing
vault-audit system-overview vault_audit.log
# Multiple files - automatically parallelizes across all CPU cores
vault-audit system-overview day1.log day2.log day3.log day4.log
# Glob expansion with many files - maximizes CPU utilization
vault-audit path-hotspots logs/*.log.gz # processes all files concurrently
Commands with Parallel Processing:
system-overview - System-wide audit analysisentity-analysis gaps - Operations without entity IDsentity-gaps - Operations without entity IDs (deprecated, use entity-analysis)path-hotspots - Most accessed pathsk8s-auth - Kubernetes authentication analysisairflow-polling - Airflow polling pattern detectionkv-analysis analyze - KV secrets usage analysistoken-analysis - Token operations analysisHow it works:
Performance benefits:
# Analyze entity creation patterns by auth path (auto-preprocessing enabled)
vault-audit entity-analysis creation vault_audit.log
# Track entity lifecycle across multiple days (auto-preprocessing enabled)
vault-audit entity-analysis churn day1.log day2.log day3.log --baseline baseline_entities.json
# Analyze specific entity behavior
vault-audit entity-analysis timeline --entity-id <UUID> day1.log day2.log
# Detect activity gaps (potential security issues)
vault-audit entity-analysis gaps vault_audit.log --window-seconds 300
# Token analysis with multiple output modes
vault-audit token-analysis vault_audit.log # Summary view (per-entity)
vault-audit token-analysis vault_audit.log --abuse-threshold 10000 # Abuse detection
vault-audit token-analysis vault_audit.log --filter lookup,revoke # Filter operation types
vault-audit token-analysis vault_audit.log --export tokens.csv # Export per-accessor detail (907 tokens)
vault-audit token-analysis vault_audit.log --export tokens.csv --min-operations 1000 # High-volume tokens only
# Analyze Airflow polling with burst detection
vault-audit airflow-polling vault_audit.log
# Query Vault API for client activity metrics
vault-audit client-activity --start 2025-10-01T00:00:00Z --end 2025-11-01T00:00:00Z
# Generate KV usage report (new unified command with parallel processing)
vault-audit kv-analysis analyze vault_audit.log --kv-prefix "appcodes/" --output kv_usage.csv
# Multi-file analysis - 40% faster with parallel processing
vault-audit kv-analysis analyze logs/*.log --output kv_usage.csv
# Compare two time periods
vault-audit kv-analysis compare old_usage.csv new_usage.csv
# Get summary statistics
vault-audit kv-analysis summary kv_usage.csv
Tested on production audit logs:
Single File:
Multi-File Sequential (7 days):
Multi-File Parallel (multiple files, multi-core):
Compressed Files:
Parallel Processing Benchmarks (Real-World):
kv-analysis analyze)
Most commands produce formatted text output with:
CSV export commands generate standard CSV files for:
cd vault-audit-tools
cargo build --release
cargo test
To measure performance and memory usage on macOS/Linux:
# macOS - shows execution time and peak memory usage
/usr/bin/time -l ./target/release/vault-audit <command> <args> 2>&1 | grep -E "(real|maximum resident)"
# Linux - shows execution time and peak memory usage
/usr/bin/time -v ./target/release/vault-audit <command> <args> 2>&1 | grep -E "(Elapsed|Maximum resident)"
# Example: Benchmark KV analysis
/usr/bin/time -l ./target/release/vault-audit kv-analysis analyze logs/*.log
Key metrics:
MIT
Contributions welcome! Please open an issue or PR.
For issues or questions, please open a GitHub issue.