• NviWatch

  • Demo

  • Benchmarks

    • Benchmark Results
    • Installation Size Comparison
    • Analysis

  • Features

  • InfluxDB Integration

    • Setting up InfluxDB
    • Running NviWatch with InfluxDB Logging
    • Dashboard Features
    • Custom Configuration

  • Installing and Using the Tool

    • Option 1: Download Pre-built Binary
    • Option 2: Install via Cargo
    • Option 3: Build from Source

  • Usage

  • Key Bindings

  • View Modes

    • 1. Default Mode
    • 2. Bar Mode
    • 3. Tabbed Mode: GPU graphs in tabs for multi GPU nodes

  • Star History

  • License

  • Contributing

  • Acknowledgments

NviWatch

NviWatch is an interactive terminal user interface (TUI) application for monitoring NVIDIA GPU devices and processes. Built with Rust, it provides real-time insights into GPU performance metrics, including temperature, utilization, memory usage, and power consumption.

Demo

https://github.com/user-attachments/assets/176565fe-4467-4129-b783-071543c52bf4

Benchmarks

We conducted performance benchmarks comparing nviwatch with other popular GPU monitoring tools: nvtop, nvitop, and gpustat. The results demonstrate nviwatch's efficiency in terms of CPU and memory usage. All tools except nvitop were run at 100ms interval. nvitop was set to 250ms because that is the minimum allowed value. The benchmark scripts and logs are available in the benchmarks folder. The test system had 32 GB RAM.

Benchmark Results

Installation Size Comparison

We used python-package-size for determining the pip package sizes. For nvtop We used this apt show nvtop | grep Installed-Size.

Analysis

• CPU Usage: nviwatch demonstrates excellent CPU efficiency, with an average usage of just 0.28% and a maximum of 10%. It outperforms gpustat and nvitop and is comparable to nvtop in terms of average CPU usage. Important to note that nvtop supports more GPUs than just Nvidia so nviwatch isn't a complete alternative for nvwatch.

• Memory Usage: nviwatch shows the lowest memory footprint among all tested tools, using only 0.12% of system memory on average, which translates to about 18.26 MB. This is notably less than nvitop (41.07 MB) and gpustat (33.82 MB), and slightly better than nvtop (20.46 MB).

• Consistency: nviwatch maintains consistent memory usage throughout its operation, as indicated by the identical mean and max values for memory usage.

• Package Size: At 1.98 MB, nviwatch offers a balanced package size. It's significantly smaller than nvitop (4.1 MB) and gpustat (3.7 MB), while being significantly larger than nvtop (106 KB).

Features

• Real-Time Monitoring: View real-time data on GPU temperature, utilization, memory usage, and power consumption.
• Process Management: Monitor processes running on the GPU and terminate them directly from the interface.
• Graphical Display: Visualize GPU performance metrics using bar charts and tabbed graphs.
• Customizable Refresh Rate: Set the refresh interval for updating GPU metrics.
• InfluxDB Integration: Stream GPU metrics to InfluxDB for long-term monitoring and visualization.

InfluxDB Integration

NviWatch supports streaming GPU metrics to InfluxDB for persistent storage and advanced visualization. This enables long-term monitoring, historical analysis, and custom dashboards.

Setting up InfluxDB

1. Run the setup script (recommended for Ubuntu/Debian):

   chmod +x scripts/setup_influxdb.sh
   ./scripts/setup_influxdb.sh
   

   This script will:

   • Install InfluxDB 3.x (with fallback to InfluxDB 2.x if 3.x is not available)
   • Configure the database with default settings:
     • Username: admin
     • Password: password12345
     • Organization: my-org
     • Bucket: gpu-metrics
     • Data retention: 7 days
   • Import a pre-configured GPU monitoring dashboard

2. Get your admin token:

   influx auth list
   

3. Access the InfluxDB dashboard:

   • Open your web browser and navigate to http://localhost:8086
   • Login with the credentials:
     • Username: admin
     • Password: password12345

Running NviWatch with InfluxDB Logging

To start NviWatch with InfluxDB integration, use the following command-line options:

nviwatch \
  --influx-url "http://localhost:8086" \
  --influx-org "my-org" \
  --influx-bucket "gpu-metrics" \
  --influx-token "your-admin-token-here"

Example with all options:

nviwatch \
  --watch 1000 \
  --tabbed-graphs \
  --influx-url "http://localhost:8086" \
  --influx-org "my-org" \
  --influx-bucket "gpu-metrics" \
  --influx-token "your-admin-token-here"

Dashboard Features

The included dashboard provides:

• Memory Utilization: Real-time GPU memory usage tracking
• GPU Utilization: Performance utilization percentage over time
• Temperature: GPU temperature monitoring
• Power Usage: Power consumption tracking in watts

Custom Configuration

You can modify the scripts/setup_influxdb.sh script to change default settings:

• Edit the configuration variables at the top of the script
• Re-run the script to apply changes
• Update your nviwatch command with the new credentials

Installing and Using the Tool

Option 1: Download Pre-built Binary

1. Go to the project's GitHub repository.

2. Navigate to the "Releases" section.

3. Download the latest binary release for linux.

4. Once downloaded, open a terminal and navigate to the directory containing the downloaded binary.

5. Make the binary executable with the following command:

   chmod +x nviwatch
   

6. You can now run the tool using:

   ./nviwatch
   

Option 2: Install via Cargo

If you have Rust and Cargo installed on your system, you can easily install NviWatch directly from crates.io:

1. Open a terminal and run the following command:

   cargo install nviwatch
   

2. Once the installation is complete, you can run NviWatch from anywhere in your terminal:

   nviwatch
   

Note: Ensure you have the NVIDIA Management Library (NVML) available on your system before running NviWatch.

Option 3: Build from Source

To build and run NviWatch, ensure you have Rust and Cargo installed on your system. You will also need the NVIDIA Management Library (NVML) available.

1. Clone the repository:

   git clone https://github.com/msminhas93/nviwatch.git
   cd nviwatch
   

2. Build the project:

   cargo build --release
   

3. Run the application:

   chmod +x ./target/release/nviwatch
   ./target/release/nviwatch
   

Usage

NviWatch provides a command-line interface with several options:

• -w, --watch <MILLISECONDS>: Set the refresh interval in milliseconds. Default is 1000 ms.
• -t, --tabbed-graphs: Display GPU graphs in a tabbed view.
• -b, --bar-chart: Display GPU graphs as bar charts.
• --influx-url <URL>: InfluxDB server URL (e.g., "http://localhost:8086").
• --influx-org <ORG>: InfluxDB organization name.
• --influx-bucket <BUCKET>: InfluxDB bucket name for storing metrics.
• --influx-token <TOKEN>: InfluxDB authentication token.

Basic examples:

# Run with default settings
./nviwatch

# Run with custom refresh rate and tabbed graphs
./nviwatch --watch 500 --tabbed-graphs

# Run with InfluxDB integration
./nviwatch \
  --influx-url "http://localhost:8086" \
  --influx-org "my-org" \
  --influx-bucket "gpu-metrics" \
  --influx-token "your-admin-token-here"

# Run with all features enabled
./nviwatch \
  --watch 1000 \
  --tabbed-graphs \
  --influx-url "http://localhost:8086" \
  --influx-org "my-org" \
  --influx-bucket "gpu-metrics" \
  --influx-token "your-admin-token-here"

Key Bindings

• q: Quit the application
• ↑/↓: Navigate through the list of processes
• ←/→: Switch between GPU tabs (when using tabbed graphs)
• x: Terminate the selected process
• d: Switch to default view mode
• t: Switch to tabbed graphs view mode
• b: Switch to bar charts view mode

View Modes

The application supports three different view modes:

1. Default Mode

Shows all GPU information in a single view

2. Bar Mode

Presents GPU information using bar charts

3. Tabbed Mode: GPU graphs in tabs for multi GPU nodes

Displays GPU graphs in a tabbed interface

You can switch between these modes at any time using the corresponding key bindings.

Star History

License

This project is licensed under the GNU General Public License v3.0. See the LICENSE file for details.

Contributing

Contributions are welcome! Please open an issue or submit a pull request for any improvements or bug fixes.

Acknowledgments

• Built with Rust and Ratatui.
• Utilizes the NVIDIA Management Library (NVML) via the nvml_wrapper crate.