diffai

AI/ML specialized diff tool for PyTorch, Safetensors, NumPy, and MATLAB files

A next-generation diff tool specialized for AI/ML and scientific computing workflows that understands model structures, tensor statistics, and numerical data - not just text changes. Native support for PyTorch, Safetensors, NumPy arrays, MATLAB files, and structured data.

# Traditional diff fails with binary model files
$ diff model_v1.safetensors model_v2.safetensors
Binary files model_v1.safetensors and model_v2.safetensors differ

# diffai shows meaningful model changes with full analysis
$ diffai model_v1.safetensors model_v2.safetensors
  ~ fc1.bias: mean=0.0018->0.0017, std=0.0518->0.0647
  ~ fc1.weight: mean=-0.0002->-0.0001, std=0.0514->0.0716
  ~ fc2.weight: mean=-0.0008->-0.0018, std=0.0719->0.0883
  gradient_analysis: flow_health=healthy, norm=0.015000, ratio=1.0500
  deployment_readiness: readiness=0.92, strategy=blue_green, risk=low
  quantization_analysis: compression=0.0%, speedup=1.8x, precision_loss=1.5%

[WARNING]
• Memory usage increased moderately (+250MB). Monitor resource consumption.
• Inference speed moderately affected (1.3x slower). Consider optimization opportunities.

Key Features

• AI/ML Native: Direct support for PyTorch (.pt/.pth), Safetensors (.safetensors), NumPy (.npy/.npz), and MATLAB (.mat) files
• Tensor Analysis: Automatic calculation of tensor statistics (mean, std, min, max, shape, memory usage)
• Comprehensive ML Analysis: 30+ analysis functions including quantization, architecture, memory, convergence, anomaly detection, and deployment readiness - all enabled by default
• Scientific Data Support: NumPy arrays and MATLAB matrices with complex number support
• Pure Rust Implementation: No system dependencies, works on Windows/Linux/macOS without additional installations
• Multiple Output Formats: Colored CLI, JSON for MLOps integration, YAML for human-readable reports
• Fast and Memory Efficient: Built in Rust for handling large model files efficiently

Why diffai?

Traditional diff tools are inadequate for AI/ML workflows:

diffai vs MLOps Tools

diffai complements existing MLOps tools by focusing on structural comparison rather than experiment management:

Installation

From crates.io (Recommended)

cargo install diffai

From Source

git clone https://github.com/kako-jun/diffai.git
cd diffai
cargo build --release

Quick Start

Basic Model Comparison

# Compare PyTorch models with full analysis (default)
diffai model_old.pt model_new.pt

# Compare Safetensors with complete ML analysis
diffai checkpoint_v1.safetensors checkpoint_v2.safetensors

# Compare NumPy arrays
diffai data_v1.npy data_v2.npy

# Compare MATLAB files
diffai experiment_v1.mat experiment_v2.mat

ML Analysis Features

# Full ML analysis runs automatically for PyTorch/Safetensors
diffai baseline.safetensors finetuned.safetensors
# Outputs: 30+ analysis types including quantization, architecture, memory, etc.

# JSON output for automation
diffai model_v1.safetensors model_v2.safetensors --output json

# Detailed diagnostic information with verbose mode
diffai model_v1.safetensors model_v2.safetensors --verbose

# YAML output for human-readable reports
diffai model_v1.safetensors model_v2.safetensors --output yaml

📚 Documentation

• Working Examples & Demonstrations - See diffai in action with real outputs
• API Documentation - Rust library documentation
• User Guide - Comprehensive usage guide
• ML Analysis Guide - Deep dive into ML-specific features

Supported File Formats

ML Model Formats

• Safetensors (.safetensors) - HuggingFace standard format
• PyTorch (.pt, .pth) - PyTorch model files with Candle integration

Scientific Data Formats

• NumPy (.npy, .npz) - NumPy arrays with full statistical analysis
• MATLAB (.mat) - MATLAB matrices with complex number support

Structured Data Formats

• JSON (.json) - JavaScript Object Notation
• YAML (.yaml, .yml) - YAML Ain't Markup Language
• TOML (.toml) - Tom's Obvious Minimal Language
• XML (.xml) - Extensible Markup Language
• INI (.ini) - Configuration files
• CSV (.csv) - Comma-separated values

ML Analysis Functions

Automatic Comprehensive Analysis (v0.3.4)

When comparing PyTorch or Safetensors files, diffai automatically runs 30+ ML analysis features:

Automatic Features Include:

• Statistical Analysis: Detailed tensor statistics (mean, std, min, max, shape, memory)
• Quantization Analysis: Analyze quantization effects and efficiency
• Architecture Comparison: Compare model architectures and structural changes
• Memory Analysis: Analyze memory usage and optimization opportunities
• Anomaly Detection: Detect numerical anomalies in model parameters
• Convergence Analysis: Analyze convergence patterns in model parameters
• Gradient Analysis: Analyze gradient information when available
• Deployment Readiness: Assess production deployment readiness
• Regression Testing: Automatic performance degradation detection
• Plus 20+ additional specialized features

Future Enhancements

• TensorFlow format support (.pb, .h5, SavedModel)
• ONNX format support
• Advanced visualization and charting features

Design Philosophy

diffai provides comprehensive analysis by default for ML models, eliminating choice paralysis. Users get all relevant insights without needing to remember or specify dozens of analysis flags.

Debugging and Diagnostics

Verbose Mode (--verbose / -v)

Get comprehensive diagnostic information for debugging and performance analysis:

# Basic verbose output
diffai model1.safetensors model2.safetensors --verbose

# Verbose with structured data filtering
diffai data1.json data2.json --verbose --epsilon 0.001 --ignore-keys-regex "^id$"

Verbose output includes:

• Configuration diagnostics: Format settings, filters, analysis modes
• File analysis: Paths, sizes, detected formats, processing context
• Performance metrics: Processing time, difference counts, optimization status
• Directory statistics: File counts, comparison summaries (with --recursive)

Example verbose output:

=== diffai verbose mode enabled ===
Configuration:
  Input format: Safetensors
  Output format: Cli
  ML analysis: Full analysis enabled (all 30 features)
  Epsilon tolerance: 0.001

File analysis:
  Input 1: model1.safetensors
  Input 2: model2.safetensors
  Detected format: Safetensors
  File 1 size: 1048576 bytes
  File 2 size: 1048576 bytes

Processing results:
  Total processing time: 1.234ms
  Differences found: 15
  ML/Scientific data analysis completed

📚 See Verbose Output Guide for detailed usage

Output Formats

CLI Output (Default)

Colored, human-readable output with intuitive symbols:

• ~ Changed tensors/arrays with statistical comparison
• + Added tensors/arrays with metadata
• - Removed tensors/arrays with metadata

JSON Output

Structured output for MLOps integration and automation:

diffai model1.safetensors model2.safetensors --output json | jq .

YAML Output

Human-readable structured output for documentation:

diffai model1.safetensors model2.safetensors --output yaml

Real-World Use Cases

Research & Development

# Compare model before and after fine-tuning (full analysis automatic)
diffai pretrained_model.safetensors finetuned_model.safetensors
# Outputs: learning_progress, convergence_analysis, parameter stats, and 27 more analyses

# Analyze architectural changes during development
diffai baseline_architecture.pt improved_architecture.pt
# Outputs: architecture_comparison, param_efficiency_analysis, and full ML analysis

MLOps & CI/CD

# Automated model validation in CI/CD (comprehensive analysis)
diffai production_model.safetensors candidate_model.safetensors
# Outputs: deployment_readiness, regression_test, risk_assessment, and 27 more analyses

# Performance impact assessment with JSON output for automation
diffai original_model.pt optimized_model.pt --output json
# Outputs: quantization_analysis, memory_analysis, performance_impact_estimate, etc.

Scientific Computing

# Compare NumPy experiment results
diffai baseline_results.npy new_results.npy

# Analyze MATLAB simulation data
diffai simulation_v1.mat simulation_v2.mat

# Compare compressed NumPy archives
diffai dataset_v1.npz dataset_v2.npz

Experiment Tracking

# Generate comprehensive reports
diffai experiment_baseline.safetensors experiment_improved.safetensors \
  --generate-report --markdown-output --review-friendly

# A/B test analysis
diffai model_a.safetensors model_b.safetensors \
  --statistical-significance --hyperparameter-comparison

Command-Line Options

Basic Options

• -f, --format <FORMAT> - Specify input file format
• -o, --output <OUTPUT> - Choose output format (cli, json, yaml)
• -r, --recursive - Compare directories recursively

Note: For ML models (PyTorch/Safetensors), comprehensive analysis including statistics runs automatically

Advanced Options

• --path <PATH> - Filter differences by specific path
• --ignore-keys-regex <REGEX> - Ignore keys matching regex pattern
• --epsilon <FLOAT> - Set tolerance for float comparisons
• --array-id-key <KEY> - Specify key for array element identification
• --sort-by-change-magnitude - Sort by change magnitude

Examples

Basic Tensor Comparison (Automatic)

$ diffai simple_model_v1.safetensors simple_model_v2.safetensors
anomaly_detection: type=none, severity=none, action="continue_training"
architecture_comparison: type1=feedforward, type2=feedforward, deployment_readiness=ready
convergence_analysis: status=converging, stability=0.92
gradient_analysis: flow_health=healthy, norm=0.021069
memory_analysis: delta=+0.0MB, efficiency=1.000000
quantization_analysis: compression=0.0%, speedup=1.8x, precision_loss=1.5%
regression_test: passed=true, degradation=-2.5%, severity=low
  ~ fc1.bias: mean=0.0018->0.0017, std=0.0518->0.0647
  ~ fc1.weight: mean=-0.0002->-0.0001, std=0.0514->0.0716
  ~ fc2.bias: mean=-0.0076->-0.0257, std=0.0661->0.0973
  ~ fc2.weight: mean=-0.0008->-0.0018, std=0.0719->0.0883
  ~ fc3.bias: mean=-0.0074->-0.0130, std=0.1031->0.1093
  ~ fc3.weight: mean=-0.0035->-0.0010, std=0.0990->0.1113

JSON Output for Automation

$ diffai baseline.safetensors improved.safetensors --output json
{
  "anomaly_detection": {"type": "none", "severity": "none"},
  "architecture_comparison": {"type1": "feedforward", "type2": "feedforward"},
  "deployment_readiness": {"readiness": 0.92, "strategy": "blue_green"},
  "quantization_analysis": {"compression": "0.0%", "speedup": "1.8x"},
  "regression_test": {"passed": true, "degradation": "-2.5%"}
  // ... plus 25+ additional analysis features
}

Scientific Data Analysis

$ diffai experiment_data_v1.npy experiment_data_v2.npy
  ~ data: shape=[1000, 256], mean=0.1234->0.1456, std=0.9876->0.9654, dtype=float64

MATLAB File Comparison

$ diffai simulation_v1.mat simulation_v2.mat
  ~ results: var=results, shape=[500, 100], mean=2.3456->2.4567, std=1.2345->1.3456, dtype=double
  + new_variable: var=new_variable, shape=[100], dtype=single, elements=100, size=0.39KB

Performance

diffai is optimized for large files and scientific workflows:

• Memory Efficient: Streaming processing for GB+ files
• Fast: Rust implementation with optimized tensor operations
• Scalable: Handles models with millions/billions of parameters
• Cross-Platform: Works on Windows, Linux, and macOS without dependencies

Contributing

We welcome contributions! Please see CONTRIBUTING for guidelines.

Development Setup

git clone https://github.com/kako-jun/diffai.git
cd diffai
cargo build
cargo test

Running Tests

# Run all tests
cargo test

# Run specific test categories
cargo test --test integration
cargo test --test ml_analysis

License

This project is licensed under the MIT License - see the LICENSE file for details.

Related Projects

• diffx - General-purpose structured data diff tool (diffai's sibling project)
• safetensors - Simple, safe way to store and distribute tensors
• PyTorch - Machine learning framework
• NumPy - Fundamental package for scientific computing with Python