Magma Ingest CLI Tool

A Rust-based CLI tool for ingesting code files into the Magma scanner system. This tool automatically collects source code files from your repository, respects ignore patterns, and packages them for analysis.

Features

• Simple CLI Interface: Easy-to-use command with minimal required parameters
• Git Integration: Automatically extracts commit hash, branch name, and author information
• Smart File Filtering: Uses both default patterns and custom .magmaignore files to exclude unwanted files
• Efficient Packaging: Creates compressed zip files with metadata for optimal transfer
• CI/CD Ready: Designed for seamless integration into continuous integration pipelines

Installation

Via Cargo (Recommended)

cargo install magma-code

Direct Binary Download

Linux x86_64

# Download and install
curl -L -o magma-ingest https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-linux-x86_64
chmod +x magma-ingest
sudo mv magma-ingest /usr/local/bin/

# Verify installation
magma-ingest --version

macOS Intel

# Download and install
curl -L -o magma-ingest https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-macos-x86_64
chmod +x magma-ingest
sudo mv magma-ingest /usr/local/bin/

# Verify installation
magma-ingest --version

macOS ARM (Apple Silicon)

# Download and install
curl -L -o magma-ingest https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-macos-aarch64
chmod +x magma-ingest
sudo mv magma-ingest /usr/local/bin/

# Verify installation
magma-ingest --version

Windows PowerShell (Recommended)

# Download and setup (run PowerShell as Administrator)
cd $env:USERPROFILE\Downloads
Invoke-WebRequest -Uri "https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-windows-x86_64.exe" -OutFile "magma-ingest.exe"

# Create tools directory and move binary
mkdir C:\tools -Force
Move-Item "magma-ingest.exe" "C:\tools\magma-ingest.exe"

# Add to system PATH permanently
$currentPath = [Environment]::GetEnvironmentVariable("PATH", "Machine")
if ($currentPath -notlike "*C:\tools*") {
    [Environment]::SetEnvironmentVariable("PATH", "$currentPath;C:\tools", "Machine")
    Write-Host "✅ Added C:\tools to system PATH"
}

# Add to current session
$env:PATH += ";C:\tools"

# Verify installation
magma-ingest --version

Windows Git Bash

# Download
cd ~/Downloads
curl -L -o magma-ingest.exe https://github.com/bluemagma-compliance/magma-ingest/releases/latest/download/magma-ingest-windows-x86_64.exe

# Setup directory (run in PowerShell as Administrator)
# powershell -Command "mkdir C:\tools -Force; Move-Item '$env:USERPROFILE\Downloads\magma-ingest.exe' 'C:\tools\magma-ingest.exe'; \$env:PATH += ';C:\tools'"

# Add to Git Bash PATH permanently
echo 'export PATH="$PATH:/c/tools"' >> ~/.bashrc
source ~/.bashrc

# Verify installation (restart Git Bash first)
magma-ingest --version

Verify Installation

magma-ingest --version

Troubleshooting PATH Issues

"Command not found" or "magma-ingest is not recognized"

Check if binary exists:

# Linux/macOS
ls -la /usr/local/bin/magma-ingest
ls -la ~/.local/bin/magma-ingest

# Windows
dir C:\tools\magma-ingest.exe

Check current PATH:

# Linux/macOS/Git Bash
echo $PATH

# Windows PowerShell
$env:PATH -split ';'

Manually run binary:

# Linux/macOS
/usr/local/bin/magma-ingest --version
~/.local/bin/magma-ingest --version

# Windows
C:\tools\magma-ingest.exe --version

Restart terminal/shell:

• Close and reopen your terminal
• PATH changes often require a restart to take effect

Permission Issues (Linux/macOS)

# Make sure binary is executable
chmod +x /usr/local/bin/magma-ingest
# or
chmod +x ~/.local/bin/magma-ingest

Windows PATH Not Working

# Check if C:\tools is in PATH
$env:PATH -split ';' | Select-String "tools"

# Manually add for current session
$env:PATH += ";C:\tools"

# Restart PowerShell and try again

Usage

Basic Usage

magma-ingest --org <organization-id> --key <api-key> --repo-name <repository-name>

Advanced Usage

magma-ingest \
  --org my-org-123 \
  --key sk-1234567890abcdef \
  --repo-name my-awesome-project \
  --root-path backend \
  --ignore-file custom-ignore.txt \
  --verbose

Parameters

• --org (required): Organization ID for the Magma scanner
• --key (required): API key for authentication
• --repo-name (required): Repository name for identification
• --root-path (optional): Root path to scan - use '/' for current directory or specify subdirectory
• --ignore-file (optional): Custom ignore file path (defaults to .magmaignore)
• --verbose (optional): Enable verbose logging

Configuration

The API base URL is configured in config.yaml (embedded in the binary) and defaults to http://localhost. The tool automatically:

• Scans the current working directory
• Creates temporary zip files in the system temp directory
• Cleans up temporary files after upload

Ignore Patterns

The tool uses a .magmaignore file to specify which files and directories to exclude from scanning. The syntax is similar to .gitignore files and supports glob patterns.

Default Ignore Patterns

The tool automatically ignores common files and directories:

• Version control directories (.git, .svn, .hg)
• Dependency directories (node_modules, target, build, __pycache__)
• IDE files (.vscode, .idea)
• Binary files (*.exe, *.dll, *.so)
• Archive files (*.zip, *.tar.gz)
• Media files (*.jpg, *.png, *.mp4)
• Log files (*.log)

Custom Ignore File

Create a .magmaignore file in your repository root to specify additional patterns:

# Custom ignores
config/secrets.json
*.env
temp/**

# Large data files
data/**
*.csv

# Documentation
docs/**
*.md

Output

The tool performs the following actions:

1. Collects Files: Gathers all non-ignored files from your repository
2. Creates Package: Generates a zip file with source files and metadata
3. Uploads to API: Sends the package to your Magma ingest endpoint via HTTP POST
4. Cleans Up: Optionally removes the local zip file after successful upload (use --keep-zip to retain)

API Request Format

The tool sends a multipart/form-data POST request to {api-base-url}/ingest/ingest/ with:

• org_id: Organization identifier
• repo_name: Repository name
• commit_hash: Git commit hash
• branch_name: Git branch name
• root_path: Repository root path
• file: Zip file containing source code and metadata

Metadata Format

{
  "version": "1.0",
  "timestamp": "2024-01-15T10:30:00Z",
  "git": {
    "branch": "main",
    "commit_hash": "abc123...",
    "commit_message": "Add new feature",
    "author_name": "John Doe",
    "author_email": "john@example.com"
  },
  "scan_info": {
    "file_count": 150,
    "tool_version": "0.1.0",
    "tool_name": "magma-ingest"
  }
}

CI/CD Integration

GitHub Actions

name: Code Analysis
on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install Rust
        uses: actions-rs/toolchain@v1
        with:
          toolchain: stable

      - name: Install magma-ingest
        run: cargo install --git <repository-url>

      - name: Run Magma Ingest
        run: magma-ingest --org ${{ secrets.MAGMA_ORG_ID }} --key ${{ secrets.MAGMA_API_KEY }}

GitLab CI

stages:
  - scan

magma_scan:
  stage: scan
  image: rust:latest
  script:
    - cargo install --git <repository-url>
    - magma-ingest --org $MAGMA_ORG_ID --key $MAGMA_API_KEY
  variables:
    MAGMA_ORG_ID: $MAGMA_ORG_ID
    MAGMA_API_KEY: $MAGMA_API_KEY

Error Handling

The tool provides clear error messages for common issues:

• No Git Repository: Ensure you're running the tool in a Git repository
• No Files to Ingest: Check your ignore patterns if no files are being collected
• Invalid Patterns: Verify your .magmaignore file syntax
• Permission Issues: Ensure the tool has read access to files and write access to output directory

Logging

Use the --verbose flag to enable detailed logging:

magma-ingest --org my-org --key my-key --verbose

This will show:

• Files being processed
• Ignore patterns being applied
• Git information extraction
• Zip file creation progress

Project Structure

Magma-yeeter/
├── src/
│   ├── main.rs          # Main application entry point
│   ├── cli.rs           # Command-line argument parsing
│   ├── git.rs           # Git repository integration
│   ├── ignore.rs        # File ignore pattern management
│   ├── zipper.rs        # File compression and packaging
│   └── error.rs         # Error handling and types
├── examples/
│   ├── github-actions.yml  # GitHub Actions workflow example
│   └── gitlab-ci.yml       # GitLab CI configuration example
├── .magmaignore         # Default ignore patterns
├── Cargo.toml           # Rust project configuration
└── README.md            # This file

Development

Building

# Debug build
cargo build

# Release build
cargo build --release

Testing

cargo test

Running

# Development
cargo run -- --org test-org --key test-key

# Release binary
./target/release/magma-ingest --org test-org --key test-key

License

MIT License - see LICENSE file for details.