Platform Library

Platform-specific optimizations and utilities for the DiskANN project, providing cross-platform abstractions for file I/O, performance monitoring, and system-specific features.

Overview

The Platform library provides platform-specific optimizations and abstractions that enable DiskANN to achieve high performance across different operating systems and architectures. It handles file I/O, performance monitoring, and system-specific features in a cross-platform manner.

Features

• Cross-platform file I/O - Optimized file operations for different platforms
• Performance monitoring - System-specific performance counters and timing
• Memory management - Platform-optimized memory allocation and management
• I/O completion ports - High-performance asynchronous I/O (Windows)
• File handles - Platform-specific file handle abstractions
• Performance counters - CPU and memory usage monitoring

Quick Start

use platform::{FileHandle, FileIO, Perf};

// Open a file with platform optimizations
let file = FileHandle::open("data.bin", FileIO::Read)?;

// Monitor performance
let perf = Perf::new();
perf.start_measurement();

// Perform file operations
let data = file.read_at(0, 1024)?;

perf.end_measurement();
println!("Operation took: {:?}", perf.elapsed());

File Operations

Basic File I/O

use platform::{FileHandle, FileIO};

// Open file for reading
let file = FileHandle::open("input.bin", FileIO::Read)?;

// Read data
let mut buffer = vec![0u8; 1024];
let bytes_read = file.read(&mut buffer)?;

// Open file for writing
let mut file = FileHandle::create("output.bin", FileIO::Write)?;
file.write(&buffer)?;

Aligned File I/O

For optimal performance with vector data:

use platform::{FileHandle, FileIO};

let file = FileHandle::open("vectors.bin", FileIO::Read)?;

// Read aligned data for SIMD operations
let mut buffer = vec![0f32; 1024];
let bytes_read = file.read_aligned(&mut buffer, 32)?; // 32-byte alignment

Asynchronous I/O

use platform::{FileHandle, FileIO, IOCompletionPort};

// Create I/O completion port
let iocp = IOCompletionPort::new()?;

// Register file with completion port
let file = FileHandle::open("data.bin", FileIO::Read)?;
iocp.associate_file(&file)?;

// Submit async read
let mut buffer = vec![0u8; 1024];
iocp.read_async(&file, &mut buffer, 0)?;

// Wait for completion
let result = iocp.wait_for_completion()?;

Performance Monitoring

CPU Performance

use platform::Perf;

let perf = Perf::new();

// Start monitoring
perf.start_cpu_measurement();

// Your computation here
let result = compute_vectors();

// End monitoring
perf.end_cpu_measurement();

println!("CPU time: {:?}", perf.cpu_time());
println!("Wall time: {:?}", perf.wall_time());

Memory Usage

use platform::Perf;

let perf = Perf::new();

// Get current memory usage
let memory_info = perf.get_memory_info()?;
println!("RSS: {} MB", memory_info.rss_mb);
println!("Peak RSS: {} MB", memory_info.peak_rss_mb);

System Information

use platform::Perf;

let perf = Perf::new();

// Get system information
let sys_info = perf.get_system_info()?;
println!("CPU cores: {}", sys_info.cpu_cores);
println!("Memory: {} GB", sys_info.total_memory_gb);
println!("Page size: {} bytes", sys_info.page_size);

Platform-Specific Features

Windows

#[cfg(target_os = "windows")]
use platform::windows::{IOCompletionPort, FileHandle};

// Windows-specific optimizations
let iocp = IOCompletionPort::new()?;
let file = FileHandle::open_with_options("data.bin", FileIO::Read, &iocp)?;

Linux

#[cfg(target_os = "linux")]
use platform::linux::{FileHandle, Perf};

// Linux-specific features
let perf = Perf::with_pid(std::process::id())?;
let file = FileHandle::open_with_direct_io("data.bin", FileIO::Read)?;

macOS

#[cfg(target_os = "macos")]
use platform::macos::{FileHandle, Perf};

// macOS-specific optimizations
let file = FileHandle::open_with_fcntl("data.bin", FileIO::Read)?;
let perf = Perf::with_mach_port()?;

Memory Management

Aligned Allocation

use platform::memory;

// Allocate aligned memory for SIMD operations
let ptr = memory::aligned_alloc(1024, 32)?; // 32-byte alignment
let slice = unsafe { std::slice::from_raw_parts_mut(ptr, 1024) };

// Use the memory
// ...

// Free aligned memory
memory::aligned_free(ptr);

Memory Mapping

use platform::memory;

// Memory map a file
let mapping = memory::map_file("large_data.bin", FileIO::Read)?;
let data = mapping.as_slice::<f32>();

// Access memory-mapped data
for &value in data {
    // Process value
}

Performance Optimization

File I/O Optimization

use platform::{FileHandle, FileIO, Perf};

let perf = Perf::new();
perf.start_measurement();

// Use platform-optimized file operations
let file = FileHandle::open_with_options("data.bin", FileIO::Read, &Default::default())?;

// Read in large chunks for better performance
let mut buffer = vec![0u8; 64 * 1024]; // 64KB chunks
let mut total_read = 0;

while let Ok(bytes_read) = file.read(&mut buffer) {
    if bytes_read == 0 {
        break;
    }
    total_read += bytes_read;
}

perf.end_measurement();
println!("Read {} bytes in {:?}", total_read, perf.elapsed());

Batch Operations

use platform::{FileHandle, FileIO};

let file = FileHandle::open("vectors.bin", FileIO::Read)?;

// Batch read operations
let mut vectors = Vec::new();
let batch_size = 1000;
let vector_size = 128 * 4; // 128 f32 values

for i in 0..batch_size {
    let mut buffer = vec![0f32; 128];
    file.read_at(i * vector_size, &mut buffer)?;
    vectors.push(buffer);
}

Error Handling

use platform::{FileHandle, FileIO, PlatformError};

fn process_file(filename: &str) -> Result<(), PlatformError> {
    let file = FileHandle::open(filename, FileIO::Read)
        .map_err(|e| PlatformError::FileOpenError {
            filename: filename.to_string(),
            source: e,
        })?;

    // Process file
    Ok(())
}

Benchmarks

Performance comparison across platforms (Intel i7-8700K):

Benchmarks on NVMe SSD

Development

Building

cargo build --release

Testing

cargo test

Platform-Specific Testing

# Test on specific platform
cargo test --target x86_64-unknown-linux-gnu
cargo test --target x86_64-pc-windows-msvc
cargo test --target x86_64-apple-darwin

API Reference

Core Types

• FileHandle - Platform-optimized file handle
• FileIO - File I/O mode enumeration
• Perf - Performance monitoring
• IOCompletionPort - Asynchronous I/O (Windows)
• PlatformError - Platform-specific error types

Main Functions

• FileHandle::open() - Open file with optimizations
• FileHandle::read() - Read data from file
• FileHandle::write() - Write data to file
• Perf::start_measurement() - Start performance monitoring
• Perf::end_measurement() - End performance monitoring

Utility Functions

• memory::aligned_alloc() - Allocate aligned memory
• memory::aligned_free() - Free aligned memory
• memory::map_file() - Memory map file
• get_system_info() - Get system information

Dependencies

• libc - System calls and constants
• winapi - Windows API (Windows only)
• mach - macOS system calls (macOS only)
• errno - Error handling

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

We welcome contributions! Please see the main README for contribution guidelines.