loqa-voice-dsp

Shared DSP library for voice analysis, providing core digital signal processing functionality for both Loqa backend and VoiceFind mobile app.

Features

• Pitch Detection: YIN and pYIN algorithms for fundamental frequency (F0) estimation
  • NEW in v0.3.0: Stateful VoiceAnalyzer API for streaming analysis
  • NEW in v0.3.0: pYIN (probabilistic YIN) for better noisy/breathy voice detection
  • NEW in v0.3.0: Configurable algorithm selection (Auto/pYIN/YIN/Autocorr)
  • NEW in v0.4.0: Custom pYIN implementation optimized for voice (no external dependencies)
• Formant Extraction: Linear Predictive Coding (LPC) for formant analysis
• FFT Utilities: Fast Fourier Transform for spectral analysis
• Spectral Analysis: Spectral centroid, tilt, and rolloff calculations
• HNR (Harmonics-to-Noise Ratio): Breathiness measurement using Boersma's autocorrelation method
• H1-H2 Amplitude Difference: Vocal weight analysis (lighter vs fuller voice quality)

Installation

iOS (CocoaPods)

Add to your Podfile:

pod 'LoqaVoiceDSP', '~> 0.3.0'

Then run:

pod install

iOS (Swift Package Manager)

In Xcode:

1. File → Add Packages
2. Enter repository URL: https://github.com/loqalabs/loqa
3. Select version: 0.3.0 or later

Or add to Package.swift:

dependencies: [
    .package(url: "https://github.com/loqalabs/loqa", from: "0.3.0")
]

Rust (Cargo)

Add to your Cargo.toml:

[dependencies]
loqa-voice-dsp = "0.3.0"

Usage

Buffer Size Recommendations

Pitch detection algorithms analyze buffers in frames. For best results:

• Recommended: 2048-4096 samples (~46-93ms @ 44100 Hz)
• Minimum: 1024 samples for real-time applications
• Maximum: 4096 samples per frame (accuracy degrades beyond this)

Why this matters:

• Large buffers (>4096) may contain pitch variations, voiced/unvoiced transitions, or multiple syllables
• The algorithm requires buffer size ≥ 2× longest period for the lowest expected frequency:
  • For 80 Hz (male voice): minimum ~1103 samples
  • For 400 Hz (female voice): minimum ~221 samples

Frame-based analysis for long audio (v0.3.0+):

For buffers larger than 4096 samples, use the new VoiceAnalyzer API:

use loqa_voice_dsp::{VoiceAnalyzer, AnalysisConfig};

let config = AnalysisConfig::default()
    .with_frame_size(2048)
    .with_hop_size(1024);  // 50% overlap

let mut analyzer = VoiceAnalyzer::new(config)?;
let results = analyzer.process_stream(&long_audio_buffer);

Legacy approach (v0.2.x):

fn analyze_long_buffer(buffer: &[f32], sample_rate: u32) -> Vec<PitchResult> {
    const FRAME_SIZE: usize = 2048;
    const HOP_SIZE: usize = 1024;  // 50% overlap

    let mut results = Vec::new();
    for i in (0..buffer.len().saturating_sub(FRAME_SIZE)).step_by(HOP_SIZE) {
        let frame = &buffer[i..i + FRAME_SIZE];
        if let Ok(pitch) = detect_pitch(frame, sample_rate, 80.0, 400.0) {
            results.push(pitch);
        }
    }
    results
}

Rust (Loqa Backend)

New in v0.3.0 - Stateful API:

use loqa_voice_dsp::{VoiceAnalyzer, AnalysisConfig, PitchAlgorithm};

let audio_samples: Vec<f32> = /* your audio data */;

// Create analyzer with pYIN algorithm
let config = AnalysisConfig::default()
    .with_sample_rate(16000)
    .with_frame_size(2048)
    .with_algorithm(PitchAlgorithm::PYIN);

let mut analyzer = VoiceAnalyzer::new(config)?;

// Process single frame
let pitch = analyzer.process_frame(&audio_samples)?;
println!("Frequency: {} Hz", pitch.frequency);
println!("Confidence: {}", pitch.confidence);
println!("Voiced Probability: {}", pitch.voiced_probability);

// Or process a stream
let results = analyzer.process_stream(&long_audio_buffer);
for (i, pitch) in results.iter().enumerate() {
    println!("Frame {}: {} Hz (conf: {})", i, pitch.frequency, pitch.confidence);
}

Legacy API (still supported):

use loqa_voice_dsp::{detect_pitch, extract_formants, compute_fft, calculate_hnr, calculate_h1h2};

let audio_samples: Vec<f32> = /* your audio data */;
let sample_rate = 16000;

// Pitch detection (single-shot)
let pitch = detect_pitch(&audio_samples, sample_rate, 80.0, 400.0)?;
println!("Frequency: {} Hz, Confidence: {}", pitch.frequency, pitch.confidence);

// Formant extraction
let formants = extract_formants(&audio_samples, sample_rate, 14)?;
println!("F1: {} Hz, F2: {} Hz", formants.f1, formants.f2);

// HNR (breathiness)
let hnr = calculate_hnr(&audio_samples, sample_rate, 75.0, 500.0)?;
println!("HNR: {} dB, Voiced: {}", hnr.hnr, hnr.is_voiced);

// H1-H2 (vocal weight)
let h1h2 = calculate_h1h2(&audio_samples, sample_rate, Some(pitch.frequency))?;
println!("H1-H2: {} dB", h1h2.h1h2);

// FFT
let fft_result = compute_fft(&audio_samples, sample_rate, 2048)?;

iOS (Swift via FFI)

New in v0.3.0 - Stateful Analyzer:

// Create analyzer configuration
var config = loqa_analysis_config_default()
config.algorithm = 1  // 0=Auto, 1=PYIN, 2=YIN, 3=Autocorr
config.frame_size = 2048
config.sample_rate = 16000

// Create analyzer
let analyzer = loqa_voice_analyzer_new(config)
defer { loqa_voice_analyzer_free(analyzer) }  // Always free

// Process single frame
let pitchResult = samples.withUnsafeBufferPointer { buffer in
    loqa_voice_analyzer_process_frame(
        analyzer,
        buffer.baseAddress!,
        buffer.count
    )
}
if pitchResult.success {
    print("Pitch: \(pitchResult.frequency)Hz")
    print("Confidence: \(pitchResult.confidence)")
    print("Voiced Probability: \(pitchResult.voiced_probability)")
}

// Or process stream
var results = [PitchResultFFI](repeating: PitchResultFFI(), count: 100)
let count = samples.withUnsafeBufferPointer { buffer in
    results.withUnsafeMutableBufferPointer { resultsBuffer in
        loqa_voice_analyzer_process_stream(
            analyzer,
            buffer.baseAddress!,
            buffer.count,
            resultsBuffer.baseAddress!,
            100
        )
    }
}
print("Got \(count) pitch results")

Legacy API (still supported):

// Call C-compatible FFI functions
let samples: [Float] = /* your audio data */

// Pitch detection
let pitchResult = samples.withUnsafeBufferPointer { buffer in
    loqa_detect_pitch(
        buffer.baseAddress!,
        buffer.count,
        16000,  // sample rate
        80.0,   // min freq
        400.0   // max freq
    )
}
if pitchResult.success {
    print("Pitch: \(pitchResult.frequency)Hz, Confidence: \(pitchResult.confidence)")
}

// HNR (breathiness)
let hnrResult = samples.withUnsafeBufferPointer { buffer in
    loqa_calculate_hnr(
        buffer.baseAddress!,
        buffer.count,
        16000,  // sample rate
        75.0,   // min freq
        500.0   // max freq
    )
}
if hnrResult.success {
    print("HNR: \(hnrResult.hnr) dB, Voiced: \(hnrResult.is_voiced)")
}

// H1-H2 (vocal weight) - pass 0.0 for f0 to auto-detect
let h1h2Result = samples.withUnsafeBufferPointer { buffer in
    loqa_calculate_h1h2(
        buffer.baseAddress!,
        buffer.count,
        16000,  // sample rate
        pitchResult.frequency  // use detected pitch, or 0.0 to auto-detect
    )
}
if h1h2Result.success {
    print("H1-H2: \(h1h2Result.h1h2) dB")
}

Android (Java via JNI)

// Build with --features android-jni
import com.voicefind.VoiceFindDSP;

float[] audioSamples = /* your audio data */;
VoiceFindDSP.PitchResult pitch = VoiceFindDSP.detectPitch(
    audioSamples,
    16000,  // sample rate
    80.0f,  // min freq
    400.0f  // max freq
);

System.out.println("Frequency: " + pitch.frequency + " Hz");

Note: Android JNI requires building with --features android-jni

FFI Safety & Parameter Validation

FFI Safety Requirements

Critical: All FFI structs use #[repr(C)] to ensure C-compatible memory layout. Failure to maintain this can cause alignment issues and incorrect values (see historical issues #1, #2, #3).

Memory safety:

• All FFI functions validate null pointers before dereferencing
• FFT results (loqa_compute_fft) allocate memory that must be freed using loqa_free_fft_result
• Never free FFT results more than once
• Never use FFT result pointers after calling loqa_free_fft_result

Swift/iOS example with proper cleanup:

let fftResult = loqa_compute_fft(buffer, count, sampleRate, fftSize)
defer { loqa_free_fft_result(&fftResult) }  // Always free

if fftResult.success {
    let spectral = loqa_analyze_spectrum(&fftResult)
    // Use spectral features...
}

Parameter Validation Ranges

Important: All validation happens in the Rust core. Higher-level layers (Swift/TypeScript) should trust Rust validation rather than implementing their own rules.

Pitch Detection (loqa_detect_pitch)

Formant Extraction (loqa_extract_formants)

Historical Note: Issue loqa-expo-dsp#8 - TypeScript calculated lpc_order = sample_rate / 1000 + 2 which gave 46 for 44.1kHz. Swift layer rejected this as out of range, causing all calls to fail. Solution: Use fixed range 8-24 for all sample rates.

FFT (loqa_compute_fft)

HNR Calculation (loqa_calculate_hnr)

H1-H2 Calculation (loqa_calculate_h1h2)

Auto-detect F0: Pass 0.0 (or any negative value) for f0 parameter to automatically detect pitch before calculating H1-H2.

Common FFI Pitfalls (Lessons Learned)

1. Struct Alignment Issues (Fixed in v0.2.1)

• Problem: Missing #[repr(C)] caused field misalignment
• Symptom: Correct frequency in Rust, wrong value in Swift/Java
• Solution: All FFI structs now have #[repr(C)] - verified by CI tests

2. Parameter Validation Mismatches (Fixed in v0.2.2)

• Problem: TypeScript/Swift layers calculated different validation rules than Rust
• Symptom: Valid parameters rejected, invalid parameters accepted
• Solution: Single source of truth - Rust core validates, higher layers trust it

3. Buffer Size Confusion (Documented in v0.2.2)

• Problem: Users passing large buffers (16384 samples) got false negatives
• Symptom: Pitch detection failed despite valid audio
• Solution: Documentation + optional validation warnings (see Issue #5)

4. Memory Leaks with FFT (Prevented by design)

• Problem: Forgetting to free FFT results leaks ~16-32KB per call
• Symptom: Gradual memory growth in long-running apps
• Solution: Always use Swift defer or RAII patterns to ensure cleanup

Implementation Status

• Crate structure created
• Pitch detection (YIN + autocorrelation)
• Formant extraction (LPC-based)
• FFT utilities
• Spectral analysis (centroid, tilt, rolloff)
• HNR calculation (Boersma's autocorrelation method)
• H1-H2 amplitude difference (vocal weight)
• iOS FFI layer (C exports for all functions)
• Android JNI layer (with jni feature)
• Unit tests (68 passing)
• FFI integration tests (9 passing)
• SVD consistency tests (5 passing)
• Synthetic consistency tests (4 passing)
• Documentation tests (8 passing)
• Benchmarks harness
• Performance benchmarks (validated)

Performance Benchmarks

Validated Performance (2025-11-07) - All targets exceeded ✅

Note: Benchmarks run on Apple M-series silicon. All latency targets easily met with significant performance headroom for real-time voice processing.

Algorithm Details

Custom pYIN Implementation (v0.4.0+)

Starting in v0.4.0, we use a custom pYIN implementation optimized for voice analysis, removing the external pyin crate dependency.

What is pYIN?

pYIN (Mauch & Dixon, 2014) extends the YIN pitch detection algorithm to produce probabilistic pitch estimates, making it more robust for noisy or breathy voice signals.

Key Differences from Standard YIN:

• YIN: Returns single pitch estimate per frame
• pYIN: Returns multiple pitch candidates with probabilities, then uses Hidden Markov Model (HMM) to find the smoothest pitch track

Our Voice-Optimized Implementation:

1. Two-Stage Process:

   • Stage 1: Generate multiple pitch candidates using Beta distribution β(2,18) for thresholds
   • Stage 2: Use Viterbi algorithm on HMM to find optimal pitch track

2. Voice-Specific Optimizations:

   • Narrower frequency range (80-400 Hz vs. general audio 50-2000 Hz)
   • Tighter HMM transition constraints (voice pitch changes slowly)
   • Voice-tuned Beta distribution concentrating probability near threshold=0.1

3. Benefits:

   • No external dependencies - fully integrated implementation
   • Better handling of breathy voice - multiple candidates provide robustness
   • Smoother pitch tracks - HMM enforces temporal consistency
   • Voiced probability per frame - soft voiced/unvoiced decisions (not just binary)
   • Smaller binary size - only includes what we need for voice

4. Performance:

   • ~65-67 µs per 100ms frame (16kHz sample rate)
   • ~1.5-2x overhead vs. standard YIN (acceptable tradeoff for improved accuracy)
   • Still meets 160-500x real-time performance target

References:

• Mauch & Dixon (2014) - pYIN paper
• de Cheveigné & Kawahara (2002) - YIN algorithm

Acoustic Measures Reference

HNR (Harmonics-to-Noise Ratio)

Measures the ratio of harmonic (periodic) to noise (aperiodic) energy in voice - the primary acoustic indicator of breathiness.

H1-H2 (First/Second Harmonic Difference)

Measures the amplitude difference between the fundamental and second harmonic - indicates vocal weight.

Test Data

Saarbrücken Voice Database

This library uses samples from the Saarbrücken Voice Database for consistency validation testing.

License: CC BY 4.0

Attribution: Pützer, M. & Barry, W.J., Former Institute of Phonetics, Saarland University. Available at Zenodo.

The SVD provides lab-quality voice recordings including:

• Sustained vowels (/a:/, /i:/, /u:/) at low, normal, and high pitch
• 851 healthy control speakers
• 1002 speakers with documented voice pathologies
• 50 kHz sample rate, controlled recording conditions

Setting Up Test Data

# 1. Download SVD from Zenodo (CC BY 4.0 license)
#    https://zenodo.org/records/16874898

# 2. Install conversion dependencies
pip install scipy numpy

# 3. Convert SVD files to test format
python scripts/download_svd.py /path/to/extracted/svd

Test Sample Requirements

For comprehensive validation, the library needs test samples with these characteristics:

Development

# Build
cargo build --release

# Test
cargo test

# Benchmark
cargo bench

# Documentation
cargo doc --open

License

MIT