GCP Observability for Rust

A lightweight, high-performance Google Cloud Platform observability library for Rust applications. Provides easy-to-use APIs for Cloud Logging, Cloud Monitoring, and Cloud Trace with a background worker for non-blocking operations.

Builder notes

This crate was devoloped by Santiago Amoretti in the context of the devlopment of Genevabm(https://genevabm.com) after facing multiple issue mainly caused by Googles inhability to provide good crates and APIS to interact with its services.

We are used to this idea of using crates to do work that, under the hood, just concist of the most simple way of communication beetween services: A plain HTTP hit. When this happens, i would argue its much better to generate your own structure than to use existing ones. Thats what i did. I figured out the JSON structures, interacted with GCP from the command line and created a single threaded model which is taylor made for microservices.

You can use what i did. You could also modify it to extend its behaviour. Maybe, youll just benefit from the idea of knowing that most of this APIS are high level and intricated for no reason, which makes a good argument for creating your own stuff.

This documentation is AI generated. Not the code, tho. So rest assured.

Features

• 🚀 Non-Blocking: Fire-and-forget API with background processing
• 📝 Cloud Logging: Send structured logs to Google Cloud Logging
• 📊 Cloud Monitoring: Create custom metrics in Google Cloud Monitoring
• 🔍 Cloud Trace: Create distributed traces in Google Cloud Trace
• 🔐 Auto Token Refresh: Automatic handling of gcloud token expiration
• ⚡ High Performance: Single-threaded worker with bounded channel
• 🛡️ Error Resilient: Automatic retry logic for auth failures
• 🎯 Builder Pattern: Fluent API for constructing observability data
• 🪶 Lightweight: Uses gcloud CLI for auth, REST APIs for data

Installation

Add this to your Cargo.toml:

[dependencies]
gcp-observability-rs = "0.1.3"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

Prerequisites

1. Google Cloud Project with APIs enabled:

   • Cloud Logging API
   • Cloud Monitoring API
   • Cloud Trace API

2. Service Account JSON with roles:

   • roles/logging.logWriter
   • roles/monitoring.metricWriter
   • roles/cloudtrace.agent

3. gcloud CLI (automatically installed if missing)

Architecture

The library uses a channel-based architecture for optimal performance:

┌─────────────┐         ┌─────────┐         ┌──────────────┐
│ Your App    │────────▶│ Channel │────────▶│ Worker Thread│
│ (main)      │ queue   │ (1027)  │ process │ (async ops)  │
└─────────────┘         └─────────┘         └──────────────┘
                                                    │
                                                    ▼
                                            ┌──────────────┐
                                            │  GCP APIs    │
                                            └──────────────┘

Quick Start

Fire-and-Forget (Recommended)

use gcp_observability_rs::{ObservabilityClient, LogEntry, MetricData, TraceSpan};
use std::collections::HashMap;
use std::time::{SystemTime, Duration};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize client (performs authentication)
    let client = ObservabilityClient::new(
        "your-project-id".to_string(),
        "/path/to/service-account.json".to_string(),
    ).await?;

    // Fire-and-forget logging (returns immediately)
    client.send_log(LogEntry::new("INFO", "App started"))?;
    
    // With service name
    client.send_log(
        LogEntry::new("ERROR", "DB connection failed")
            .with_service_name("api-server")
    )?;

    // Send metrics with labels
    let mut labels = HashMap::new();
    labels.insert("environment".to_string(), "production".to_string());
    
    client.send_metric(
        MetricData::new(
            "custom.googleapis.com/requests_total",
            42.0,
            "INT64",
            "GAUGE"
        ).with_labels(labels)
    )?;

    // Create distributed traces
    let trace_id = ObservabilityClient::generate_trace_id();
    let span_id = ObservabilityClient::generate_span_id();
    
    client.send_trace(
        TraceSpan::new(
            trace_id,
            span_id,
            "HTTP Request",
            SystemTime::now(),
            Duration::from_millis(150)
        )
    )?;

    Ok(())
}

Async (Wait for Completion)

When you need confirmation that an operation succeeded:

// Wait for operation to complete
client.send_log_async(LogEntry::new("INFO", "Critical log")).await?;
client.send_metric_async(MetricData::new("metric", 1.0, "INT64", "GAUGE")).await?;
client.send_trace_async(TraceSpan::new(...)).await?;

Using Convenience Macros

use gcp_observability_rs::{gcp_info, gcp_warn, gcp_error};

gcp_info!(client, "User {} logged in", user_id)?;
gcp_warn!(client, "High memory usage: {}%", usage)?;
gcp_error!(client, "Failed to process: {}", error)?;

API Reference

ObservabilityClient

Initialization

• new(project_id, service_account_path) → Result<Self, ObservabilityError>
  • Creates and authenticates a new client
  • Starts background worker thread

Fire-and-Forget Methods

• send_log(log_entry: LogEntry) → Result<(), ObservabilityError>
• send_metric(metric_data: MetricData) → Result<(), ObservabilityError>
• send_trace(trace_span: TraceSpan) → Result<(), ObservabilityError>

Async Methods (Wait for Completion)

• send_log_async(log_entry: LogEntry) → Future<Result<(), ObservabilityError>>
• send_metric_async(metric_data: MetricData) → Future<Result<(), ObservabilityError>>
• send_trace_async(trace_span: TraceSpan) → Future<Result<(), ObservabilityError>>

Utility Methods

• generate_trace_id() → String - Generate a 32-character hex trace ID
• generate_span_id() → String - Generate a 16-character hex span ID

Data Structures

LogEntry

LogEntry::new(severity: impl Into<String>, message: impl Into<String>)
    .with_service_name(name: impl Into<String>)

MetricData

MetricData::new(
    metric_type: impl Into<String>,
    value: f64,
    value_type: impl Into<String>,  // "INT64" | "DOUBLE"
    metric_kind: impl Into<String>  // "GAUGE" | "CUMULATIVE"
)
    .with_labels(labels: HashMap<String, String>)

TraceSpan

TraceSpan::new(
    trace_id: impl Into<String>,
    span_id: impl Into<String>,
    display_name: impl Into<String>,
    start_time: SystemTime,
    duration: Duration
)
    .with_parent_span_id(parent_span_id: impl Into<String>)

Convenience Macros

• gcp_info!(client, "message") - Send an INFO log (fire-and-forget)
• gcp_warn!(client, "message") - Send a WARNING log (fire-and-forget)
• gcp_error!(client, "message") - Send an ERROR log (fire-and-forget)
• gcp_log!(client, "LEVEL", "message") - Send a log with custom severity

Error Handling

The library provides comprehensive error handling:

Error Types

• AuthenticationError - Failed to authenticate with gcloud
• ApiError - Google Cloud API request failed
• SetupError - Failed to setup/install gcloud CLI
• Shutdown - Special internal error for worker shutdown

Token Expiration

The library automatically handles token expiration:

1. Detects expired tokens (401/403 HTTP responses)
2. Re-authenticates using your service account
3. Retries the failed operation with a fresh token
4. All happens transparently in the background

Silent Failures

Background operations fail silently to avoid disrupting your application. If you need error feedback, use the async methods:

match client.send_log_async(LogEntry::new("INFO", "Important")).await {
    Ok(()) => println!("Log sent successfully"),
    Err(e) => eprintln!("Failed to send log: {}", e),
}

Performance

Characteristics

• Non-blocking: Fire-and-forget operations return immediately
• Bounded Channel: 1027-item buffer prevents memory overflow
• Single Worker: One background thread prevents API rate limiting
• No Synchronization Overhead: Minimal locking and contention
• Fast Compilation: No heavy protobuf or gRPC dependencies

Benchmarks

On a typical development machine:

• Fire-and-forget operation: < 1µs
• Background processing: ~50-200ms per operation (network dependent)
• Channel capacity: 1027 items before blocking

Features

[dependencies]
gcp-observability-rs = { version = "0.1.0", features = ["logging", "monitoring"] }

Available features:

• logging - Cloud Logging functionality
• monitoring - Cloud Monitoring functionality
• tracing - Cloud Trace functionality
• default - Includes all features

Examples

Run the example:

# Set your project ID and service account path
export GCP_PROJECT_ID="your-project-id"
export GCP_SERVICE_ACCOUNT="/path/to/service-account.json"

cargo run --example basic_usage

Architecture

This library uses a unique approach that balances simplicity with performance:

• Lightweight: No heavy protobuf or gRPC dependencies
• Simple: Uses standard HTTP/REST APIs via curl
• Reliable: Leverages battle-tested gcloud CLI for authentication
• Fast: Minimal overhead and fast compilation times
• Resilient: Automatic token refresh and retry logic

Background Worker

The single-threaded worker model provides natural rate limiting:

• One thread processes all operations sequentially
• Prevents overwhelming GCP APIs
• No complex rate limiting logic needed
• Predictable memory usage

Troubleshooting

Authentication Issues

# Verify gcloud is installed
gcloud version

# Verify service account works
gcloud auth activate-service-account --key-file=/path/to/key.json
gcloud auth list

API Not Enabled

Enable required APIs in your GCP project:

gcloud services enable logging.googleapis.com
gcloud services enable monitoring.googleapis.com
gcloud services enable cloudtrace.googleapis.com

Permission Issues

Ensure your service account has the required roles:

• roles/logging.logWriter
• roles/monitoring.metricWriter
• roles/cloudtrace.agent

License

Licensed under the Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0).

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.