wp-data-fmt

wp-data-fmt is the formatting layer that powers WarpParse style connectors. It turns wp_model_core::model::DataRecord instances into text so they can be sent to logs, key/value stores, SQL databases, or snapshot tests. The crate provides concrete formatters (JSON, CSV, KV, raw, ProtoText, SQL) as well as a FormatType enum that makes it easy to pick a formatter at runtime from a TextFmt definition.

Highlights

• Works directly with the strongly typed wp_model_core data model.
• Consistent escaping rules across JSON, CSV, KV, ProtoText, SQL, and raw text outputs.
• Helpers such as SqlInsert for generating INSERT, batch inserts, table schemas, and UPSERT statements from existing records.
• Snapshot-friendly output that is covered by integration tests for each supported format.

Installation

Add the crate through Cargo:

[dependencies]
wp-data-fmt = "0.1"
wp-model-core = "0.7"

Quick start

use chrono::NaiveDateTime;
use std::net::{IpAddr, Ipv4Addr};
use wp_data_fmt::{DataFormat, FormatType};
use wp_model_core::model::{fmt_def::TextFmt, DataField, DataRecord};

let record = DataRecord {
    items: vec![
        DataField::from_ip("ip", IpAddr::V4(Ipv4Addr::new(192, 168, 1, 2))),
        DataField::from_time(
            "time",
            NaiveDateTime::parse_from_str("2019-08-06 12:12:19", "%Y-%m-%d %H:%M:%S").unwrap(),
        ),
        DataField::from_chars("http/request", "GET /nginx-logo.png HTTP/1.1"),
        DataField::from_digit("http/status", 200),
    ],
};

let fmt = FormatType::from(&TextFmt::Json);
let line = fmt.format_record(&record);
assert_eq!(
    line,
    r#"{"ip":"192.168.1.2","time":"2019-08-06 12:12:19","http/request":"GET /nginx-logo.png HTTP/1.1","http/status":200}"#
);

Because every formatter implements the DataFormat trait, the same snippet can be reused for CSV (FormatType::from(&TextFmt::Csv)), KV, raw, ProtoText, or custom SQL encodings.

SQL helpers

When you already have a DataRecord named record (and possibly a Vec<DataRecord> called records) from your ingestion pipeline, SqlInsert implements DataFormat and adds convenience methods:

use wp_data_fmt::SqlInsert;

let sql = SqlInsert::new_with_json("nginx_logs").format_record(&record);
// INSERT INTO "nginx_logs" ("ip", "time", ...) VALUES (...);

let batch_sql = SqlInsert::new_with_json("nginx_logs").format_batch(&records);
let schema = SqlInsert::new_with_json("nginx_logs").generate_create_table(&records);
let upsert = SqlInsert::new_with_json("nginx_logs").format_upsert(&record, &["ip", "time"]);

Development

cargo fmt
cargo clippy
cargo test

This repository follows the Elastic License 2.0 (see Cargo.toml for details).