Crates.io | clickhouse-arrow |
lib.rs | clickhouse-arrow |
version | 0.1.6 |
created_at | 2025-07-01 01:27:05.751225+00 |
updated_at | 2025-08-24 18:39:23.318028+00 |
description | ClickHouse Arrow Client for Rust |
homepage | https://github.com/georgeleepatterson/clickhouse-arrow |
repository | https://github.com/georgeleepatterson/clickhouse-arrow |
max_upload_size | |
id | 1732631 |
size | 2,113,673 |
ClickHouse
Native Protocol Rust Client w/ Arrow CompatibilityClickHouse
access in rust over ClickHouse
's native protocol.
Currently supports revision 54479
, DBMS_MIN_REVISION_WITH_VERSIONED_CLUSTER_FUNCTION_PROTOCOL
, the latest revision as of June 2025.
A high-performance, async Rust client for ClickHouse
with native Arrow integration. Designed to be faster and more memory-efficient than existing alternatives.
ClickHouse
Cloud compatibility#[derive(Row)]
macroAll benchmarks run on Apple M2 Pro (12-core) with 16GB RAM using ClickHouse
25.5.2.47 and Rust 1.89.0 with LTO optimizations.
Rows | clickhouse-arrow (none) | clickhouse-arrow (LZ4) | clickhouse-rs (none) | clickhouse-rs (LZ4) |
---|---|---|---|---|
10k | 5.60ms | 5.20ms | 6.16ms | 7.53ms |
100k | 41.97ms | 42.12ms | 47.30ms | 51.66ms |
200k | 97.21ms | 116.81ms | 126.67ms | 134.28ms |
300k | 143.06ms | 160.53ms | 196.39ms | 183.42ms |
400k | 188.21ms | 223.60ms | 255.80ms | 303.35ms |
Key Performance Insights:
pool
feature enables connection reuse for better throughput# Run all benchmarks with LTO optimizations
just bench-lto
# Run specific benchmark
just bench-one insert
# View detailed results
open target/criterion/report/index.html
Benchmarks use realistic workloads with mixed data types (integers, strings, timestamps, arrays) representative of typical ClickHouse
usage patterns. To benchmark with scalar data only, similar to the benchmarks in ch-go
, use the scalar
bench
The crate supports two "modes" of operation:
ArrowFormat
Support allowing interoperability with arrow.
NativeFormat
Uses internal types and custom traits if a dependency on arrow is not required.
CreateOptions
, SchemaConversions
, and Schemasclickhouse-arrow
provides powerful DDL capabilities through CreateOptions
, allowing you to create ClickHouse
tables directly from Arrow schemas:
use clickhouse_arrow::{Client, ArrowFormat, CreateOptions};
use arrow::datatypes::{Schema, Field, DataType};
// Define your Arrow schema
let schema = Schema::new(vec![
Field::new("id", DataType::UInt64, false),
Field::new("name", DataType::Utf8, false),
Field::new("status", DataType::Dictionary(Box::new(DataType::Int8), Box::new(DataType::Utf8)), false),
]);
// Configure table creation
let options = CreateOptions::new("MergeTree")
.with_order_by(&["id".to_string()])
.with_partition_by("toYYYYMM(created_at)")
.with_setting("index_granularity", 8192);
// Create the table
client.create_table(None, "my_table", &schema, &options, None).await?;
SchemaConversions
(type alias for HashMap<String, Type>
) provides fine-grained control over Arrow-to-ClickHouse type mappings. This is especially important for:
LowCardinality(String)
. Use SchemaConversions
to map them to Enum8
or Enum16
instead:use clickhouse_arrow::{Type, CreateOptions};
use std::collections::HashMap;
let schema_conversions = HashMap::from([
// Convert status column from Dictionary to Enum8
("status".to_string(), Type::Enum8(vec![
("active".to_string(), 0),
("inactive".to_string(), 1),
("pending".to_string(), 2),
])),
// Convert category to Enum16 for larger enums
("category".to_string(), Type::Enum16(vec![
("electronics".to_string(), 0),
("clothing".to_string(), 1),
// ... up to 65k values
])),
]);
let options = CreateOptions::new("MergeTree")
.with_order_by(&["id".to_string()])
.with_schema_conversions(schema_conversions);
Date
and Date32
When working with complex Arrow types, use these constants to ensure compatibility:
use clickhouse_arrow::arrow::types::*;
// For List types - inner field is named "item"
let list_field = Field::new("data", DataType::List(
Arc::new(Field::new(LIST_ITEM_FIELD_NAME, DataType::Int32, true))
), true);
// For Struct/Tuple types - fields are named "field_0", "field_1", etc.
let tuple_fields = vec![
Field::new(format!("{}{}", TUPLE_FIELD_NAME_PREFIX, 0), DataType::Int32, false),
Field::new(format!("{}{}", TUPLE_FIELD_NAME_PREFIX, 1), DataType::Utf8, false),
];
// For Map types - uses specific field names
let map_type = DataType::Map(
Arc::new(Field::new(MAP_FIELD_NAME, DataType::Struct(
vec![
Field::new(STRUCT_KEY_FIELD_NAME, DataType::Utf8, false),
Field::new(STRUCT_VALUE_FIELD_NAME, DataType::Int32, true),
].into()
), false)),
false
);
These constants ensure your Arrow schemas align with ClickHouse
's expectations and maintain compatibility with arrow-rs conventions.
The clickhouse_arrow::Settings
type allows configuring ClickHouse
query settings. You can import it directly:
use clickhouse_arrow::Settings;
// or via prelude
use clickhouse_arrow::prelude::*;
Refer to the settings module documentation for details and examples.
There are cases where a round trip may deserialize a different type by schema or array than the schema and array you used to create the table.
will try to maintain an accurate and updated list as they occur. In addition, when possible, I will provide options or other functionality to alter this behavior.
(String|Binary)View
/Large(List|String|Binary)
variations are normalized.ClickHouse
does not make the same distinction between Utf8
, Utf8View
, or LargeUtf8
. All of these are mapped to either Type::Binary
(the default, see above) or Type::String
ClickHouse
, manual modification will be necessary to use these data types.Utf8
-> Binary
Type::String
/DataType::Utf8
will be represented as Binary.strings_as_strings
(default: false
).false
).true
to strip map Type::String
-> DataType::Utf8
. Binary tends to be more efficient to work with in high throughput scenariosArray
sClickHouse
does not allow Nullable(Array(...))
, but insertion with non-null data is allowed by default. To modify this behavior, set array_nullable_error
to true
.array_nullable_error
(default: false
).false
).LowCardinality(Nullable(...))
vs Nullable(LowCardinality(...))
ClickHouse
does not allow nullable low cardinality. The default behavior is to push down the nullability.low_cardinality_nullable_error
(default: false
).false
).Enum8
/Enum16
vs. LowCardinality
Dictionary
types map to LowCardinality
, but ClickHouse
Enum
types may also map to Dictionary
, altering the type on round-trip.enum_i8
and/or enum_i16
for CreateOptions
during schema creation.[!NOTE] For examples of these cases, refer to the tests in the module arrow::types
[!NOTE] The configuration for the options above can be found in options
[!NOTE] For a builder of create options use during schema creation (eg
Engine
,Order By
,Enum8
andEnum16
lookups), refer to CreateOptions