# Rust Dual Interface for q/kdb+ As Rust is becoming a popular programming language for its performance and type safety, the desire to use it with still a maniac time-series database kdb+ is brewing. The aspiration is understandable since we know kdb+ is fast and its interface or a shared library should be fast as well. This interface was created to satisfy such a natural demand, furthermore, in a manner users do not feel any pain to use. The notrious esoteric function names of the q/kdb+ C API is not an interest of Rust developers. *"Give us a **Rust** interface!!"* Here is your choice. This interface provides two features: - IPC interface (Rust client of q/kdb+ process) - API (build a shared library for q/kdb+) You can find the detail descriptions of each feature below. ## Rust IPC Interface of q/kdb+ As Rust was conceived to address type unsafety of C/C++, replacing C/C++ with Rust can happen if possible. This interface is purposed to be used as a Rust client of q/kdb+ process that sends a query and receives its response. Query to kdb+ is supported in two ways: - text query - functional query which is represented by a compound list of kdb+ ([See detail of IPC](https://code.kx.com/q4m3/11_IO/#116-interprocess-communication)). Compression/decompression of messages is also implemented following [kdb+ implementation](https://code.kx.com/q/basics/ipc/#compression). As for connect method, usually client interfaces of q/kdb+ do not provide a listener due to its protocol. However, sometimes Rust process is connecting to an upstream and q/kdb+ starts afterward or is restarted more frequently. Then providing a listener method is a natural direction and it was achieved here. Following ways are supported to connect to kdb+: - TCP - TLS - Unix domain socket Furthermore, in order to improve inter-operatability some casting, getter and setter methods are provided. ### Environmental Variables This crate uses q-native or crate-specific environmental variables. - `KDBPLUS_ACCOUNT_FILE`: A file path to a credential file which an acceptor loads in order to manage access from a q client. This file contains a user name and SHA-1 hashed password in each line which are delimited by `':'` without any space. For example, a file containing two credentials `"mattew:oracle"` and `"reluctant:slowday"` looks like this: mattew:431364b6450fc47ccdbf6a2205dfdb1baeb79412 reluctant:d03f5cc1cdb11a77410ee34e26ca1102e67a893c The hashed password can be generated with q using a function `.Q.sha1`: q).Q.sha1 "slowday" 0xd03f5cc1cdb11a77410ee34e26ca1102e67a893c - `KDBPLUS_TLS_KEY_FILE` and `KDBPLUS_TLS_KEY_FILE_SECRET`: The pkcs12 file and its password which TLS acceptor uses. - `QUDSPATH` (optional): q-native environmental variable to define an astract namespace. This environmental variable is used by UDS acceptor too. The abstract nameapace will be `@${QUDSPATH}/kx.[server process port]` if this environmental variable is defined; otherwise it will be `@/tmp/kx.[server process port]`. *Notes:* - Messages will be sent with OS native endian. - When using this crate for a TLS client you need to set two environmental variables `KX_SSL_CERT_FILE` and `KX_SSL_KEY_FILE` on q side to make q/kdb+ to work as a TLS server. For details, see [the KX website](https://code.kx.com/q/kb/ssl/). ### Type Mapping All types are expressed as `K` struct which is quite similar to the `K` struct of `api` module but its structure is optimized for IPC usage and for the convenience to interact with. The table below shows the input types of each q type which is used to construct `K` object. Note that the input type can be different from the inner type. For example, timestamp has an input type of `chrono::DateTime` but the inner type is `i64` denoting an elapsed time in nanoseconds since `2000.01.01D00:00:00`. | q | Rust | |------------------|---------------------------------------------------| | `bool` | `bool` | | `GUID` | `[u8; 16]` | | `byte` | `u8` | | `short` | `i16` | | `int` | `i32` | | `long` | `i64` | | `real` | `f32` | | `float` | `f64` | | `char` | `char` | | `symbol` | `String` | | `timestamp` | `chrono::DateTime` | | `month` | `chrono::NaiveDate` | | `date` | `chrono::NaiveDate` | | `datetime` | `chrono::DateTime` | | `timespan` | `chrono::Duration` | | `minute` | `chrono::Duration` | | `second` | `chrono::Duration` | | `time` | `chrono::Duration` | | `list` | `Vec` (`Item` is a corrsponding type above) | | `compound list` | `Vec` | | `table` | `Vec` | | `dictionary` | `Vec` | | `null` | `()` | ### Examples #### Client ```rust use kdbplus::ipc::*; #[tokio::main(flavor = "multi_thread", worker_threads = 2)] async fn main() -> Result<()> { // Connect to qprocess running on localhost:5000 via UDS let mut socket = QStream::connect(ConnectionMethod::UDS, "", 5000_u16, "ideal:person").await?; println!("Connection type: {}", socket.get_connection_type()); // Set remote function with asynchronous message socket.send_async_message(&"collatz:{[n] seq:enlist n; while[not n = 1; seq,: n:$[n mod 2; 1 + 3 * n; `long$n % 2]]; seq}").await?; // Send a query synchronously let mut result = socket.send_sync_message(&"collatz[12]").await?; println!("collatz[12]: {}", result); result = socket.send_sync_message(&"collatz[`a]").await?; println!("collatz[`a]: {}", result); // Send a functional query. let mut message = K::new_compound_list(vec![K::new_symbol(String::from("collatz")), K::new_long(100)]); result = socket.send_sync_message(&message).await?; println!("collatz[100]: {}", result); // Modify query to (`collatz; 20) message.pop().unwrap(); message.push(&K::new_long(20)).unwrap(); result=socket.send_sync_message(&message).await?; println!("collatz[20]: {}", result); // Send a functional asynchronous query. message = K::new_compound_list(vec![K::new_string(String::from("show"), qattribute::NONE), K::new_symbol(String::from("goodbye"))]); socket.send_async_message(&message).await?; socket.shutdown().await?; Ok(()) } ``` #### Listener ```rust use kdbplus::ipc::*; #[tokio::main] async fn main() -> Result<()> { // Start listenening over TCP at the port 7000 with authentication enabled. let mut socket_tcp = QStream::accept(ConnectionMethod::TCP, "127.0.0.1", 7000).await?; // Send a query with the socket. let greeting = socket_tcp.send_sync_message(&"string `Hello").await?; println!("Greeting: {}", greeting); socket_tcp.shutdown().await?; Ok(()) } ``` Then q client can connect to this acceptor with the acceptor's host, port and the credential configured in `KDBPLUS_ACCOUNT_FILE`: ```q q)h:hopen `::7000:reluctant:slowday ``` ### Installation Use `kdbplus` as a library name in `Cargo.toml` with `"ipc"` feature. ```toml [dependencies] kdbplus={version="^0.3", features=["ipc"]} ``` ## Rust Wrapper of q/kdb+ C API Programming language q (kdb+ is a database written in q) is providing only C API but sometimes an external library provides Rust interface but not C/C++ interface. From the fame of its performance, Rust still should be feasible to build a shared library for kdb+. This library is provided to address such a natural demand (desire, if you will). Since there is no way for everyone but creating a wrapper like this to write a shared library for kdb+, it probably make sense for someone to provide the wrapper, and it was done here. In order to avoid writing too large `unsafe` block which leads to poor optimization, most of native C API functions are provided with a wrapper funtion with a bit of ergonomic safety and with intuitive implementation as a trait method. The only exceptions are `knk` and `k` which are using elipsis (`...`) as its argument. These functions are provided under `native` namespace with the other C API functions. **Note:** This library is purposed to be used to build a sared library; therefore some unrelated functions are removed. For example, connection functions to kdb+ like `khpu` are not included. ### Installation Use `kdbplus` as a library name in `Cargo.toml` with `"api"` feature. ```toml [dependencies] kdbplus={version="^0.3", features=["api"]} ``` ### Examples The examples of using C API wrapper are included in `api_examples` folder. The examples are mirroring the examples in the document of `kdbplus::api` module and the functions are also used for simple tests of the library. The test is conducted in the `test.q` under `tests/` by loading the functions defined in a shared library built from the examples. Here are some examples: #### C API Style ```rust use kdbplus::qtype; use kdbplus::api::*; use kdbplus::api::native::*; #[no_mangle] pub extern "C" fn create_symbol_list(_: K) -> K { unsafe{ let mut list=ktn(qtype::SYMBOL_LIST as i32, 0); js(&mut list, ss(str_to_S!("Abraham"))); js(&mut list, ss(str_to_S!("Isaac"))); js(&mut list, ss(str_to_S!("Jacob"))); js(&mut list, sn(str_to_S!("Josephine"), 6)); list } } #[no_mangle] pub extern "C" fn catchy(func: K, args: K) -> K { unsafe{ let result=ee(dot(func, args)); if (*result).qtype == qtype::ERROR{ println!("error: {}", S_to_str((*result).value.symbol)); // Decrement reference count of the error object r0(result); KNULL } else { result } } } #[no_mangle] pub extern "C" fn dictionary_list_to_table() -> K { unsafe{ let dicts = knk(3); let dicts_slice = dicts.as_mut_slice::(); for i in 0..3 { let keys = ktn(qtype::SYMBOL_LIST as i32, 2); let keys_slice = keys.as_mut_slice::(); keys_slice[0] = ss(str_to_S!("a")); keys_slice[1] = ss(str_to_S!("b")); let values = ktn(qtype::INT_LIST as i32, 2); values.as_mut_slice::()[0..2].copy_from_slice(&[i*10, i*100]); dicts_slice[i as usize] = xD(keys, values); } // Format list of dictionary as a table. // ([] a: 0 10 20i; b: 0 100 200i) k(0, str_to_S!("{[dicts] -1 _ dicts, (::)}"), dicts, KNULL) } } ``` q can use these functions like this: ```q q)summon:`libc_api_examples 2: (`create_symbol_list; 1) q)summon[] `Abraham`Isaac`Jacob`Joseph q)`Abraham`Isaac`Jacob`Joseph ~ summon[] q)catchy: `libc_api_examples 2: (`catchy; 2); q)catchy[$; ("J"; "42")] 42 q)catchy[+; (1; `a)] error: type q)behold: `libc_api_examples 2: (`dictionary_list_to_table; 1); q)behold[] a b ------ 0 0 10 100 20 200 ``` #### Rust Style The examples below are written without `unsafe` code. You can see how comfortably breathing are the wrapped functions in the code. ```rust use kdbplus::qtype; use kdbplus::api::*; use kdbplus::api::native::*; #[no_mangle] pub extern "C" fn create_symbol_list2(_: K) -> K { let mut list = new_list(qtype::SYMBOL_LIST, 0); list.push_symbol("Abraham").unwrap(); list.push_symbol("Isaac").unwrap(); list.push_symbol("Jacob").unwrap(); list.push_symbol_n("Josephine", 6).unwrap(); list } #[no_mangle] fn no_panick(func: K, args: K) -> K { let result = error_to_string(apply(func, args)); if let Ok(error) = result.get_error_string() { println!("FYI: {}", error); // Decrement reference count of the error object which is no longer used. decrement_reference_count(result); KNULL } else{ println!("success!"); result } } #[no_mangle] pub extern "C" fn create_table2(_: K) -> K { // Build keys let keys = new_list(qtype::SYMBOL_LIST, 2); let keys_slice = keys.as_mut_slice::(); keys_slice[0] = enumerate(str_to_S!("time")); keys_slice[1] = enumerate_n(str_to_S!("temperature_and_humidity"), 11); // Build values let values = new_list(qtype::COMPOUND_LIST, 2); let time = new_list(qtype::TIMESTAMP_LIST, 3); // 2003.10.10D02:24:19.167018272 2006.05.24D06:16:49.419710368 2008.08.12D23:12:24.018691392 time.as_mut_slice::().copy_from_slice(&[119067859167018272_i64, 201766609419710368, 271897944018691392]); let temperature = new_list(qtype::FLOAT_LIST, 3); temperature.as_mut_slice::().copy_from_slice(&[22.1_f64, 24.7, 30.5]); values.as_mut_slice::().copy_from_slice(&[time, temperature]); flip(new_dictionary(keys, values)) } ``` And q code is here: ```q q)summon:`libc_api_examples 2: (`create_symbol_list2; 1) q)summon[] `Abraham`Isaac`Jacob`Joseph q)chill: `libc_api_examples 2: (`no_panick; 2); q)chill[$; ("J"; "42")] success! 42 q)chill[+; (1; `a)] FYI: type q)climate_change: libc_api_examples 2: (`create_table2; 1); q)climate_change[] time temperature ----------------------------------------- 2003.10.10D02:24:19.167018272 22.1 2006.05.24D06:16:49.419710368 24.7 2008.08.12D23:12:24.018691392 30.5 ``` #### Test Test is conducted in two ways: 1. Using cargo 2. Running a q test script ##### 1. Using Cargo Before starting the test, you need to start a q process on the port 5000: ```bash kdbplus]$ q -p 5000 q) ``` Then fire the cargo test: ```bash kdbplus]$ cargo test ``` **Note:** Currently 20 tests fails for `api` examples in document. This is because the examples do not have `main` function by nature of `api` but still use `#[macro_use]`. ##### 2. Running a q Test Script Tests are conducted with `tests/test.q` by loading the example functions built in `api_examples`. ```bash kdbplus]$ cargo build kdbplus]$ cp target/debug/libapi_examples.so tests/ kdbplus]$ cd tests tests]$ q test.q Initialized something, probably it is your mindset. bool: true bool: false byte: 0xc4 GUID: 8c6b-8b-64-68-156084 short: 10 int: 42 int: 122 int: 7336 int: 723 int: 14240 int: 2056636 long: -109210 long: 43200123456789 long: -325389000000021 long: 0 real: 193810.31 float: -37017.09330000 float: 742.41927468 char: "k" symbol: `locust string: "gnat" string: "grasshopper" error: type What do you see, son of man?: a basket of summer fruit What do you see, son of man?: boiling pot, facing away from the north symbol: `rust success! FYI: type this is KNULL Planet { name: "earth", population: 7500000000, water: true } Planet { name: "earth", population: 7500000000, water: true } おいしい！ おいしい！ おいしい！ おいしい！ おいしい！ おいしい！ おいしい！ おいしい！ おいしい！ おいしい！ "Collect the clutter of apples!" test result: ok. 147 passed; 0 failed q)What are the three largest elements?: `belief`love`hope ``` ### Projects Using This Library - [qrpc](https://github.com/diamondrod/qrpc) (gRPC client) - [q_comtrade](https://github.com/diamondrod/q_comtrade) (COMTRADE file parser) ## Document The document of this crate itself is on the [crates.io page](https://docs.rs/kdbplus/). For details of C API itself, check the documents of KX website. - [Refernce](https://code.kx.com/q/interfaces/capiref/) - [Memory management](https://code.kx.com/q/interfaces/c-client-for-q/#managing-memory-and-reference-counting) ## License Crate `kdbplus` is licensed under Apache2.0. See [license in the crate](kdbplus/LICENSE).