Datafet

Few utility functions for our mostly data engineering applications.

EID

Few functions to generate consistent hash based [external][1] ids.

pub fn get_external_id_str(name: &str, prefix: &str) -> String {
    let mut spooky_hasher = SpookyHasher::new(1337, 7331);
    spooky_hasher.write(name.as_bytes());
    let hash_uint64 = spooky_hasher.finish();
    let hash_hex = format!("{:x}", hash_uint64);
    let base32 = base32::encode(Alphabet::RFC4648 { padding: false }, hash_hex.as_bytes());
    format!("{}-{}", prefix, base32.to_ascii_lowercase())
}

There are few kinds we have in most of our applications:

pub fn get_db_eid(database_name: &str) -> String {
    get_external_id_str(database_name, "db")
}

pub fn get_table_eid(table_name: &str) -> String {
    get_external_id_str(table_name, "table")
}

pub fn get_job_eid(job_name: &str) -> String {
    get_external_id_str(job_name, "job")
}

pub fn get_query_eid(query_name: &str) -> String {
    get_external_id_str(query_name, "query")
}

[1]: meaning that components outside of the his codebase can hold onto a particular id and it means the same underlying entity all the time.

Time

There is only one small function in this to get the string version of Utc::now().

pub fn utc_now() -> String {
    let now: DateTime<Utc> = Utc::now();
    now.to_rfc3339()
}