Crates.io | tpctools |
lib.rs | tpctools |
version | 0.7.0 |
source | src |
created_at | 2021-05-15 14:27:23.262482 |
updated_at | 2024-03-03 18:24:08.54266 |
description | Utilities for generating and converting TPC-H and TPC-DS data sets |
homepage | https://github.com/sql-benchmarks/tpctools/ |
repository | https://github.com/sql-benchmarks/tpctools/ |
max_upload_size | |
id | 397881 |
size | 133,823 |
Command-line tools for invoking TPC-H and TPC-DS data generators in parallel and re-organizing the output files into directory structures that can be consumed by tools such as Apache Spark or Apache Arrow DataFusion/Ballista.
Also supports converting the output to Parquet.
Install dependencies.
sudo apt install gcc make flex bison byacc git
Download data generator from https://www.tpc.org/tpc_documents_current_versions/current_specifications5.asp
cd /path/to/DSGen-software-code-3.2.0rc1/tools
make
Generate data.
mkdir /tmp/tpcds/sf1000
cargo run --release -- generate --benchmark tpcds \
--scale 1000 \
--partitions 48 \
--generator-path /path/to/DSGen-software-code-3.2.0rc1/tools \
--output /tmp/tpcds/sf1000/
Example output.
Generated TPC-DS data at scale factor 1000 with 48 partitions in: 6247.155671938s
Convert to Parquet
mkdir /tmp/tpcds/sf1000-parquet
cargo run --release -- convert --benchmark tpcds \
--input /tmp/tpcds/sf1000/
--output /tmp/tpcds/sf1000-parquet/
Install dependencies.
git clone git@github.com:databricks/tpch-dbgen.git
cd tpch-dbgen
make
cd ..
Generate data.
mkdir /tmp/tpch
cargo run --release -- generate --benchmark tpch \
--scale 1 \
--partitions 2 \
--generator-path ./tpch-dbgen/ \
--output /tmp/tpch
Convert data to Parquet
mkdir /tmp/tpch-parquet
cargo run --release -- convert \
--benchmark tpch \
--input /tmp/tpch/ \
--output /tmp/tpch-parquet/
TPC-H is Copyright © 1993-2022 Transaction Processing Performance Council. The full TPC-H specification in PDF format can be found here
TPC-DS is Copyright © 2021 Transaction Processing Performance Council. The full TPC-DS specification in PDF format can be found here
TPC, TPC Benchmark, TPC-H, and TPC-DS are trademarks of the Transaction Processing Performance Council.