Open table formats

Import and export data into/from open table formats on filesystem and cloud storage.

Install

Persistent.

uv tool install --upgrade 'cratedb-toolkit[io-curated]'

Ephemeral.

uvx 'cratedb-toolkit[io-curated]'

Container.

docker run --rm ghcr.io/crate/cratedb-toolkit ctk

Integrations

I/O adapters for Apache Iceberg tables [1] and Delta Lake tables [2], to enhance interoperability with open table formats. They build upon Apache Parquet data files, a free and open-source column-oriented data storage format, effectively succeeding and superseding Apache Hive use cases from the Hadoop era.

Apache Iceberg

Apache Iceberg I/O

Delta Lake

Delta Lake I/O

---

[1]

Iceberg is a specification and high-performance format for huge analytic tables, making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time. Apache Iceberg is its reference implementation.

[2]

Delta Lake (paper) is the optimized storage layer that provides the foundation and default format for all table operations on Databricks. It was developed for tight integration with Structured Streaming, allowing you to easily use a single copy of data for both batch and streaming operations and providing incremental processing at scale.