Files

Import and export data into/from files on filesystem and cloud storage.

Install

Persistent.

uv tool install --upgrade 'cratedb-toolkit[io-ingest]'

Ephemeral.

uvx 'cratedb-toolkit[io-ingest]'

Container.

docker run --rm ghcr.io/crate/cratedb-toolkit-ingest ctk

Cloud storage

Amazon S3

ctk load \
    "s3://?access_key_id=${AWS_ACCESS_KEY_ID}&secret_access_key=${AWS_SECRET_ACCESS_KEY}&table=openaq-fetches/realtime/2023-02-25/1677351953_eea_2aa299a7-b688-4200-864a-8df7bac3af5b.ndjson#jsonl" \
    "crate://crate:na@localhost:4200/testdrive/s3_ndjson"

See documentation about ingestr and Amazon S3 about details of the URI format, file globbing patterns, compression options, and file type hinting options.

Source URL template: s3://?access_key_id=<aws-access-key-id>&secret_access_key=<aws-secret-access-key>&table=<bucket-name>/<file-glob>

Google Cloud Storage (GCS)

ctk load \
    "gs://?credentials_path=/path/to/service-account.json&table=<bucket-name>/<file-glob>" \
    "crate://crate:na@localhost:4200/testdrive/gcs"

File formats

CSV

ctk load \
    "csv://./examples/cdc/postgresql/diamonds.csv?table=sample" \
    "crate://crate:na@localhost:4200/testdrive/csv_diamonds"