Kafka exporter

The Kafka exporter is SQLLoad’s mode for Kafka. It runs continuously, reads Kafka topics and writes each one into a Snowflake table, through Snowpipe Streaming or staged files. It’s separate from loader projects: its sqlload.toml has [source] with kind = "kafka", [destination], [staging] and the sections in Configuration, instead of [connections], [targets] and [sources]. The repository’s sqlload.example.toml is a complete example.

Commands

Command	Does
sqlload tables [--topics-file FILE]	Prints topic<TAB>table<TAB>stream<TAB>group for each topic, without connecting to Snowflake.
sqlload plan	Shows each topic’s table and the objects it needs.
sqlload apply	Creates missing tables, stages and pipes.
sqlload run [--max-duration D] [--shard N]	Exports until stopped, or for --max-duration, such as 10m.
sqlload status	Shows per-partition lag and committed offsets.
sqlload debug	Validates the configuration and tests the connections.

Option	Meaning
--topics-file <FILE>	tables only: read topic names, one per line, from this file instead of Kafka.
--max-duration <D>	run only: stop after this long.
--shard <N>	run only: this instance’s index below source.shards. Also SQLLOAD_SHARD.

How rows reach Snowflake

[staging] kind chooses the route:

kind	Route
none	Snowpipe Streaming.
internal	Files on a table stage, loaded with Snowpipe or COPY.
s3	Files in your bucket, loaded with Snowpipe or COPY.

With Snowpipe loads, Kafka offsets are committed only once Snowpipe reports the files loaded. Each topic’s rows carry the record’s envelope (topic, partition, offset, key, value, timestamps); see Envelope columns.

Naming topics

tables shows how topics map to tables before anything connects. With the example configuration and a file of topic names:

$ sqlload tables --topics-file topics.txt
shop.legacy-orders  ORDERS_V1  orders_v1  sqlload-orders_v1
shop.orders_v2.events  ORDERS__EVENTS  orders__events  sqlload-orders__events
shop.payments_v1.events  PAYMENTS__EVENTS  payments__events  sqlload-payments__events

shop.orders.dlq was in the file too, but matches the example’s exclude, so it isn’t listed.

Naming topics covers the rules.