Hello 🖐️ !

Following the mechanisms described here to send CSV files into Kafka in JSON using kcat through shell pipelines, we packed 📦 all of that in a new home-made tool named producer-file. Here are the features :

• 💨 Sending line by line
• 💪 CSV to JSON conversion
• 🔦 Support message headers by file and/or by line
• ⚡️ Delta-changes mode (only send changes occurred after last send)

☝️ More about the shell way :

⌨️ Pipe xlsx files into/from Kafka... From cli with (k)cat 🙀

adriens for opt-nc ・ Oct 10 '22 ・ 2 min read

#tutorial #productivity #kafka #linux

Motivations

Using jq + kcat with shell pipelines do the job well, but this doesn't fit the company's requirements today about software packaging and logging capabilities 🤷🏻‍♂️.

📽️ Demo

Short video demonstrating how to use this tool to send a CSV file in JSON into Kafka within a Docker environnement :

📖 Ressources

• 🐳 producer-file
• 🥷 Killercoda about the "shell way"
• 🪶 Kafka