Where the data comes from and where it goes
- From: a table, view or SELECT in SQL Server, PostgreSQL, Oracle, MySQL, MariaDB, Firebird, SQLite or InterSystems IRIS; or a JSON or CSV file.
- To: Kafka, Redpanda, Apache Pulsar, RabbitMQ, a REST API, a SOAP service, a gRPC method, or a stored procedure in another database.
The source is only read.
Shape the message, then look before you send
- Map fields from the source to the JSON you need, including nested objects (
customer.name). - Clean values on the way: trim, change case, replace, fixed values, number and date formats, value maps, masking and hashing for sensitive fields. No scripts to write.
- Preview builds the JSON for the first records without sending anything and shows which records would fail and why.
- Send a sample of a few records to check the destination, then run the rest.
Know what happened to every record
Each record ends in one of these states, so you can tell a rejection from a timeout:
| State | Meaning |
|---|---|
| Confirmed | The destination acknowledged it |
| Rejected | The destination refused it (for example HTTP 400, or no queue to route to) |
| Unknown | Sent, but no confirmation arrived; it may have been delivered |
| Invalid | Failed your mapping rules; not sent |
Failed records are listed in a file with their number and reason, without the record content, and every run is kept in the history.
Run it again tomorrow
Save the definition and run it from the CLI or Windows Task Scheduler. Credentials are stored protected for your Windows user, not in clear text.
What it is not
- Not exactly-once. There is no transaction between the database and the broker. Running the same definition again sends the records again, and an interrupted run starts over from the beginning. Design the consumer to handle duplicates (an idempotent key is usually enough).
- Not change data capture. It sends the rows you select when you run it; it does not stream changes as they happen. Use a WHERE filter (for example
updated_at > …) for recurring runs. - No Schema Registry. Messages are plain JSON in UTF-8; Avro and Protobuf with a registry are not supported.
- Produce only. It does not consume from topics or queues.
Part of data coexistence →