Most integrations today run through APIs and queues. An order is written in the system of record, an event goes to Kafka or RabbitMQ, a microservice stores its own copy, and a cache or a CDN serves the result to customers. It works until something in the middle hiccups: a network blip, a timeout, a deploy at the wrong moment, a bug in one consumer.
The pipeline keeps running, so nothing looks broken. But the copies are no longer the same. The source has the order; the service database does not; the cache still shows yesterday's price. Usually a customer is the first to notice.
Why retries are not enough
Retries cover the case where a call failed and everyone knows it. The hard cases are the quiet ones:
- a message left the producer and never reached the consumer;
- the consumer received it, hit a bug, and stored the wrong value;
- the cache was rebuilt from a service database that was already behind.
In all three, the question is the same: which records are different right now? Answering it by hand means writing comparison queries between two databases, often on two different engines, and it is the kind of script nobody wants to maintain.
Step 1 — compare key by key
dbSlice compares a table in the source with the same table in the service database, key by key, and reports three lists: rows missing in the target, rows extra in the target and rows whose values changed. The two sides can be different engines: SQL Server on one side and PostgreSQL on the other is a normal case.
The keys that differ come out comma-separated, ready to paste into a WHERE … IN (…):
SELECT * FROM orders WHERE order_id IN (100231, 100245, 100246, 100318);Step 2 — fix the service database
Use that SELECT as the source of an Upsert run into the service database. Only the listed rows are read and written: missing rows are inserted, changed rows are updated, the rest of the table is untouched.
Step 3 — re-send only what drifted
The same SELECT can feed Send data, which posts each row to a REST API (with OAuth2 if needed), SOAP, gRPC or a database procedure, or publishes it to Kafka, Redpanda, Pulsar or RabbitMQ. That is how the downstream pieces catch up:
- point it at your endpoint that rebuilds a cache entry or purges a CDN path, so customers see the fixed record;
- or publish the records again to the topic or queue the consumers already read.
For REST calls, dbSlice repeats a request on its own only when that is safe: 429 and 503 answers, a refused connection, and for GET, PUT and DELETE also 5xx and timeouts.
Step 4 — make it a nightly check
The comparison does not need a person in front of the screen. Saved as a read-only reconciliation job, it runs from the command line or Windows Task Scheduler and sends its report by e-mail or webhook, like any other dbSlice job. On most nights the report says "no differences". On the night it doesn't, you have the list of keys before a customer calls.
If you also use Postback (a call to your API or queue for each row dbSlice writes), its failures go to a separate log with the job, table, key, operation, event ID and reason. That log is the list of keys to re-send with the same SELECT … IN.
What this is not
- Not real-time replication or CDC. These are batch jobs, on demand or scheduled.
- Not a queue consumer. dbSlice does not read from Kafka or RabbitMQ or handle dead-letter queues; it compares
databases and sends to the queue as a destination.
- Deletes show up as "extra in the target". Decide per table whether to remove them or keep them.
Checklist
- Pick the tables that matter most when an integration fails (orders, prices, stock, customers).
- Save a nightly reconciliation job for each, with the report to the team's e-mail or webhook.
- When it finds differences: Upsert the service database with SELECT … IN, then Send data to the API that
refreshes your cache or to the queue your consumers read.
Try it on your own data: the free version on the Microsoft Store connects, browses and compares; writing to a target needs a Desktop or Server license. More on per-row calls in Notify apps on every row.
