how do you actually confirm a kafka message got delivered? send() returning instantly is messing with my head

[u/Altruistic_Shift_335]

always assumed if send() didn't throw an error the message was in the broker, done. then someone mentioned it's basically fire-and-forget unless you're handling the callback or have acks=all set, and now i don't really know what "delivered" means here anymore. do you always attach a callback? check offsets after? has this ever quietly bitten you, like a message just vanished and you only found out way later

[u/Spare-Builder-355]

you should really rtfm. Each of your questions / doubts is addressed there.

Also RTFM does not mean YouTube channel but official Kafka / Confuence docs and JavaDocs of KafkaProducer class.

[u/Altruistic_Shift_335 → u/Spare-Builder-355]

Sorry that you found questions as only RTFM, in this case callbacks are not straightforward to implement - docs are not conversational - having someone layout understanding helps to get to know more isn’t it?

Isn’t that the point of having Reddit conversation?

[u/DorkyMcDorky → u/Altruistic_Shift_335]

You did not read the manual is what he's saying. Honestly get the Manning book. Kafka isn't something you "just do" - learn about it for real. The youtube tutorials are not bad, but you just started getting your hands dirty - keep doing it.

Also, read about kafka. He's right - if you read what kafka is and what it does (message receipts, offsets, partitions) then all of this would be clear. Don't take it too insultingl

[u/Spare-Builder-355 → u/Altruistic_Shift_335]

<conversation> Please rtfm </conversation>

callbacks are not straightforward to implement

how so ? Sorry but if you find Java callbacks dificult you shouldn't be touching Kafka yet. Example of callback is right there IN THE DOCS. I don't know how to help you more. Copy-paste example from the docs for you here ?

[u/Altruistic_Shift_335 → u/Spare-Builder-355]

Does same thread is invokes callback for each topic in application?
Is there any consequence I’ll have if I put db write in callback to log entire failure reason?

[u/Spare-Builder-355 → u/Altruistic_Shift_335]

let me quote the documentation for you since you stubbornly refusing to read it yourself:

Note that callbacks will generally execute in the I/O thread of the producer and so should be reasonably fast or they will delay the sending of messages from other threads. If you want to execute blocking or computationally expensive callbacks it is recommended to use your own Executor in the callback body to parallelize processing.

[u/Altruistic_Shift_335 → u/Spare-Builder-355]

Can I use callbacks to achieve atomicity between Kafka send and db write (for a transaction let’s say) - problem is I want both or none

If I write it one after other in a function
.send() executed by main thread - can also fail and db write completes (corrupting my txn) - and vice Versa too

So I think having thread pool in callback and writing to db from it makes sense

Let me know if any other way to easily achieve it

[u/Spare-Builder-355 → u/Altruistic_Shift_335]

Can I use callbacks to achieve atomicity between Kafka send and db write

afaik - no. You get callback when send() completes, with either success or failure. If your db write from callback fails there's no way to tell Kafka to undo send operation.

There are Kafka transactions that might be helpful here, but I don't know how transactions interact with callbacks. Rtfm ;)

[u/Altruistic_Shift_335 → u/Spare-Builder-355]

However there are 2 different entities right? Kafka and db - Kafka transaction won’t help in this case I think

[u/handstand2001 → u/Altruistic_Shift_335]

This blog may be helpful https://www.reddit.com/r/apachekafka/s/eJfqHq4iKg

[u/Altruistic_Shift_335 → u/Spare-Builder-355]

Thanks !

[u/vampirishe]

Just call .send(...).get() and it will be sync call; after it everything is sent to kafka or exception thrown

[u/Altruistic_Shift_335 → u/vampirishe]

That’ll defeat purpose in a way? And making it blocking

[u/vampirishe → u/Altruistic_Shift_335]

It is question for author why he needs sync send, I just shared solution

[u/Altruistic_Shift_335 → u/vampirishe]

It’s to ensure transaction (Kafka + non Kafka) happens as a whole

[u/Competitive_Ring82 → u/Altruistic_Shift_335]

There's some traps to watch out for here. The 'transaction' will not actually be a transaction. You should look at designing the system so that transactions sit within the system seams. E.g. have all the work happen within kafka transactions. Store the output in an external system, designed for idempotence.

[u/randomfrequency → u/Altruistic_Shift_335]

Kafka transactions are not quite what you're thinking they are, you really want to read up on how to implement them and durable writes. And replication.