Speaker: Damien Gasparina, Engineer, Confluent
Here's how to fail at Apache Kafka brilliantly!
https://www.meetup.com/Paris-Data-Engineers/events/260694777/
31. Builtin Retries
Retries : will cause the client to resend any record
whose send fails with a potentially transient error.
Default value : MAX_INT (before AK 2.1 : 0...)
32. What’s happening in case of issue with
retry ?
Producer No longer leader
L
The leader moved to a
different broker
53. 2) Write to a dead letter queue and
continue
while (this.getRunning()) {
var consumerRecords = consumer.poll(1000) ;
for (var record : consumerRecords) {
try {
/* Processing messages */
} catch (Exception e) {
producer.send(« dead-my-topic », new ProducerRecord(…)) ;
Logger.error(e);
}
}
}
54. 3) Ignore & continue
kafkaProducer.send(record, (
(metadata, exception)→ {
if (exception != null) {
/* Something bad happened */
/* But those are ephemaral data anyway */
Logger.error(exception) ;
}
})
);
65. The best things to do is to .…
Read the goddam manual !
Running Apache Kafka in Production
Running Apache Zookeeper in Production
* How to succeed *