Join Log in
Score:0
Florin Andrei avatar Server

Kafka: monitoring length of queues / topics

in flag
Florin Andrei

We have several producers and several consumers connected via Kafka - basically, it's batch processing jobs that are created on demand, placed on Kafka in several queues, and the batch processors pick them up via Kafka and process them one by one.

I want to visualize and monitor the lengths of these queues in Kafka. The queue lengths will serve as proxies for the "load" of the overall system. The more jobs waiting in queues, the more "loaded" the system is.

Our Kafka is an AWS MSK cluster. I've enabled Prometheus JMX monitoring, and I'm scraping all metrics every 10 seconds.

Looking at the metrics, there's nothing that appears immediately obvious in terms of queue length. Is that parameter exposed as a metric by default?

If queue length is not exposed by default, what is a good way to collect that metric? Assume I can write a Python script with any libraries installed, and I have full access to the Kafka endpoints from it.

Note: I understand the basic concepts, but I don't have a lot of practical experience with Kafka (I've only used RabbitMQ before), so apologies if my vocabulary is not very precise. E.g. what I called "queue" is apparently called "topic".

126
0 + 0
de flag
kofemann
a) this question does not belong here. b) in kafka each consumer has its own 'last seen', thus you have to monitor last event - last seen for each consumer.
0
Reply
Florin Andrei avatar
in flag
Florin Andrei
@kofemann Thank you for the tips. Would SuperUser be a better fit for this question?
0
Reply
de flag
kofemann
To me, the question sounds like a programing question. I guess stackoverflow probably a fits better.
1
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.