This document describes how to monitor the health of clients that produce or consume data in your Managed Service for Apache Kafka cluster.
It's important to monitor client applications as part of your overall reliability strategy. Metrics such as throughput, error rates, and consumer lag can tell you whether client applications are experiencing reliability issues. Problems might be caused by client configuration, uneven distribution of keys across partitions, or cluster issues that affect only a specific partition.
Server-side metrics
While it's useful to monitor client behavior directly, server-side metrics don't require any additional instrumentation, and can help you to detect client-side issues that affect reliability.
Server-side metrics are especially useful for detecting load imbalances across brokers (hot brokers), and deviations in normal operations, such as spikes in latency.
Throughput
Monitor the following throughput metrics and compare them against your expected throughput:
- Message rate, per broker and per topic.
- Byte rate, per broker and per topic.
- Request rates. A misconfigured client might flood brokers with a high rate of small requests (0-1000 bytes per request), reducing throughput.
- Request latency. Spikes in producer request latency might signal imbalanced load or a problem with client configuration.
Kafka offers throughput metrics per topic and for the cluster. These metrics don't always have the same values when aggregated for all topics. Use an aggregated metric for high-level monitoring and alerting, and look at per-topic metrics when you troubleshoot throughput problems. Isolate any issues to specific brokers.
Request error rates
The topic_error_count
metric tracks the number of failed fetch and produce requests on the server
side. However, some classes of error aren't reflected in this metric. For
example:
Misconfigured authorization settings might prevent a client from producing to a topic, without the error appearing in this metric.