> For the complete documentation index, see [llms.txt](https://ztrust.gitbook.io/livestreamiq/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ztrust.gitbook.io/livestreamiq/readme/3.-guide-to-navigation/3.4-cluster-overview/3.4.3-broker.md).
# 3.4.3 Broker
Kafka Broker is a server in an Kafka cluster that stores, manages, and processes messages from Kafka topics. It handles communication between producers and consumers by receiving, storing, and delivering data efficiently.
Key Components on the Kafka Broker:
1. **Uptime:**\
Uptime in Kafka Broker is the total time a broker remains active and available without failure or restart. It shows the reliability and operational status of the Kafka broker.
1. **Broker Count:** Broker Count is the total number of active brokers in a Kafka cluster that manage and store data.
2. **Active Controller:** Active Controller is the Kafka broker that manages cluster metadata and partition activities.
3. **Version:** Version indicates the current Kafka version running on the broker or cluster.
2. **Partitions:** \
Partitions in Kafka are divisions of a topic that allow data to be distributed and processed in parallel. Each partition stores messages in an ordered sequence and helps improve Kafka performance and scalability.
1. **Online:** Online Partitions are active Kafka partitions available for data read and write operations.
2. **URP:** URP (Under Replicated Partitions) are Kafka partitions that do not have all required replicas available. It indicates a replication issue and may affect data reliability.
3. **In Sync Replicas:** In-Sync Replicas are Kafka replicas that are fully updated and synchronized with the leader partition. They are ready to take over if the leader fails.
4. **Out of Sync Replicas:** Out of Sync Replicas are Kafka replicas that are not updated with the leader partition. They may be unavailable for data replication or failover.
3. **Broker List:**\
Broker List in Kafka is a collection of all broker details available in a Kafka cluster. It provides information about Kafka servers that handle data storage, message processing, and communication between producers and consumers.
1. **Broker Id:** A unique identifier assigned to each Kafka broker in the cluster.
2. **Disk Usage:** The amount of storage space currently used by the Kafka broker.
3. **Partitions skew:** Indicates an uneven distribution of partitions across Kafka brokers.
4. **Leaders:** The number of partition leaders managed by a Kafka broker.
5. **Leader skew:** Indicates an uneven distribution of partition leaders across Kafka brokers.
6. **Online Partitions:** The number of active partitions available for reading and writing data.
7. **Online:** Shows whether the Kafka broker is currently active and available.
8. **Host:** The hostname or IP address of the Kafka broker.
**After clicking inside any Broker navigate to Broker setting:**
* **Host:** The hostname or IP address where the Kafka broker is running.
* **Port:** The network port used by the Kafka broker to accept client connections.
* **Segment Count:** The total number of log segment files stored by the Kafka broker.
* **Segment Size:** The total disk space used by all log segments on the Kafka broker.
**Navigate to Configurations of the specific Broker:**
1. **Log Directories:** Displays the directories where the broker stores Kafka log data.
1. **Segment Size:** The total disk space used by all log segments on the Kafka broker.
2. **Segment Count:** The total number of log segment files stored by the Kafka broker.
3. **Port:** The network port used by the Kafka broker to accept client connections.
4. **Host:** The hostname or IP address where the Kafka broker is running.
2. **Configs:** Displays the configuration settings applied to the selected Kafka broker.
1. **Key:** Displays the name of the Kafka broker configuration parameter.
2. **Value:** Displays the current value of the configuration parameter. These values can be modified using the **Edit** option.
*When you click on the value field, it becomes editable. You can update the broker configuration value and click **Save** to apply the changes. If you click **Cancel**, the changes will be discarded and the previous value will remain unchanged.*
* **Source:** Indicates the source from which the configuration value is applied (for example, DEFAULT\_CONFIG).
* **Search option:** Users can search for the required entry using either the key or the corresponding value.
3. **Metrics:** Displays real-time performance and resource utilization metrics for the Kafka broker.\
\
LiveStreamIQ provides a comprehensive set of metrics to monitor the health, performance, and operational status of Kafka environments.
1. **Producer Latency (ms):**
1. The average time taken by producers to send messages to Kafka.
2. Displays the time taken by producers to send messages to Kafka.
3. X-axis: Shows the time range.
4. Y-axis: Shows latency in milliseconds (ms).
5. The graph helps monitor producer performance and identify message sending delays.
6. **Percentile Dropdown (P50, P75, etc.)**:
1. This dropdown allows you to select the latency percentile view for Kafka metrics.
2. It shows how different percentage ranges of requests/messages are performing.
7. Filter Options:
1. P50: Shows the median latency. 50% of requests are completed within this time.
2. P75: Shows the latency where 75% of requests are completed.
3. P95: Shows the latency where 95% of requests are completed (useful for identifying slow requests).
4. P98: Shows the latency for 98% of requests.
5. P99: Shows the latency for 99% of requests and highlights high delays.
6. P99.9: Shows the latency for 99.9% of requests, used to identify extreme performance issues.
8. Use:
1. Use lower percentiles (P50) to check normal performance.
2. Use higher percentiles (P95–P99.9) to detect slow requests and performance problems.
2. Consumer Latency (ms): The average time taken by consumers to receive and process messages.
1. Displays the time taken by consumers to receive and process messages from Kafka.
2. X-axis: Shows the time range.
3. Y-axis: Shows latency in milliseconds (ms).
4. The graph shows a constant latency of around 3 ms, indicating stable message consumption performance.
5. Helps monitor consumer performance and identify message processing delays.
3. Follower Latency (ms): The delay between the leader partition and its follower replicas.
1. Displays the delay between the leader partition and its follower replicas in Kafka.
2. X-axis: Shows the time range.
3. Y-axis: Shows latency in milliseconds (ms).
4. The graph shows a constant latency of around 500 ms, indicating the replication delay is stable.
5. It helps monitor Kafka replication performance and identify follower synchronization delays.
4. OS utilization: The percentage of operating system resources currently in use.
1. Process CPU Load: Shows CPU usage by the Kafka process.
2. System CPU Load: Shows overall operating system CPU usage.
3. CPU Load: Shows total CPU utilization.
4. System Load Average: Shows the current system workload.
5. Available Processors: Displays the number of CPU cores.
6. Total Memory: Shows total RAM available.
7. Free Memory: Shows unused memory.
8. Used Memory: Shows memory currently in use.
9. Memory Usage: Shows the percentage of RAM used.
10. Formula: Used Memory / Total Memory × 100
11. Used to monitor system performance and identify resource issues.
5. Heap Memory Consumption: The amount of JVM heap memory currently being used.
1. isplays the JVM heap memory usage of the Kafka broker.
2. Used (23.36%): Shows the amount of heap memory currently used by Kafka.
3. Available (76.64%): Shows the remaining free heap memory available.
4. Calculation: *Used Memory / Total Heap Memory × 100*
6. Use:
1. Helps monitor Kafka memory usage.
2. Helps identify memory pressure and prevent OutOfMemory issues.
7. Non-Heap Memory Consumption: The amount of JVM non-heap memory currently being used.
1. Displays the JVM non-heap memory usage of the Kafka broker.
2. Used (67.01%): Shows the non-heap memory currently used by JVM (such as class metadata and JVM internal data).
3. Available (32.99%): Shows the remaining non-heap memory available.
4. Calculation: *Used Non-Heap Memory / Total Non-Heap Memory × 100*
5. Use:
1. Helps monitor JVM non-heap memory usage.
2. Helps identify memory allocation issues and JVM performance problems
8. **GC Collection Events:** The total number of garbage collection events performed by the JVM.
1. Displays the number of times the JVM performs Garbage Collection (GC) to clean unused memory.
2. Old Gen: Shows GC events for long-lived objects.
3. Young Gen: Shows GC events for newly created objects.
4. Current value: 119054 events
5. GC Collection Time (seconds):
1. Displays the total time spent by JVM performing garbage collection.
2. Old Gen: Total GC time for old generation.
3. Current value: 0 seconds
4. Young Gen: Total GC time for young generation.
5. Current value: 1702.508 seconds
6. Use:
1. Helps monitor JVM memory cleanup activity.
2. Helps identify if frequent garbage collection is affecting Kafka performance.
9. **GC Collection Time:** The total time spent by the JVM performing garbage collection.
1. Process CPU Load(%):The CPU usage of the Kafka broker process.
10. **Process CPU Load (%) Graph:**
1. Displays the CPU usage of the Kafka broker process.
2. Used (4.90%): Shows the CPU currently used by the Kafka process.
3. Available (95.10%): Shows the remaining CPU capacity available.
4. Calculation: *Process CPU Usage / Total CPU Capacity × 100*
5. Use:
1. Helps monitor Kafka process CPU performance.
2. Helps identify high CPU usage and resource bottlenecks.
11. **System Load Avg(%):** The average system workload over a period of time.
1. Displays the average workload of the system over a period of time.
2. Used (0.00%): Shows the current system load being used.
3. Available (100.00%): Shows the remaining system capacity available.
4. Calculation: *System Load Average / Total System Capacity × 100*
5. Use:
1. Helps monitor overall system workload.
2. Helps identify if the server is under heavy load or has enough resources available.
12. **System CPU Load(%):** The overall CPU usage of the host system.
1. Displays the overall CPU usage of the host operating system.
2. Used (1.50%): Shows the CPU currently used by the system.
3. Available (98.50%): Shows the remaining CPU capacity available.
4. Calculation: *Used CPU Time / Total CPU Time × 100*
5. Use:
1. Helps monitor overall system CPU performance.
2. Helps identify high CPU usage and system resource issues.
13. **Total Bytes/Sec:**The total amount of data transferred per second.
1. Displays the total amount of data received by a Kafka topic per second.
2. X-axis: Shows the time range.
3. Y-axis: Shows data transfer rate in Bytes/sec.
4. Helps monitor the incoming data traffic to Kafka topics.
5. A higher value indicates more data is being produced to the topic.
6. Use this metric to track topic data flow and identify traffic changes.
14. **Produce requests/Sec:** The number of producer requests received per second.
1. Displays the number of producer requests received by the Kafka broker per second.
2. X-axis: Shows the time range.
3. Y-axis: Shows the number of requests per second.
4. Helps monitor producer activity and message publishing rate.
5. A higher value indicates more producer requests being sent to Kafka.
6. Use this metric to analyze producer traffic and broker workload
15. **Messages in/Sec:** The number of messages received by the Kafka broker per second.
1. Displays the number of messages received by the Kafka broker per second.
2. X-axis: Shows the time range.
3. Y-axis: Shows the number of messages per second.
4. Helps monitor the incoming message traffic and message flow rate.
5. A higher value indicates more messages are being produced to Kafka.
6. Use this metric to track message activity and identify traffic changes.
16. **Time Range Configuration:** Allows users to select the time period for viewing metric data.
1. Refresh Time Interval: Specifies how often the metrics are automatically refreshed on the dashboard.
2. Allows users to select the time period for viewing Kafka metrics data.
3. Absolute Time Range: Allows selecting a custom start and end date/time.
1. From: Select the starting date and time.
2. To: Select the ending date and time.
4. Click Apply time range to view data for the selected period.
5. Quick Time Range Options: Provides predefined time periods:
1. Last 5 minutes
2. Last 15 minutes
3. Last 30 minutes
4. Last 1 hour
5. Last 3 hours
6. Last 6 hours
7. Last 12 hours
8. Last 24 hours
6. Use:
1. Use this option to analyze Kafka metrics for a specific time period and monitor performance trends.
2. Allows users to set how frequently the Kafka metrics dashboard refreshes automatically.
3. It updates the displayed metric data based on the selected interval.
17. The following available Options are:
1. Off: Stops automatic refresh.
2. 5s: Refreshes data every 5 seconds.
3. 10s: Refreshes data every 10 seconds.
4. 15s: Refreshes data every 15 seconds.
5. 30s: Refreshes data every 30 seconds.
6. 1m: Refreshes data every 1 minute.
