[Kafka] Add new dashboards and link the newly added dashboards #15328

agithomas · 2025-09-16T04:01:36Z

Enhancement

Proposed commit message

Add Kafka dashboards for the newly added datasets - jvm, log_manager, raft, replicamanager, topic, controller, network datasets. Link the existing dashboards to the newly added dashboards

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Package upgrade testing
Dashboard loading check

How to test this PR locally

elastic-package build && elastic-package stack up -v -d --services package-registry

Related issues

#15243

Screenshots

Existing dashboards updated (with link panel)

Overview

Logs

Newly added dashboards

Controller

JVM

Log manager

Network

Raft

Replica manager

Topic

…troller network datasets

elastic-vault-github-plugin-prod · 2025-09-16T04:31:29Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

shmsr · 2025-09-17T07:58:54Z

Overall the PR looks good to me.

Few comments:

In "Logs" overview; in the logs, we can filter out "INFO", "DEBUG" by default as the user is more interested in"WARN", "ERROR", etc. I think it is a good default to have.
nitpick: Some chart titles have "Sentence case" and some have "Title Case". Should be consistent.
In metricbeat-kafka-raft page, clarify "Current state" as the state is just listed as "follower." It would be helpful to have a panel that lists all brokers and their current Raft state (e.g., leader, follower, candidate). But currently, it just says current state but the state is for which broker and all is not there.

agithomas · 2025-09-17T13:01:00Z

In metricbeat-kafka-raft page, clarify "Current state" as the state is just listed as "follower." It would be helpful to have a panel that lists all brokers and their current Raft state (e.g., leader, follower, candidate). But currently, it just says current state but the state is for which broker and all is not there.

This gives (ONLY) the status of the node that is connected to. This does not give the status of all the nodes and its type (leader, follower, etc)

Some chart titles have "Sentence case" and some have "Title Case". Should be consiste

Addressed for the newly added dashboards. For the dashboards that were existing already, this will be taken up in the upcoming PR

In "Logs" overview; in the logs, we can filter out "INFO", "DEBUG" by default as the user is more interested in"WARN", "ERROR", etc. I think it is a good default to have.

This is an existing dashboard. Improvements will be taken up in the upcoming enhancement.

agithomas · 2025-09-17T13:55:44Z

/test

daniela-elastic · 2025-09-17T21:05:49Z

packages/kafka/img/filebeat-kafka-logs-overview.png

What is a "stracktrace" - the first widget ("number of stracktraces by class"). Also, we should spell this at "stack trace" (two words) in all the widgets

Logs dashboard was an existing dashboard. I have made the corrections as per your inputs.

daniela-elastic · 2025-09-17T21:16:18Z

packages/kafka/img/metricbeat-kafka-jvm.png

The "JVM name" widget is too small to hold the entire name. We should use a widget that is big enough to store the longest JVM name + a buffer

"JVM version" widget seems like it's showing the version in some scientific notation. Are we supposed to see "+9" in the number?

"JVM Vendor" widget - is this going to be big enough to hold any vendor name length?

"Uptime" says "a month" - is there a number missing?

"Thread count" widget - we shouldn't need the two zeros after the decimal point. Presumably thread count is integers, there are no thread fractions AFAIK

Heap usage, especially % should be one of the top widgets, given how important memory is. We should of course keep the memory section but do show the most important memory widget also at the top of this dashboard

The "JVM name" widget is too small to hold the entire name. We should use a widget that is big enough to store the longest JVM name + a buffer

Updated.

"JVM version" widget seems like it's showing the version in some scientific notation. Are we supposed to see "+9" in the number?

The version number do appear in this format.

"JVM Vendor" widget - is this going to be big enough to hold any vendor name length?

Updated to give a larger width

"Uptime" says "a month" - is there a number missing?

Updated from friendly -> accurate option.

"Thread count" widget - we shouldn't need the two zeros after the decimal point. Presumably thread count is integers, there are no thread fractions AFAIK

This is because of the user of compact options. For large value of Thread count, having a decimal value become relevant. But, i agree that it is irrelevant if the value is small (lesser than 100). Presently, we don't have a way to limit this.

Heap usage, especially % should be one of the top widgets, given how important memory is. We should of course keep the memory section but do show the most important memory widget also at the top of this dashboard

Updated as suggested.

daniela-elastic · 2025-09-17T21:22:34Z

packages/kafka/img/metricbeat-kafka-log_manger.png

Shouldn't "Logs" and "Log manager" dashboard tabs be right next to each other, rather than have "Raft" tab between them?

What does the "Log recovery status" widget show? Count of what? Is it possible to name the y-axis with what is actually counted?

"Log directory status" = 0. What does it mean when the status is 0? Is this is a good or a bad thing? Does everybody know what 0 means when it comes to log directory status?

What is a "Dead cleaner threads"? Is that "dead letter queue"?

We should use US spelling - eg "utilization" instead of "utilisation", "behavior" instead of "behaviour", etc

What does the "Log recovery status" widget show? Count of what? Is it possible to name the y-axis with what is actually counted?

There are two count values here - Remaining logs to recover , Remaining segments to recover. As you can see in the image below, it is represented as two series colours at the bottom of the panel.

"Log directory status" = 0. What does it mean when the status is 0? Is this is a good or a bad thing? Does everybody know what 0 means when it comes to log directory status?

Corrected by changing from a table panel to a search view panel

What is a "Dead cleaner threads"? Is that "dead letter queue"?

Updated as Dead log cleaner threads. These metrics are part of log cleaner metrics

We should use US spelling - eg "utilization" instead of "utilisation", "behavior" instead of "behaviour", etc

Updated.

daniela-elastic · 2025-09-17T21:25:57Z

packages/kafka/img/metricbeat-kafka-network.png

Fix spelling to use US spelling (eg "utilization" instead of "utilisation", etc

What re the units of "Temporary memory" and "Request size distribution" widgets for the y-axis?

Fix spelling to use US spelling (eg "utilization" instead of "utilisation", etc

Updated

What re the units of "Temporary memory" and "Request size distribution" widgets for the y-axis?

It is bytes. It is correctly configured to appear in the Y-axis, next to the Y-axis value (B in the screenshot)

daniela-elastic · 2025-09-17T21:27:47Z

packages/kafka/img/metricbeat-kafka-raft.png

"Poll idle ratio" - is this Poll-to-Idle ratio?

The ratio of time the Raft IO thread is idle as opposed to doing work.

I have update the panel title as - I/O Thread Idle Ratio , to avoid confusions.

daniela-elastic · 2025-09-17T21:31:23Z

packages/kafka/img/metricbeat-kafka-replica_manager.png

In the description of the dashboard as well as the first section named "ISR Changes" it might be worth spelling out once (in both places) what ISR means, eg "In-Sync Replicas (ISRs)". In the description of the dashboard - on first mention of the acronym. (optionally) In the naming of the section - "In-Sync Replicas (ISRs) Changes"

Updated as suggested - at the first mention of the acronym and the section title.

daniela-elastic · 2025-09-17T21:34:07Z

packages/kafka/img/metricbeat-kafka-topic.png

What are the units for the "At minimum ISR status per topic" and "Under minimum ISR status per topic" widgets?

Its the partition count. Updated the y-axis as Partition count.

The title of the panel is updated as - Partitions below minimum ISR per topic for better clarity.

daniela-elastic · 2025-09-17T21:36:13Z

packages/kafka/img/metricbeat_kafka_dashboard.png

Are we missing a description widget for the "Overview" dashboard?

We normally have more colors in the Overview dashboard especially for the most important metrics at the top. Shall we bring back some colors?

Updated as suggested

daniela-elastic

Left a few comments and questions. Please let me know if you have any questions

elasticmachine · 2025-09-18T17:43:59Z

💚 Build Succeeded

Buildkite Build
Commit: 527c301

History

💚 Build #31663 succeeded 25efb63
💚 Build #31559 succeeded 0abad29
💔 Build #31551 failed 29d944f
💚 Build #31400 succeeded 800ce4a

cc @agithomas

elastic-sonarqube · 2025-09-18T17:44:06Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

agithomas · 2025-09-19T05:05:44Z

Left a few comments and questions. Please let me know if you have any questions

All the comments have been addressed.

The placement of the "Logs" menu option is kept in a way so that it is not missed between the Metrics based dashboards. We have two more dashboard panels, for producer and consumers, to be added to the list of available dashboards. I suggest, the menu re-arrangement is taken up as part of this upcoming PR, when we have all the dashboards available.

agithomas added 2 commits September 16, 2025 03:44

Add kafka dashboards for jvm logmanager raft replicamanager topic con…

f71b43e

…troller network datasets

Updated PR link and screenshot

800ce4a

agithomas added Integration:kafka Kafka dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Sep 16, 2025

agithomas marked this pull request as ready for review September 16, 2025 04:54

agithomas requested a review from a team as a code owner September 16, 2025 04:54

lalit-satapathy requested a review from daniela-elastic September 16, 2025 05:41

Updated from title case to sentence case

29d944f

agithomas and others added 2 commits September 17, 2025 19:27

Merge branch 'main' into issue-obs-integrations-400-kafkadashboards

72dc962

Updated the version number. Updated the image resolutions

0abad29

shmsr assigned agithomas Sep 17, 2025

daniela-elastic reviewed Sep 17, 2025

View reviewed changes

agithomas added 2 commits September 18, 2025 16:02

Updated dashboards with minor fixes

25efb63

Minor spell corrections

527c301

[Kafka] Add new dashboards and link the newly added dashboards #15328

Are you sure you want to change the base?

[Kafka] Add new dashboards and link the newly added dashboards #15328

Conversation

agithomas commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Existing dashboards updated (with link panel)

Overview

Logs

Newly added dashboards

Controller

JVM

Log manager

Network

Raft

Replica manager

Topic

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 16, 2025

🚀 Benchmarks report

Uh oh!

shmsr commented Sep 17, 2025

Uh oh!

agithomas commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agithomas commented Sep 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniela-elastic Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniela-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Sep 18, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Sep 18, 2025

Quality Gate passed

Uh oh!

agithomas commented Sep 19, 2025

Uh oh!

agithomas commented Sep 16, 2025 •

edited

Loading

agithomas commented Sep 17, 2025 •

edited

Loading

daniela-elastic Sep 17, 2025 •

edited

Loading