Skip to content

[Kafka] Add new dashboards and link the newly added dashboards #15328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Sep 30, 2025
Merged

[Kafka] Add new dashboards and link the newly added dashboards #15328

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f71b43e
Add kafka dashboards for jvm logmanager raft replicamanager topic con…
agithomas Sep 16, 2025
800ce4a
Updated PR link and screenshot
agithomas Sep 16, 2025
29d944f
Updated from title case to sentence case
agithomas Sep 17, 2025
72dc962
Merge branch 'main' into issue-obs-integrations-400-kafkadashboards
agithomas Sep 17, 2025
0abad29
Updated the version number. Updated the image resolutions
agithomas Sep 17, 2025
25efb63
Updated dashboards with minor fixes
agithomas Sep 18, 2025
527c301
Minor spell corrections
agithomas Sep 18, 2025
6b1d9a7
Minor fixes to JVM dashboard
agithomas Sep 30, 2025
2dacde1
Merge branch 'main' into issue-obs-integrations-400-kafkadashboards
agithomas Sep 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/kafka/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.24.0"
changes:
- description: Add dashboard for the datasets topic, jvm, controller, network, replica_manager, log_manager and raft.
type: enhancement
link: https://github.com/elastic/integrations/pull/15328
- version: "1.23.0"
changes:
- description: Add support for producer and consumer data streams.
Expand Down
Binary file modified packages/kafka/img/filebeat-kafka-logs-overview.png
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is a "stracktrace" - the first widget ("number of stracktraces by class"). Also, we should spell this at "stack trace" (two words) in all the widgets

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logs dashboard was an existing dashboard. I have made the corrections as per your inputs.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packages/kafka/img/metricbeat-kafka-controller.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packages/kafka/img/metricbeat-kafka-jvm.png
View file
Open in desktop
Copy link

@daniela-elastic daniela-elastic Sep 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. The "JVM name" widget is too small to hold the entire name. We should use a widget that is big enough to store the longest JVM name + a buffer
  2. "JVM version" widget seems like it's showing the version in some scientific notation. Are we supposed to see "+9" in the number?
  3. "JVM Vendor" widget - is this going to be big enough to hold any vendor name length?
  4. "Uptime" says "a month" - is there a number missing?
  5. "Thread count" widget - we shouldn't need the two zeros after the decimal point. Presumably thread count is integers, there are no thread fractions AFAIK
  6. Heap usage, especially % should be one of the top widgets, given how important memory is. We should of course keep the memory section but do show the most important memory widget also at the top of this dashboard

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "JVM name" widget is too small to hold the entire name. We should use a widget that is big enough to store the longest JVM name + a buffer

Updated.

"JVM version" widget seems like it's showing the version in some scientific notation. Are we supposed to see "+9" in the number?

The version number do appear in this format.

"JVM Vendor" widget - is this going to be big enough to hold any vendor name length?

Updated to give a larger width

"Uptime" says "a month" - is there a number missing?

Updated from friendly -> accurate option.

"Thread count" widget - we shouldn't need the two zeros after the decimal point. Presumably thread count is integers, there are no thread fractions AFAIK

This is because of the user of compact options. For large value of Thread count, having a decimal value become relevant. But, i agree that it is irrelevant if the value is small (lesser than 100). Presently, we don't have a way to limit this.

Heap usage, especially % should be one of the top widgets, given how important memory is. We should of course keep the memory section but do show the most important memory widget also at the top of this dashboard

Updated as suggested.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why for GC; we have two charts; one says rate and other says time. But y-axis is same.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, why GC is there memory pool chart? Looking at related charts online; it looks to me as G1GC Memory Pools. Can you check; I did not search much but looks like that only.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JVM name should be should "JVM type"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Classes Loading / loaded; has inconsistency. Also is the "total" classes loaded so far? Check both charts.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About JIT compilation; is it the rate; or it showing compiler activity. Also y-axis seems unclear.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packages/kafka/img/metricbeat-kafka-log_manger.png
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Shouldn't "Logs" and "Log manager" dashboard tabs be right next to each other, rather than have "Raft" tab between them?
  2. What does the "Log recovery status" widget show? Count of what? Is it possible to name the y-axis with what is actually counted?
  3. "Log directory status" = 0. What does it mean when the status is 0? Is this is a good or a bad thing? Does everybody know what 0 means when it comes to log directory status?
  4. What is a "Dead cleaner threads"? Is that "dead letter queue"?
  5. We should use US spelling - eg "utilization" instead of "utilisation", "behavior" instead of "behaviour", etc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does the "Log recovery status" widget show? Count of what? Is it possible to name the y-axis with what is actually counted?

There are two count values here - Remaining logs to recover , Remaining segments to recover. As you can see in the image below, it is represented as two series colours at the bottom of the panel.

image

"Log directory status" = 0. What does it mean when the status is 0? Is this is a good or a bad thing? Does everybody know what 0 means when it comes to log directory status?

Corrected by changing from a table panel to a search view panel

What is a "Dead cleaner threads"? Is that "dead letter queue"?

Updated as Dead log cleaner threads. These metrics are part of log cleaner metrics

We should use US spelling - eg "utilization" instead of "utilisation", "behavior" instead of "behaviour", etc

Updated.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packages/kafka/img/metricbeat-kafka-network.png
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Fix spelling to use US spelling (eg "utilization" instead of "utilisation", etc
  2. What re the units of "Temporary memory" and "Request size distribution" widgets for the y-axis?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix spelling to use US spelling (eg "utilization" instead of "utilisation", etc

Updated

What re the units of "Temporary memory" and "Request size distribution" widgets for the y-axis?

It is bytes. It is correctly configured to appear in the Y-axis, next to the Y-axis value (B in the screenshot)

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packages/kafka/img/metricbeat-kafka-raft.png
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. "Poll idle ratio" - is this Poll-to-Idle ratio?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ratio of time the Raft IO thread is idle as opposed to doing work.

I have update the panel title as - I/O Thread Idle Ratio , to avoid confusions.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packages/kafka/img/metricbeat-kafka-replica_manager.png
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. In the description of the dashboard as well as the first section named "ISR Changes" it might be worth spelling out once (in both places) what ISR means, eg "In-Sync Replicas (ISRs)". In the description of the dashboard - on first mention of the acronym. (optionally) In the naming of the section - "In-Sync Replicas (ISRs) Changes"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated as suggested - at the first mention of the acronym and the section title.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packages/kafka/img/metricbeat-kafka-topic.png
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the units for the "At minimum ISR status per topic" and "Under minimum ISR status per topic" widgets?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its the partition count. Updated the y-axis as Partition count.

The title of the panel is updated as - Partitions below minimum ISR per topic for better clarity.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified packages/kafka/img/metricbeat_kafka_dashboard.png
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Are we missing a description widget for the "Overview" dashboard?
  2. We normally have more colors in the Overview dashboard especially for the most important metrics at the top. Shall we bring back some colors?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated as suggested

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1,750 changes: 1,750 additions & 0 deletions packages/kafka/kibana/dashboard/kafka-20094380-8713-487d-8036-5cbbb529f0ca.json
View file
Open in desktop

Large diffs are not rendered by default.

2,430 changes: 2,430 additions & 0 deletions packages/kafka/kibana/dashboard/kafka-28d2bf78-a638-4a1f-a46f-1a794c99166b.json
View file
Open in desktop

Large diffs are not rendered by default.

2,156 changes: 2,156 additions & 0 deletions packages/kafka/kibana/dashboard/kafka-32a191b0-c8ac-4ea1-812d-0e3d406f9a41.json
View file
Open in desktop

Large diffs are not rendered by default.

3,052 changes: 3,052 additions & 0 deletions packages/kafka/kibana/dashboard/kafka-894682fd-555e-49c6-b469-f8fbb3f0b9dc.json
View file
Open in desktop

Large diffs are not rendered by default.

286 changes: 277 additions & 9 deletions packages/kafka/kibana/dashboard/kafka-943caca0-87ee-11e7-ad9c-db80de0bf8d3.json
View file
Open in desktop

Large diffs are not rendered by default.

2,393 changes: 2,393 additions & 0 deletions packages/kafka/kibana/dashboard/kafka-c2cb8091-e812-4a23-9b59-3d461c4898ce.json
View file
Open in desktop

Large diffs are not rendered by default.

1,746 changes: 1,746 additions & 0 deletions packages/kafka/kibana/dashboard/kafka-e3be4fe8-c7b4-4cea-bb2e-953032cc322a.json
View file
Open in desktop

Large diffs are not rendered by default.

244 changes: 228 additions & 16 deletions packages/kafka/kibana/dashboard/kafka-ea488d90-8e63-11e8-8fa2-3d5f811fbd0f.json
View file
Open in desktop

Large diffs are not rendered by default.

2,763 changes: 2,763 additions & 0 deletions packages/kafka/kibana/dashboard/kafka-ff622b8a-bbe2-4903-add2-ba3ad50806d2.json
View file
Open in desktop

Large diffs are not rendered by default.

34 changes: 31 additions & 3 deletions packages/kafka/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: "3.0.2"
name: kafka
title: Kafka
version: "1.23.0"
version: "1.24.0"
description: Collect logs and metrics from Kafka servers with Elastic Agent.
type: integration
categories:
Expand All @@ -15,11 +15,39 @@ conditions:
screenshots:
- src: /img/filebeat-kafka-logs-overview.png
title: filebeat kafka logs overview
size: 1363x657
size: 2272x2386
type: image/png
- src: /img/metricbeat_kafka_dashboard.png
title: metricbeat kafka dashboard
size: 1726x711
size: 1653x1003
type: image/png
- src: /img/metricbeat-kafka-controller.png
title: Kafa controller dashboard
size: 2142x2620
type: image/png
- src: /img/metricbeat-kafka-jvm.png
title: Kafa jvm dashboard
size: 1663x3713
type: image/png
- src: /img/metricbeat-kafka-log_manger.png
title: Kafa log manager dashboard
size: 2698x2344
type: image/png
- src: /img/metricbeat-kafka-network.png
title: Kafa network dashboard
size: 2336x2536
type: image/png
- src: /img/metricbeat-kafka-raft.png
title: Kafa raft dashboard
size: 1663x1949
type: image/png
- src: /img/metricbeat-kafka-replica_manager.png
title: Kafa replica manager dashboard
size: 1663x2209
type: image/png
- src: /img/metricbeat-kafka-topic.png
title: Kafa topic dashboard
size: 2036x2646
type: image/png
icons:
- src: /img/logo_kafka.svg
Expand Down