Add Kafka topic pattern matching and metadata-based dynamic index naming

[Copilot]

Implements regex-based topic discovery and dynamic index naming using Kafka metadata, similar to Logstash's topics_pattern and %{[@metadata][kafka][topic]} functionality.

Changes

Configuration (stash/config/)

• Added TopicsPattern field to KafkaConf for regex-based topic matching
• Implemented FetchMatchingTopics() with broker failover to discover topics dynamically

Metadata Injection (stash/handler/)

• Enhanced MessageHandler to inject Kafka topic name into @metadata.kafka.topic
• Extracted ensureMetadataStructure() helper for cleaner metadata initialization

Index Formatting (stash/es/)

• Added getNestedValue() to support dot-notation field access (e.g., @metadata.kafka.topic)
• Updated buildIndexFormatter() to resolve nested fields in index templates

Main (stash/stash.go)

• Modified toKqConf() to fetch topics via pattern matching
• Updated handler initialization to pass topic context per consumer

Usage

Topic pattern matching:

Input:
  Kafka:
    TopicsPattern: "^app-.*-logs$"  # Matches app-service1-logs, app-service2-logs, etc.

Dynamic index naming:

Output:
  ElasticSearch:
    Index: "{[email protected]}-{{yyyy-MM-dd}}"  # Creates my-topic-2024-01-15

Both features work independently or together. Existing explicit Topics configurations remain fully supported.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>能实现像logstash中topics_pattern功能么？</issue_title>
<issue_description>目前go-stash只支持配置单个topic,如果有上千个topic,需要一个一个配置还是可以支持正则？
另外，output也可以实现像logstash中的index => "%{[@metadata][kafka][topic]}-%{+YYYY-MM-dd}" 类似功能么？</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes 能实现像logstash中topics_pattern功能么？ #14

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.