Forward port fixes to 9.2 (#2552)

* Change entity multi_field text type to match_only_text (#2547)

Change the field type for multi_field values in entity from text to
match_only_text.

This change will match the multi_field type used for other similar fields in
other schema files, and will match the expected dynamic value in ecs@mappings.

* Fix entity generation (#2548)

Correctly add top-level entity fields to the generated files.
This also updates the make clean command to clean generated doc files.

* Update embedded semconv version to v1.37.0 (#2550)

Update the semconv version used with ECS to generate documentation to
semantic-conventions v1.37.0.

This will ensure that documentation is built with the latest semconv
version, and prepare ECS to work with more related fields.

Author: mjwolf

Makefile

@@ -40,6 +40,15 @@ check_license_headers:
 .PHONY: clean
 clean:
 	rm -rf build generated/elasticsearch/composable/component experimental/generated/elasticsearch/composable/component
+	# Clean generated documentation files
+	@echo "Removing generated documentation files..."
+	@rm -f docs/reference/index.md docs/reference/ecs-field-reference.md docs/reference/ecs-otel-alignment-details.md docs/reference/ecs-otel-alignment-overview.md
+	@for schema in $$(ls schemas/*.yml 2>/dev/null | sed 's/schemas\///' | sed 's/\.yml$$//'); do \
+		if [ -f "docs/reference/ecs-$$schema.md" ]; then \
+			echo "Removing docs/reference/ecs-$$schema.md"; \
+			rm -f "docs/reference/ecs-$$schema.md"; \
+		fi; \
+	done
 
 # Build and serve the docs
 .PHONY: docs
@@ -107,7 +116,7 @@ generate: generator
 # Run the new generator
 .PHONY: generator
 generator: ve
-	$(PYTHON) scripts/generator.py --strict --include "${INCLUDE}" --subset "${SUBSETS_DIR}" --semconv-version "${SEMCONV_VERSION}" --force-docs
+	$(PYTHON) scripts/generator.py --strict $(if $(INCLUDE),--include "$(INCLUDE)") --subset "${SUBSETS_DIR}" --semconv-version "${SEMCONV_VERSION}" --force-docs
 
 # Check Makefile format.
 .PHONY: makelint

docs/reference/ecs-container.md

@@ -29,7 +29,7 @@ These fields help correlate data based containers from any runtime.
 | $$$field-container-name$$$ [container.name](#field-container-name) | Container name.<br><br>type: keyword<br><br>![OTel Badge](https://img.shields.io/badge/OpenTelemetry-4a5ca6?style=flat&logo=opentelemetry) [![match](https://img.shields.io/badge/match-93c93e?style=flat)](/reference/ecs-opentelemetry.md#ecs-opentelemetry-relation) [container.name](https://opentelemetry.io/docs/specs/semconv/attributes-registry/container/#container-name) | extended |
 | $$$field-container-network-egress-bytes$$$ [container.network.egress.bytes](#field-container-network-egress-bytes) | The number of bytes (gauge) sent out on all network interfaces by the container since the last metric collection.<br><br>type: long | extended |
 | $$$field-container-network-ingress-bytes$$$ [container.network.ingress.bytes](#field-container-network-ingress-bytes) | The number of bytes received (gauge) on all network interfaces by the container since the last metric collection.<br><br>type: long | extended |
-| $$$field-container-runtime$$$ [container.runtime](#field-container-runtime) | Runtime managing this container.<br><br>type: keyword<br><br>example: `docker`<br><br>![OTel Badge](https://img.shields.io/badge/OpenTelemetry-4a5ca6?style=flat&logo=opentelemetry) [![match](https://img.shields.io/badge/match-93c93e?style=flat)](/reference/ecs-opentelemetry.md#ecs-opentelemetry-relation) [container.runtime](https://opentelemetry.io/docs/specs/semconv/attributes-registry/container/#container-runtime) | extended |
+| $$$field-container-runtime$$$ [container.runtime](#field-container-runtime) | Runtime managing this container.<br><br>type: keyword<br><br>example: `docker`<br><br>![OTel Badge](https://img.shields.io/badge/OpenTelemetry-4a5ca6?style=flat&logo=opentelemetry) [![equivalent](https://img.shields.io/badge/equivalent-1ba9f5?style=flat)](/reference/ecs-opentelemetry.md#ecs-opentelemetry-relation) [container.runtime.name](https://opentelemetry.io/docs/specs/semconv/attributes-registry/container/#container-runtime-name) | extended |
 | $$$field-container-security-context-privileged$$$ [container.security_context.privileged](#field-container-security-context-privileged) | Indicates whether the container is running in privileged mode.<br><br>type: boolean | extended |
 
 

docs/reference/ecs-dns.md

@@ -17,7 +17,7 @@ DNS events should either represent a single DNS query prior to getting answers (
 
 | Field | Description | Level |
 | --- | --- | --- |
-| $$$field-dns-answers$$$ [dns.answers](#field-dns-answers) | An array containing an object for each answer section returned by the server.<br><br>The main keys that should be present in these objects are defined by ECS. Records that have more information may contain more keys than what ECS defines.<br><br>Not all DNS data sources give all details about DNS answers. At minimum, answer objects must contain the `data` key. If more information is available, map as much of it to ECS as possible, and add any additional fields to the answer objects as custom fields.<br><br>type: object<br><br>Note: This field should contain an array of values. | extended |
+| $$$field-dns-answers$$$ [dns.answers](#field-dns-answers) | An array containing an object for each answer section returned by the server.<br><br>The main keys that should be present in these objects are defined by ECS. Records that have more information may contain more keys than what ECS defines.<br><br>Not all DNS data sources give all details about DNS answers. At minimum, answer objects must contain the `data` key. If more information is available, map as much of it to ECS as possible, and add any additional fields to the answer objects as custom fields.<br><br>type: object<br><br>Note: This field should contain an array of values.<br><br>![OTel Badge](https://img.shields.io/badge/OpenTelemetry-4a5ca6?style=flat&logo=opentelemetry) [![related](https://img.shields.io/badge/related-efc20d?style=flat)](/reference/ecs-opentelemetry.md#ecs-opentelemetry-relation) [dns.answers](https://opentelemetry.io/docs/specs/semconv/attributes-registry/dns/#dns-answers) | extended |
 | $$$field-dns-answers-class$$$ [dns.answers.class](#field-dns-answers-class) | The class of DNS data contained in this resource record.<br><br>type: keyword<br><br>example: `IN` | extended |
 | $$$field-dns-answers-data$$$ [dns.answers.data](#field-dns-answers-data) | The data describing the resource.<br><br>The meaning of this data depends on the type and class of the resource record.<br><br>type: keyword<br><br>example: `10.10.10.10` | extended |
 | $$$field-dns-answers-name$$$ [dns.answers.name](#field-dns-answers-name) | The domain name to which this resource record pertains.<br><br>If a chain of CNAME is being resolved, each answer's `name` should be the one that corresponds with the answer's `data`. It should not simply be the original `question.name` repeated.<br><br>type: keyword<br><br>example: `www.example.com` | extended |

docs/reference/ecs-entity.md

@@ -17,16 +17,17 @@ The entity fields provide a standardized way to represent and categorize differe
 | --- | --- | --- |
 | $$$field-entity-attributes$$$ [entity.attributes](#field-entity-attributes) | _This field is beta and subject to change._ A set of static or semi-static attributes of the entity. Usually boolean or keyword field data types. Use this field set when you need to track static or semi-static characteristics of an entity for advanced searching and correlation of normalized values across different providers/sources and entity types.<br><br>type: object | extended |
 | $$$field-entity-behavior$$$ [entity.behavior](#field-entity-behavior) | _This field is beta and subject to change._ A set of ephemeral characteristics of the entity, derived from observed behaviors during a specific time period. Usually boolean field data type. Use this field set when you need to capture and track ephemeral characteristics of an entity for advanced searching, correlation of normalized values across different providers/sources and entity types.<br><br>type: object | extended |
-| $$$field-entity-display_name$$$ [entity.display_name](#field-entity-display_name) | _This field is beta and subject to change._ An optional field used when a pretty name is desired for entity-centric operations. This field should not be used for correlation with `*.name` fields for entities with dedicated field sets (e.g., `host`).<br><br>type: keyword<br><br>Multi-fields:<br><br>* entity.display_name.text (type: text) | extended |
+| $$$field-entity-display-name$$$ [entity.display_name](#field-entity-display-name) | _This field is beta and subject to change._ An optional field used when a pretty name is desired for entity-centric operations. This field should not be used for correlation with `*.name` fields for entities with dedicated field sets (e.g., `host`).<br><br>type: keyword<br><br>Multi-fields:<br><br>* entity.display_name.text (type: match_only_text) | extended |
 | $$$field-entity-id$$$ [entity.id](#field-entity-id) | A unique identifier for the entity. When multiple identifiers exist, this should be the most stable and commonly used identifier that: 1) persists across the entity's lifecycle, 2) ensures uniqueness within its scope, 3) is commonly used for queries and correlation, and 4) is readily available in most observations (logs/events). For entities with dedicated field sets (e.g., host, user), this value should match the corresponding *.id field. Alternative identifiers (e.g., ARNs values in AWS, URLs) can be preserved in the raw field.<br><br>type: keyword | core |
-| $$$field-entity-last_seen_timestamp$$$ [entity.last_seen_timestamp](#field-entity-last_seen_timestamp) | _This field is beta and subject to change._ Indicates the date/time when this entity was last "seen," usually based upon the last event/log that is initiated by this entity.<br><br>type: date | extended |
+| $$$field-entity-last-seen-timestamp$$$ [entity.last_seen_timestamp](#field-entity-last-seen-timestamp) | _This field is beta and subject to change._ Indicates the date/time when this entity was last "seen," usually based upon the last event/log that is initiated by this entity.<br><br>type: date | extended |
 | $$$field-entity-lifecycle$$$ [entity.lifecycle](#field-entity-lifecycle) | _This field is beta and subject to change._ A set of temporal characteristics of the entity. Usually date field data type. Use this field set when you need to track temporal characteristics of an entity for advanced searching and correlation of normalized values across different providers/sources and entity types.<br><br>type: object | extended |
 | $$$field-entity-metrics$$$ [entity.metrics](#field-entity-metrics) | _This field is beta and subject to change._ Field set for any fields containing numeric entity metrics. These use dynamic field data type mapping.<br><br>type: object | extended |
-| $$$field-entity-name$$$ [entity.name](#field-entity-name) | _This field is beta and subject to change._ The name of the entity. The keyword field enables exact matches for filtering and aggregations, while the text field enables full-text search. For entities with dedicated field sets (e.g., `host`), this field should mirrors the corresponding *.name value.<br><br>type: keyword<br><br>Multi-fields:<br><br>* entity.name.text (type: text) | core |
+| $$$field-entity-name$$$ [entity.name](#field-entity-name) | _This field is beta and subject to change._ The name of the entity. The keyword field enables exact matches for filtering and aggregations, while the text field enables full-text search. For entities with dedicated field sets (e.g., `host`), this field should mirrors the corresponding *.name value.<br><br>type: keyword<br><br>Multi-fields:<br><br>* entity.name.text (type: match_only_text) | core |
 | $$$field-entity-raw$$$ [entity.raw](#field-entity-raw) | _This field is beta and subject to change._ Original, unmodified fields from the source system. Usually flattened field data type. While the attributes field should be used for normalized fields requiring advanced queries, this field preserves all source metadata with basic search capabilities.<br><br>type: object | extended |
 | $$$field-entity-reference$$$ [entity.reference](#field-entity-reference) | _This field is beta and subject to change._ A URI, URL, or other direct reference to access or locate the entity in its source system. This could be an API endpoint, web console URL, or other addressable location. Format may vary by entity type and source system.<br><br>type: keyword | extended |
 | $$$field-entity-source$$$ [entity.source](#field-entity-source) | _This field is beta and subject to change._ The module or integration that provided this entity data (similar to event.module).<br><br>type: keyword | core |
-| $$$field-entity-type$$$ [entity.type](#field-entity-type) | _This field is beta and subject to change._ A standardized high-level classification of the entity. This provides a normalized way to group similar entities across different providers or systems. Example values: `bucket`, `database`, `container`, `function`, `queue`, `host`, `user`, `application`, `session`, etc.<br><br>type: keyword<br><br>example: `host` | core |
+| $$$field-entity-sub-type$$$ [entity.sub_type](#field-entity-sub-type) | _This field is beta and subject to change._ The specific type designation for the entity as defined by its provider or system. This field provides more granular classification than the type field. Examples: `aws_s3_bucket`, `gcp_cloud_storage_bucket`, `azure_blob_container` would all map to entity type `bucket`.  `hardware` , `virtual` , `container` , `node` , `cloud_instance` would all map to entity type `host`.<br><br>type: keyword<br><br>example: `aws_s3_bucket` | extended |
+| $$$field-entity-type$$$ [entity.type](#field-entity-type) | _This field is beta and subject to change._ A standardized high-level classification of the entity. This provides a normalized way to group similar entities across different providers or systems. Example values: `bucket`, `database`, `container`, `function`, `queue`, `host`, `user`, `application`, `session`, etc.<br><br>type: keyword<br><br>Note: This field should contain an array of values.<br><br>**Important:** The field value must be one of the following:<br><br>bucket, database, container, function, queue, host, user, application, service, session<br><br>To learn more about when to use which value, visit the page [allowed values for entity.type](/reference/ecs-allowed-values-entity-type.md)<br> | core |
 
 ## Field reuse [_field_reuse]
 

docs/reference/ecs-field-reference.md

@@ -38,6 +38,7 @@ For a single page representation of all fields, please see the [generated CSV of
 | [ECS](/reference/ecs-ecs.md) | Meta-information specific to ECS. |
 | [ELF Header](/reference/ecs-elf.md) | These fields contain Linux Executable Linkable Format (ELF) metadata. |
 | [Email](/reference/ecs-email.md) | Describes an email transaction. |
+| [Entity](/reference/ecs-entity.md) | Fields to describe various types of entities across IT environments. |
 | [Error](/reference/ecs-error.md) | Fields about errors of any kind. |
 | [Event](/reference/ecs-event.md) | Fields breaking down the event details. |
 | [FaaS](/reference/ecs-faas.md) | Fields describing functions as a service. |

docs/reference/ecs-gen_ai.md

@@ -36,7 +36,7 @@ This field group definition is based on the Gen AI namespace of the OpenTelemetr
 | $$$field-gen-ai-response-finish-reasons$$$ [gen_ai.response.finish_reasons](#field-gen-ai-response-finish-reasons) | _This field is beta and subject to change._ Array of reasons the model stopped generating tokens, corresponding to each generation received.<br><br>type: nested<br><br>example: `["stop", "length"]`<br><br>![OTel Badge](https://img.shields.io/badge/OpenTelemetry-4a5ca6?style=flat&logo=opentelemetry) [![match](https://img.shields.io/badge/match-93c93e?style=flat)](/reference/ecs-opentelemetry.md#ecs-opentelemetry-relation) [gen_ai.response.finish_reasons](https://opentelemetry.io/docs/specs/semconv/attributes-registry/gen-ai/#gen-ai-response-finish-reasons) | extended |
 | $$$field-gen-ai-response-id$$$ [gen_ai.response.id](#field-gen-ai-response-id) | _This field is beta and subject to change._ The unique identifier for the completion.<br><br>type: keyword<br><br>example: `chatcmpl-123`<br><br>![OTel Badge](https://img.shields.io/badge/OpenTelemetry-4a5ca6?style=flat&logo=opentelemetry) [![match](https://img.shields.io/badge/match-93c93e?style=flat)](/reference/ecs-opentelemetry.md#ecs-opentelemetry-relation) [gen_ai.response.id](https://opentelemetry.io/docs/specs/semconv/attributes-registry/gen-ai/#gen-ai-response-id) | extended |
 | $$$field-gen-ai-response-model$$$ [gen_ai.response.model](#field-gen-ai-response-model) | _This field is beta and subject to change._ The name of the model that generated the response.<br><br>type: keyword<br><br>example: `gpt-4-0613`<br><br>![OTel Badge](https://img.shields.io/badge/OpenTelemetry-4a5ca6?style=flat&logo=opentelemetry) [![match](https://img.shields.io/badge/match-93c93e?style=flat)](/reference/ecs-opentelemetry.md#ecs-opentelemetry-relation) [gen_ai.response.model](https://opentelemetry.io/docs/specs/semconv/attributes-registry/gen-ai/#gen-ai-response-model) | extended |
-| $$$field-gen-ai-system$$$ [gen_ai.system](#field-gen-ai-system) | _This field is beta and subject to change._ The Generative AI product as identified by the client or server instrumentation.<br><br>type: keyword<br><br>example: `openai`<br><br>![OTel Badge](https://img.shields.io/badge/OpenTelemetry-4a5ca6?style=flat&logo=opentelemetry) [![match](https://img.shields.io/badge/match-93c93e?style=flat)](/reference/ecs-opentelemetry.md#ecs-opentelemetry-relation) [gen_ai.system](https://opentelemetry.io/docs/specs/semconv/attributes-registry/gen-ai/#gen-ai-system) | extended |
+| $$$field-gen-ai-system$$$ [gen_ai.system](#field-gen-ai-system) | _This field is beta and subject to change._ The Generative AI product as identified by the client or server instrumentation.<br><br>type: keyword<br><br>example: `openai`<br><br>![OTel Badge](https://img.shields.io/badge/OpenTelemetry-4a5ca6?style=flat&logo=opentelemetry) [![equivalent](https://img.shields.io/badge/equivalent-1ba9f5?style=flat)](/reference/ecs-opentelemetry.md#ecs-opentelemetry-relation) [gen_ai.provider.name](https://opentelemetry.io/docs/specs/semconv/attributes-registry/gen-ai/#gen-ai-provider-name) | extended |
 | $$$field-gen-ai-token-type$$$ [gen_ai.token.type](#field-gen-ai-token-type) | _This field is beta and subject to change._ The type of token being counted.<br><br>type: keyword<br><br>example: `input; output`<br><br>![OTel Badge](https://img.shields.io/badge/OpenTelemetry-4a5ca6?style=flat&logo=opentelemetry) [![match](https://img.shields.io/badge/match-93c93e?style=flat)](/reference/ecs-opentelemetry.md#ecs-opentelemetry-relation) [gen_ai.token.type](https://opentelemetry.io/docs/specs/semconv/attributes-registry/gen-ai/#gen-ai-token-type) | extended |
 | $$$field-gen-ai-tool-call-id$$$ [gen_ai.tool.call.id](#field-gen-ai-tool-call-id) | _This field is beta and subject to change._ The tool call identifier.<br><br>type: keyword<br><br>example: `call_mszuSIzqtI65i1wAUOE8w5H4`<br><br>![OTel Badge](https://img.shields.io/badge/OpenTelemetry-4a5ca6?style=flat&logo=opentelemetry) [![match](https://img.shields.io/badge/match-93c93e?style=flat)](/reference/ecs-opentelemetry.md#ecs-opentelemetry-relation) [gen_ai.tool.call.id](https://opentelemetry.io/docs/specs/semconv/attributes-registry/gen-ai/#gen-ai-tool-call-id) | extended |
 | $$$field-gen-ai-tool-name$$$ [gen_ai.tool.name](#field-gen-ai-tool-name) | _This field is beta and subject to change._ Name of the tool utilized by the agent.<br><br>type: keyword<br><br>example: `Flights`<br><br>![OTel Badge](https://img.shields.io/badge/OpenTelemetry-4a5ca6?style=flat&logo=opentelemetry) [![match](https://img.shields.io/badge/match-93c93e?style=flat)](/reference/ecs-opentelemetry.md#ecs-opentelemetry-relation) [gen_ai.tool.name](https://opentelemetry.io/docs/specs/semconv/attributes-registry/gen-ai/#gen-ai-tool-name) | extended |