diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 9edaa1de4e56..2c56e564e76b 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -7,8 +7,14 @@ === Beats version 8.19.3 https://github.com/elastic/beats/compare/v8.19.2\...v8.19.3[View commits] -==== Known issues +==== Known Issues + +*Filebeat* +// tag::known-issue-8.19-filestream-clean-inactive-1[] +- The Filestream input does not enforce the restrictions documented for the `clean_inactive` option, thus allowing configurations that can lead to data re-ingestion issues. +- When `clean_inactive: 0`, Filestream cleans the state of all files on start up, effectively re-ingesting all files on restart. Set `clean_inactive: -1` to disable this behavior. +// end::known-issue-8.19-filestream-clean-inactive-1[] // tag::known-issue-8.19-restart_on_cert_change[] - `restart_on_cert_change` causes panic due to seccomp policy. In versions 8.19.0 and later, enabling this option causes the Beat to panic on restart. This is due to the `eventfd2` syscall missing from the default seccomp policy. To fix this, add `eventfd2` to a custom seccomp policy. For more details, refer to {filebeat-ref}/linux-seccomp.html[Use Linux Secure Computing Mode (seccomp)]. @@ -192,16 +198,24 @@ https://github.com/elastic/beats/compare/v8.19.1\...v8.19.2[View commits] ==== Known issues +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] include::CHANGELOG.asciidoc[tags=known-issue-8.19-restart_on_cert_change] + [[release-notes-8.19.1]] === Beats version 8.19.1 https://github.com/elastic/beats/compare/v8.19.0\...v8.19.1[View commits] -==== Known issues +==== Known Issues +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] include::CHANGELOG.asciidoc[tags=known-issue-8.19-restart_on_cert_change] + ==== Bugfixes *Filebeat* @@ -225,8 +239,11 @@ include::CHANGELOG.asciidoc[tags=known-issue-8.19-restart_on_cert_change] === Beats version 8.19.0 https://github.com/elastic/beats/compare/v8.18.4\...v8.19.0[View commits] -==== Known issues +==== Known Issues + +*Filebeat* +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] include::CHANGELOG.asciidoc[tags=known-issue-8.19-restart_on_cert_change] ==== Breaking changes @@ -309,6 +326,13 @@ include::CHANGELOG.asciidoc[tags=known-issue-8.19-restart_on_cert_change] === Beats version 8.18.5 https://github.com/elastic/beats/compare/v8.18.4\...v8.18.5[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Bugfixes *Auditbeat* @@ -333,6 +357,14 @@ https://github.com/elastic/beats/compare/v8.18.4\...v8.18.5[View commits] === Beats version 8.18.4 https://github.com/elastic/beats/compare/v8.18.3\...v8.18.4[View commits] +==== Known Issues + +*Filebeat* + +- The Filestream input does not enforce the restrictions documented for the `clean_inactive` option, thus allowing configurations that can lead to data re-ingestion issues. +- When `clean_inactive: 0`, Filestream cleans the state of all files on start up, effectively re-ingesting all files on restart. Set `clean_inactive: -1` to disable this behavior. + + ==== Breaking changes *Metricbeat* @@ -373,6 +405,13 @@ https://github.com/elastic/beats/compare/v8.18.3\...v8.18.4[View commits] === Beats version 8.18.3 https://github.com/elastic/beats/compare/v8.18.2\...v8.18.3[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Bugfixes *Affecting all Beats* @@ -422,6 +461,12 @@ https://github.com/elastic/beats/compare/v8.18.2\...v8.18.3[View commits] === Beats version 8.18.2 https://github.com/elastic/beats/compare/v8.18.1\...v8.18.2[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + ==== Bugfixes *Affecting all Beats* @@ -462,6 +507,13 @@ https://github.com/elastic/beats/compare/v8.18.1\...v8.18.2[View commits] === Beats version 8.18.1 https://github.com/elastic/beats/compare/v8.18.0\...v8.18.1[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Bugfixes *Filebeat* @@ -483,6 +535,13 @@ https://github.com/elastic/beats/compare/v8.18.0\...v8.18.1[View commits] === Beats version 8.18.0 https://github.com/elastic/beats/compare/v8.17.4\...v8.18.0[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Breaking changes *Affecting all Beats* @@ -583,6 +642,10 @@ otherwise no tag is added. {issue}42208[42208] {pull}42403[42403] === Beats version 8.17.10 https://github.com/elastic/beats/compare/v8.17.9\...v8.17.10[View commits] +==== Known Issues + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + ==== Bugfixes *Filebeat* @@ -607,6 +670,13 @@ https://github.com/elastic/beats/compare/v8.17.9\...v8.17.10[View commits] === Beats version 8.17.9 https://github.com/elastic/beats/compare/v8.17.8\...v8.17.9[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Breaking changes *Metricbeat* @@ -641,6 +711,13 @@ https://github.com/elastic/beats/compare/v8.17.8\...v8.17.9[View commits] === Beats version 8.17.8 https://github.com/elastic/beats/compare/v8.17.7\...v8.17.8[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Bugfixes *Affecting all Beats* @@ -682,6 +759,13 @@ https://github.com/elastic/beats/compare/v8.17.7\...v8.17.8[View commits] === Beats version 8.17.7 https://github.com/elastic/beats/compare/v8.17.6\...v8.17.7[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Bugfixes *Affecting all Beats* @@ -709,6 +793,13 @@ https://github.com/elastic/beats/compare/v8.17.6\...v8.17.7[View commits] === Beats version 8.17.6 https://github.com/elastic/beats/compare/v8.17.5\...v8.17.6[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Bugfixes *Filebeat* @@ -727,6 +818,13 @@ https://github.com/elastic/beats/compare/v8.17.5\...v8.17.6[View commits] === Beats version 8.17.5 https://github.com/elastic/beats/compare/v8.17.4\...v8.17.5[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Breaking changes *Metricbeat* @@ -767,6 +865,13 @@ https://github.com/elastic/beats/compare/v8.17.4\...v8.17.5[View commits] === Beats version 8.17.4 https://github.com/elastic/beats/compare/v8.17.3\...v8.17.4[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Breaking changes *Affecting all Beats* @@ -812,6 +917,13 @@ https://github.com/elastic/beats/compare/v8.17.3\...v8.17.4[View commits] === Beats version 8.17.3 https://github.com/elastic/beats/compare/v8.17.2\...v8.17.3[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Bugfixes *Affecting all Beats* @@ -855,6 +967,13 @@ https://github.com/elastic/beats/compare/v8.17.2\...v8.17.3[View commits] === Beats version 8.17.2 https://github.com/elastic/beats/compare/v8.17.1\...v8.17.2[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Bugfixes *Affecting all Beats* @@ -893,6 +1012,13 @@ https://github.com/elastic/beats/compare/v8.17.1\...v8.17.2[View commits] === Beats version 8.17.1 https://github.com/elastic/beats/compare/v8.17.0\...v8.17.1[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Breaking changes *Affecting all Beats* @@ -950,6 +1076,13 @@ https://github.com/elastic/beats/compare/v8.16.1\...v8.17.0[View commits] - Standalone Beats docker image will not start if `-e` option is not added {issue}42038[42038]. +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Breaking changes *Affecting all Beats* @@ -1032,6 +1165,13 @@ https://github.com/elastic/beats/compare/v8.16.1\...v8.17.0[View commits] === Beats version 8.16.5 https://github.com/elastic/beats/compare/v8.16.4\...v8.16.5[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Bugfixes *Filebeat* @@ -1061,6 +1201,12 @@ https://github.com/elastic/beats/compare/v8.16.4\...v8.16.5[View commits] === Beats version 8.16.4 https://github.com/elastic/beats/compare/v8.16.3\...v8.16.4[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + ==== Bugfixes *Filebeat* @@ -1084,6 +1230,13 @@ https://github.com/elastic/beats/compare/v8.16.3\...v8.16.4[View commits] === Beats version 8.16.3 https://github.com/elastic/beats/compare/v8.16.2\...v8.16.3[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + + ==== Breaking changes *Affecting all Beats* @@ -1129,6 +1282,12 @@ https://github.com/elastic/beats/compare/v8.16.2\...v8.16.3[View commits] === Beats version 8.16.2 https://github.com/elastic/beats/compare/v8.16.1\...v8.16.2[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + ==== Bugfixes *Affecting all Beats* @@ -1184,6 +1343,12 @@ https://github.com/elastic/beats/compare/v8.16.1\...v8.16.2[View commits] === Beats version 8.16.1 https://github.com/elastic/beats/compare/v8.16.0\...v8.16.1[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-1] + ==== Breaking changes *Packetbeat* @@ -1218,6 +1383,9 @@ filebeat.inputs: bucket_arn: 'arn:aws:s3:::test1' ---- + - The Filestream input does not enforce the restrictions documented for the `clean_inactive` option, thus allowing configurations that can lead to data re-ingestion issues. + - When `clean_inactive: 0`, Filestream cleans the state of all files on start up, effectively re-ingesting all files on restart. Set `clean_inactive: -1` to disable this behavior. + ==== Breaking changes *Affecting all Beats* @@ -1310,6 +1478,15 @@ filebeat.inputs: === Beats version 8.15.5 https://github.com/elastic/beats/compare/v8.15.4\...v8.15.5[View commits] +==== Known Issues + +*Filebeat* + +// tag::known-issue-8.19-filestream-clean-inactive-long-value[] +- The Filestream input does not enforce the restrictions documented for the `clean_inactive` option, thus allowing configurations that can lead to data re-ingestion issues. +- When `clean_inactive: 0`, Filestream cleans the state of all files on start up, effectively re-ingesting all files on restart. Set `clean_inactive` to a very large value. For example, use `clean_inactive: 43800h0m0s`, which is 5 years. +// end::known-issue-8.19-filestream-clean-inactive-long-value[] + ==== Breaking changes *Packetbeat* @@ -1331,6 +1508,12 @@ https://github.com/elastic/beats/compare/v8.15.4\...v8.15.5[View commits] === Beats version 8.15.4 https://github.com/elastic/beats/compare/v8.15.3\...v8.15.4[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-long-value] + ==== Breaking changes *Osquerybeat* @@ -1378,6 +1561,12 @@ https://github.com/elastic/beats/compare/v8.15.3\...v8.15.4[View commits] === Beats version 8.15.3 https://github.com/elastic/beats/compare/v8.15.2\...v8.15.3[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-long-value] + ==== Breaking changes *Filebeat* @@ -1426,6 +1615,10 @@ https://github.com/elastic/beats/compare/v8.15.0\...v8.15.2[View commits] - Beats Docker images do not log to stderr by default. The workaround is to pass the CLI flag `-e` or to set `logging.to_stderr: true` in the configuration file. {issue}41118[41118] +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-long-value] + ==== Bugfixes *Affecting all Beats* @@ -1447,6 +1640,11 @@ https://github.com/elastic/beats/compare/v8.15.0\...v8.15.1[View commits] - Beats stop publishing data after a network error unless restarted. Avoid upgrading to 8.15.1. Affected Beats log `Get \"https://${ELASTICSEARCH_HOST}:443\": context canceled` repeatedly. {issue}40705{40705} +*Filebeat* + + - The Filestream input does not enforce the restrictions documented for the `clean_inactive` option, thus allowing configurations that can lead to data re-ingestion issues. + - When `clean_inactive: 0`, Filestream cleans the state of all files on start up, effectively re-ingesting all files on restart. Set `clean_inactive: -1` to disable this behavior. + ==== Bugfixes *Affecting all Beats* @@ -1516,6 +1714,9 @@ filebeat.inputs: bucket_arn: 'arn:aws:s3:::test1' ---- + - The Filestream input does not enforce the restrictions documented for the `clean_inactive` option, thus allowing configurations that can lead to data re-ingestion issues. + - When `clean_inactive: 0`, Filestream cleans the state of all files on start up, effectively re-ingesting all files on restart. Set `clean_inactive: -1` to disable this behavior. + ==== Breaking changes *Filebeat* @@ -1606,6 +1807,13 @@ filebeat.inputs: === Beats version 8.14.3 https://github.com/elastic/beats/compare/v8.14.2\...v8.14.3[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-long-value] + + ==== Bugfixes *Filebeat* @@ -1635,6 +1843,13 @@ https://github.com/elastic/beats/compare/v8.14.2\...v8.14.3[View commits] === Beats version 8.14.2 https://github.com/elastic/beats/compare/v8.14.1\...v8.14.2[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-long-value] + + ==== Breaking changes *Filebeat* @@ -1671,6 +1886,12 @@ https://github.com/elastic/beats/compare/v8.14.1\...v8.14.2[View commits] === Beats version 8.14.1 https://github.com/elastic/beats/compare/v8.14.0\...v8.14.1[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-long-value] + ==== Bugfixes *Heartbeat* @@ -1682,6 +1903,12 @@ https://github.com/elastic/beats/compare/v8.14.0\...v8.14.1[View commits] === Beats version 8.14.0 https://github.com/elastic/beats/compare/v8.13.4\...v8.14.0[View commits] +==== Known Issues + +*Filebeat* + +include::CHANGELOG.asciidoc[tags=known-issue-8.19-filestream-clean-inactive-long-value] + ==== Breaking changes *Filebeat* @@ -1719,11 +1946,13 @@ https://github.com/elastic/beats/compare/v8.13.4\...v8.14.0[View commits] ==== Bugfixes *Auditbeat* + - Set field types to correctly match ECS in sessionmd processor. {issue}38955[38955] {pull}38994[38994] - Fix failing to enrich process events in sessionmd processor. {issue}38955[38955] {pull}39173[39173] {pull}39243[39243] - Fix seccomp policy of FIM kprobes backend on arm64. {pull}39759[39759] *Filebeat* + - Fix handling of endpoint for custom domains and ensure region, default_region, and region parsed from queue_url are applied in the order specified in the documentation for the awss3 input. {pull}39709[39709] - Prevent HTTPJSON holding response bodies between executions. {issue}35219[35219] {pull}38116[38116] - Fix the incorrect values generated by the uri_parts processor. {pull}38216[38216] diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 624b5cc108e6..255637c49d88 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -413,47 +413,6 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff] ==== Known Issues - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +*Filebeat* + - The Filestream input does not enforce the restrictions documented for the `clean_inactive` option, thus allowing configurations that can lead to data re-ingestion issues. + - When `clean_inactive: 0`, Filestream cleans the state of all files on start up, effectively re-ingesting all files on restart. Set `clean_inactive: -1` to disable this behavior. \ No newline at end of file diff --git a/filebeat/docs/inputs/input-filestream-file-options.asciidoc b/filebeat/docs/inputs/input-filestream-file-options.asciidoc index 17f7c391e8d5..222fe2de551d 100644 --- a/filebeat/docs/inputs/input-filestream-file-options.asciidoc +++ b/filebeat/docs/inputs/input-filestream-file-options.asciidoc @@ -439,6 +439,9 @@ prevent a potential <>. WARNING: Only use this option if you understand that data loss is a potential side effect. +WARNING: Filestream does not validate the restrictions for `clean_inactive`. Using an invalid +configuration can lead to data duplication. + When this option is enabled, {beatname_uc} removes the state of a file after the specified period of inactivity has elapsed. The state can only be removed if the file is already ignored by {beatname_uc} (the file is older than @@ -456,6 +459,10 @@ day. This config option is also useful to prevent {beatname_uc} problems resulting from inode reuse on Linux. For more information, see <>. +You can use time strings like `5m` (5 minutes), `2h45m` (2 hours and 45 +minutes), `48h`, etc. Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. The +default value is `-1` (disabled). + NOTE: Every time a file is renamed, the file state is updated and the counter for `clean_inactive` starts at 0 again.