Skip to content

Winlog Beat does not recover from recoverable error when reading from Windows Event Log #45750

@gogochan

Description

@gogochan

It is hard to reproduce in regular setup, but it is easier to reproduce with Benchbuilder setup.

While reading from Windows eventlog, winlog beat encounters following error, but does not recover from it.

Once it enter this state, it drops all events.

{"log.level":"error","@timestamp":"2025-08-05T14:18:22.768-0700","message":"encountered recoverable error when reading from Windows Event Log","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"winlog-default","type":"winlog"},"log":{"source":"winlog-default"},"log.logger":"input.winlog","id":"winlog-microsoft_sqlserver.audit","input_source":"winlog-microsoft_sqlserver.audit","eventlog":"winlog-microsoft_sqlserver.audit","log.origin":{"file.line":73,"file.name":"eventlog/runner.go","function":"github.com/elastic/beats/v7/winlogbeat/eventlog.Run.func3"},"service.name":"filebeat","channel":"Application","error":{"message":"The query result is stale or invalid and must be recreated. This may be due to the log being cleared or rolling over after the query result was created."},"ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-08-05T14:18:22.768-0700","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":912},"message":"Unit state changed winlog-default-audit_logs-winlog (HEALTHY->DEGRADED): Retrying to read from Application: The query result is stale or invalid and must be recreated. This may be due to the log being cleared or rolling over after the query result was created.","log":{"source":"elastic-agent"},"component":{"id":"winlog-default","state":"HEALTHY"},"unit":{"id":"winlog-default-audit_logs-winlog","type":"input","state":"DEGRADED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-08-05T14:18:26.685-0700","message":"1 file is too small to be ingested, files need to be at least 1024 in size for ingestion to start. To change this behaviour set 'prospector.scanner.fingerprint.length' and 'prospector.scanner.fingerprint.offset'. Enable debug logging to see all file names.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"ecs.version":"1.6.0","log.logger":"input.scanner","log.origin":{"file.line":440,"file.name":"filestream/fswatch.go","function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles"},"service.name":"filebeat","filestream_id":"filestream-monitoring-agent","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-08-05T14:18:27.239-0700","message":"Error dialing dial tcp 35.193.143.25:9200: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","network.transport":"tcp","ecs.version":"1.6.0","log.logger":"elasticsearch.esclientleg","log.origin":{"file.line":39,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport.LoggingDialer.func1"},"server.address":"ingest-metricstore.es.us-central1.gcp.cloud.es.io:9200","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-08-05T14:18:27.769-0700","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":912},"message":"Unit state changed winlog-default-audit_logs-winlog (DEGRADED->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"winlog-default","state":"HEALTHY"},"unit":{"id":"winlog-default-audit_logs-winlog","type":"input","state":"HEALTHY","old_state":"DEGRADED"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-08-05T14:18:27.777-0700","message":"Dropping event due to rendering error.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"winlog-default","type":"winlog"},"log":{"source":"winlog-default"},"service.name":"filebeat","error":{"message":"failed to render system properties: failed to get system values: failed in EvtRender: The handle is invalid."},"log.logger":"wineventlog","id":"winlog-microsoft_sqlserver.audit","channel":"Application","ecs.version":"1.6.0","log.origin":{"file.line":287,"file.name":"eventlog/wineventlog.go","function":"github.com/elastic/beats/v7/winlogbeat/eventlog.(*winEventLog).Read"},"ecs.version":"1.6.0"}

For confirmed bugs, please report:

  • Version: 9.1.0 9.1.1
  • Operating System: Windows 10, Windows 11
  • Discuss Forum URL:
  • Steps to Reproduce:
  1. Deploy microsoft_sqlserver integration on a Agent running on a Windows.
  2. Enable Collect audit events from Windows event logs
  3. Enter Application for channel.
Image
  1. Run attached script that generates Windows event log.
$serverName = "WINDEV2108EVAL"
$userName = "User"
$applicationName = "SQLCMD"
$hostName = "WINDEV2108EVAL"

# Smaller log size removes the event before we can read it.
$targetLog = Get-WinEvent -ListLog "Application"
$targetLog.MaximumSizeInBytes = 2105344
$targetLog.SaveChanges()

function Generate-EventText {
    param (
        [int]$SequenceNumber,
        [bool]$IsSuccess
    )

    $succeeded = if ($IsSuccess) { "true" } else { "false" }
    $statement = if ($IsSuccess) { "" } else { "Login failed for user 'User'. Reason: Could not find a login matching the name provided. [CLIENT: <local machine>]" }
    $errorCode = if ($IsSuccess) { "0x00000000" } else { "0x00004818" }
    $state = if ($IsSuccess) { "0" } else { "5" }
    $sequenceGroupId = $(New-Guid)
    $connectionId = $(New-Guid)

    $eventText = @"
audit_schema_version:1
event_time:$(Get-Date -Format "yyyy-MM-dd HH:mm:ss.fffffff")
sequence_number:$SequenceNumber
action_id:$(if ($IsSuccess) { "LGIS" } else { "LGIF" })
succeeded:$succeeded
is_column_permission:false
session_id:0
server_principal_id:0
database_principal_id:0
target_server_principal_id:0
target_database_principal_id:0
object_id:0
user_defined_event_id:0
transaction_id:0
class_type:LX
duration_milliseconds:0
response_rows:0
affected_rows:0
client_ip:local machine
permission_bitmask:00000000000000000000000000000000
sequence_group_id:$sequenceGroupId
session_server_principal_name:
server_principal_name:$userName
server_principal_sid:
database_principal_name:
target_server_principal_name:
target_server_principal_sid:
target_database_principal_name:
server_instance_name:$serverName
database_name:
schema_name:
object_name:
statement:$statement
additional_information:<action_info xmlns="http://schemas.microsoft.com/sqlserver/2008/sqlaudit_data"><pooled_connection>0</pooled_connection><error>$errorCode</error><state>$state</state><address>local machine</address></action_info>
user_defined_information:
application_name:$applicationName
connection_id:$(New-Guid)
data_sensitivity_information:
host_name:$hostName
"@

    return $eventText
}

if (-not (Get-WinEvent -LogName Application -ErrorAction SilentlyContinue)) {
    try {
        New-EventLog -LogName Application -Source "TextBenchBuilder"
    } catch {
        Write-Host "Failed to create event log source. Ensure you are running as an administrator."
        exit
    }
}

$i = 0
while ($true) {
    $isSuccess = ($i % 2) -eq 0 # Alternate between success and failure
    $eventText = Generate-EventText -SequenceNumber $i -IsSuccess $isSuccess
    $source = "MSSQLSERVER"

    if (![System.Diagnostics.EventLog]::SourceExists($source)) {
        [System.Diagnostics.EventLog]::CreateEventSource($source, $logName)
    }

    try {
        [System.Diagnostics.EventLog]::WriteEntry($source, $eventText, [System.Diagnostics.EventLogEntryType]::Information, 33205, 0)
    } catch {
        Write-Host "Failed to write event $i. Ensure you are running as an administrator."
    }
    
    $i++
}

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions